2024-04-26T04:05:17.918 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-26T04:05:17.918 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yuriw@teuthology --name yuriw-2024-04-26_01:23:56-rados-wip-yuri2-testing-2024-04-24-0914-squid-distro-default-smithi --description rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/sync-many workloads/pool-create-delete} --verbose --archive /home/teuthworker/archive/yuriw-2024-04-26_01:23:56-rados-wip-yuri2-testing-2024-04-24-0914-squid-distro-default-smithi/7674515 -- /home/teuthworker/archive/yuriw-2024-04-26_01:23:56-rados-wip-yuri2-testing-2024-04-24-0914-squid-distro-default-smithi/7674515/orig.config.yaml 2024-04-26T04:05:17.963 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-26T04:05:18.046 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-04-26_01:23:56-rados-wip-yuri2-testing-2024-04-24-0914-squid-distro-default-smithi/7674515 branch: wip-yuri2-testing-2024-04-24-0914-squid description: rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/sync-many workloads/pool-create-delete} email: yweinste@redhat.com first_in_suite: false job_id: '7674515' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: yuriw-2024-04-26_01:23:56-rados-wip-yuri2-testing-2024-04-24-0914-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-yuri2-testing-2024-04-24-0914-squid ceph: conf: client: debug monc: 20 debug ms: 1 global: mon client directed command retry: 5 mon election default strategy: 1 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: mon ms inject internal delays: 0.002 ms inject socket failures: 2500 ms type: async mgr: debug mgr: 20 debug monc: 10 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 25 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub inject crc mismatch: 0.01 mon scrub inject missing keys: 0.05 mon scrub interval: 300 paxos min: 10 paxos service trim min: 5 paxos trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - ScrubResult - scrub mismatch - overall HEALTH_ - \(MGR_DOWN\) - \(PG_AVAILABILITY\) - \(SLOW_OPS\) - slow request - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(MON_DOWN\) - \(POOL_APP_NOT_ENABLED\) - slow request - overall HEALTH_ sha1: ddeaa91797339a19172c3036ff48cca58c12f448 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: ddeaa91797339a19172c3036ff48cca58c12f448 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri2-testing-2024-04-24-0914-squid sha1: ddeaa91797339a19172c3036ff48cca58c12f448 owner: scheduled_yuriw@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - osd.0 - osd.1 - osd.2 - - mon.b - mgr.x - osd.3 - osd.4 - osd.5 - client.0 seed: 1848 sha1: ddeaa91797339a19172c3036ff48cca58c12f448 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri2-testing-2024-04-24-0914-squid suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_ddeaa91797339a19172c3036ff48cca58c12f448/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: ddeaa91797339a19172c3036ff48cca58c12f448 targets: smithi063.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMPjy2nGYyGpIxXaEuFqjiO8JiR0ZipLFjyWlr1wVxKA4aYKg0370mBvmLKfaTm2JcV/i8NkujatTa7hECUx7YM= smithi186.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH8Pvzwaklc3FDhfrA4zSkFmK1Fjomd9nV+PpvdXoNCYyIKQ1F5pwid2lnW8OR0aEiG3ibraLMdukNIoa52S1/w= tasks: - install: null - ceph: null - mon_thrash: revive_delay: 90 thrash_delay: 1 thrash_many: true - exec: client.0: - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-26_01:23:56 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-26T04:05:18.047 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_ddeaa91797339a19172c3036ff48cca58c12f448/qa; will attempt to use it 2024-04-26T04:05:18.047 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_ddeaa91797339a19172c3036ff48cca58c12f448/qa/tasks 2024-04-26T04:05:18.047 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-26T04:05:18.049 INFO:teuthology.task.internal:Checking packages... 2024-04-26T04:05:18.075 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'ddeaa91797339a19172c3036ff48cca58c12f448' 2024-04-26T04:05:18.075 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-26T04:05:18.075 INFO:teuthology.packaging:ref: None 2024-04-26T04:05:18.075 INFO:teuthology.packaging:tag: None 2024-04-26T04:05:18.075 INFO:teuthology.packaging:branch: wip-yuri2-testing-2024-04-24-0914-squid 2024-04-26T04:05:18.075 INFO:teuthology.packaging:sha1: ddeaa91797339a19172c3036ff48cca58c12f448 2024-04-26T04:05:18.076 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-yuri2-testing-2024-04-24-0914-squid 2024-04-26T04:05:18.216 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2457-gddeaa917-1jammy 2024-04-26T04:05:18.217 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-26T04:05:18.223 INFO:teuthology.task.internal:no buildpackages task found 2024-04-26T04:05:18.223 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-26T04:05:18.242 INFO:teuthology.task.internal:Saving configuration 2024-04-26T04:05:18.260 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-26T04:05:18.267 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-26T04:05:18.288 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi063.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-26_01:23:56-rados-wip-yuri2-testing-2024-04-24-0914-squid-distro-default-smithi/7674515', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 03:59:30.340309', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAg0hrGH9wIq2TxQZGqSQCbYshZJ9DNaG955RYk8cbMG'} 2024-04-26T04:05:18.304 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi186.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-26_01:23:56-rados-wip-yuri2-testing-2024-04-24-0914-squid-distro-default-smithi/7674515', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-26 03:59:30.341917', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDaS7RvA/u4lbL+Vm/41BLUqoh+CfCNzcbY+PKeY1gOp'} 2024-04-26T04:05:18.304 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-26T04:05:18.311 INFO:teuthology.task.internal:roles: ubuntu@smithi063.front.sepia.ceph.com - ['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'] 2024-04-26T04:05:18.311 INFO:teuthology.task.internal:roles: ubuntu@smithi186.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0'] 2024-04-26T04:05:18.311 INFO:teuthology.run_tasks:Running task console_log... 2024-04-26T04:05:18.372 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fd74c1643a0>, signals=[15]) 2024-04-26T04:05:18.373 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-26T04:05:18.379 INFO:teuthology.task.internal:Opening connections... 2024-04-26T04:05:18.380 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi063.front.sepia.ceph.com 2024-04-26T04:05:18.381 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T04:05:18.465 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi186.front.sepia.ceph.com 2024-04-26T04:05:18.466 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T04:05:18.546 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-26T04:05:18.555 DEBUG:teuthology.orchestra.run.smithi063:> uname -m 2024-04-26T04:05:18.561 INFO:teuthology.orchestra.run.smithi063.stdout:x86_64 2024-04-26T04:05:18.561 DEBUG:teuthology.orchestra.run.smithi063:> cat /etc/os-release 2024-04-26T04:05:18.606 INFO:teuthology.orchestra.run.smithi063.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-26T04:05:18.607 INFO:teuthology.orchestra.run.smithi063.stdout:NAME="Ubuntu" 2024-04-26T04:05:18.607 INFO:teuthology.orchestra.run.smithi063.stdout:VERSION_ID="22.04" 2024-04-26T04:05:18.607 INFO:teuthology.orchestra.run.smithi063.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-26T04:05:18.607 INFO:teuthology.orchestra.run.smithi063.stdout:VERSION_CODENAME=jammy 2024-04-26T04:05:18.607 INFO:teuthology.orchestra.run.smithi063.stdout:ID=ubuntu 2024-04-26T04:05:18.607 INFO:teuthology.orchestra.run.smithi063.stdout:ID_LIKE=debian 2024-04-26T04:05:18.607 INFO:teuthology.orchestra.run.smithi063.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-26T04:05:18.607 INFO:teuthology.orchestra.run.smithi063.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-26T04:05:18.607 INFO:teuthology.orchestra.run.smithi063.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-26T04:05:18.607 INFO:teuthology.orchestra.run.smithi063.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-26T04:05:18.607 INFO:teuthology.orchestra.run.smithi063.stdout:UBUNTU_CODENAME=jammy 2024-04-26T04:05:18.608 INFO:teuthology.lock.ops:Updating smithi063.front.sepia.ceph.com on lock server 2024-04-26T04:05:18.628 DEBUG:teuthology.orchestra.run.smithi186:> uname -m 2024-04-26T04:05:18.633 INFO:teuthology.orchestra.run.smithi186.stdout:x86_64 2024-04-26T04:05:18.633 DEBUG:teuthology.orchestra.run.smithi186:> cat /etc/os-release 2024-04-26T04:05:18.682 INFO:teuthology.orchestra.run.smithi186.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-26T04:05:18.682 INFO:teuthology.orchestra.run.smithi186.stdout:NAME="Ubuntu" 2024-04-26T04:05:18.682 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION_ID="22.04" 2024-04-26T04:05:18.682 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-26T04:05:18.682 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION_CODENAME=jammy 2024-04-26T04:05:18.682 INFO:teuthology.orchestra.run.smithi186.stdout:ID=ubuntu 2024-04-26T04:05:18.683 INFO:teuthology.orchestra.run.smithi186.stdout:ID_LIKE=debian 2024-04-26T04:05:18.683 INFO:teuthology.orchestra.run.smithi186.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-26T04:05:18.683 INFO:teuthology.orchestra.run.smithi186.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-26T04:05:18.683 INFO:teuthology.orchestra.run.smithi186.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-26T04:05:18.683 INFO:teuthology.orchestra.run.smithi186.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-26T04:05:18.683 INFO:teuthology.orchestra.run.smithi186.stdout:UBUNTU_CODENAME=jammy 2024-04-26T04:05:18.683 INFO:teuthology.lock.ops:Updating smithi186.front.sepia.ceph.com on lock server 2024-04-26T04:05:18.702 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-26T04:05:18.711 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-26T04:05:18.719 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-26T04:05:18.720 DEBUG:teuthology.orchestra.run.smithi063:> test '!' -e /home/ubuntu/cephtest 2024-04-26T04:05:18.722 DEBUG:teuthology.orchestra.run.smithi186:> test '!' -e /home/ubuntu/cephtest 2024-04-26T04:05:18.729 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-26T04:05:18.735 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-26T04:05:18.736 DEBUG:teuthology.orchestra.run.smithi063:> test -z $(ls -A /var/lib/ceph) 2024-04-26T04:05:18.768 DEBUG:teuthology.orchestra.run.smithi186:> test -z $(ls -A /var/lib/ceph) 2024-04-26T04:05:18.784 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-26T04:05:18.828 INFO:teuthology.run_tasks:Running task kernel... 2024-04-26T04:05:18.843 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-26T04:05:18.843 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-26T04:05:18.843 DEBUG:teuthology.orchestra.run.smithi063:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-26T04:05:18.844 DEBUG:teuthology.orchestra.run.smithi186:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-26T04:05:18.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T04:05:18.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T04:05:18.851 DEBUG:teuthology.orchestra.run.smithi186:> uname -r 2024-04-26T04:05:18.851 DEBUG:teuthology.orchestra.run.smithi063:> uname -r 2024-04-26T04:05:18.899 INFO:teuthology.orchestra.run.smithi186.stdout:5.15.0-56-generic 2024-04-26T04:05:18.899 INFO:teuthology.task.kernel:Running kernel on smithi186: 5.15.0-56-generic 2024-04-26T04:05:18.900 DEBUG:teuthology.orchestra.run.smithi186:> sudo apt-get clean 2024-04-26T04:05:18.901 INFO:teuthology.orchestra.run.smithi063.stdout:5.15.0-56-generic 2024-04-26T04:05:18.901 INFO:teuthology.task.kernel:Running kernel on smithi063: 5.15.0-56-generic 2024-04-26T04:05:18.901 DEBUG:teuthology.orchestra.run.smithi063:> sudo apt-get clean 2024-04-26T04:05:19.027 DEBUG:teuthology.orchestra.run.smithi063:> sudo apt-get update 2024-04-26T04:05:19.047 DEBUG:teuthology.orchestra.run.smithi186:> sudo apt-get update 2024-04-26T04:05:19.173 INFO:teuthology.orchestra.run.smithi063.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-26T04:05:19.178 INFO:teuthology.orchestra.run.smithi186.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-26T04:05:19.281 INFO:teuthology.orchestra.run.smithi063.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-26T04:05:19.292 INFO:teuthology.orchestra.run.smithi186.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-26T04:05:19.363 INFO:teuthology.orchestra.run.smithi063.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-26T04:05:19.374 INFO:teuthology.orchestra.run.smithi186.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-26T04:05:19.400 INFO:teuthology.orchestra.run.smithi186.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,394 kB] 2024-04-26T04:05:19.415 INFO:teuthology.orchestra.run.smithi063.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [452 kB] 2024-04-26T04:05:19.517 INFO:teuthology.orchestra.run.smithi186.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [452 kB] 2024-04-26T04:05:19.522 INFO:teuthology.orchestra.run.smithi186.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [243 kB] 2024-04-26T04:05:19.523 INFO:teuthology.orchestra.run.smithi063.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,394 kB] 2024-04-26T04:05:19.526 INFO:teuthology.orchestra.run.smithi186.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-04-26T04:05:19.526 INFO:teuthology.orchestra.run.smithi186.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,773 kB] 2024-04-26T04:05:19.564 INFO:teuthology.orchestra.run.smithi186.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-04-26T04:05:19.564 INFO:teuthology.orchestra.run.smithi186.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [300 kB] 2024-04-26T04:05:19.570 INFO:teuthology.orchestra.run.smithi186.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-04-26T04:05:19.570 INFO:teuthology.orchestra.run.smithi186.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [601 kB] 2024-04-26T04:05:19.578 INFO:teuthology.orchestra.run.smithi186.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [848 kB] 2024-04-26T04:05:19.587 INFO:teuthology.orchestra.run.smithi186.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [163 kB] 2024-04-26T04:05:19.589 INFO:teuthology.orchestra.run.smithi186.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-04-26T04:05:19.591 INFO:teuthology.orchestra.run.smithi063.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [243 kB] 2024-04-26T04:05:19.596 INFO:teuthology.orchestra.run.smithi063.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-04-26T04:05:19.597 INFO:teuthology.orchestra.run.smithi063.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-04-26T04:05:19.599 INFO:teuthology.orchestra.run.smithi063.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,773 kB] 2024-04-26T04:05:19.609 INFO:teuthology.orchestra.run.smithi186.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-04-26T04:05:19.609 INFO:teuthology.orchestra.run.smithi186.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-04-26T04:05:19.631 INFO:teuthology.orchestra.run.smithi063.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [300 kB] 2024-04-26T04:05:19.636 INFO:teuthology.orchestra.run.smithi063.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-04-26T04:05:19.636 INFO:teuthology.orchestra.run.smithi063.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [601 kB] 2024-04-26T04:05:19.637 INFO:teuthology.orchestra.run.smithi186.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-04-26T04:05:19.644 INFO:teuthology.orchestra.run.smithi063.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [848 kB] 2024-04-26T04:05:19.648 INFO:teuthology.orchestra.run.smithi186.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-04-26T04:05:19.653 INFO:teuthology.orchestra.run.smithi063.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [163 kB] 2024-04-26T04:05:19.688 INFO:teuthology.orchestra.run.smithi063.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-04-26T04:05:19.688 INFO:teuthology.orchestra.run.smithi063.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-04-26T04:05:19.688 INFO:teuthology.orchestra.run.smithi063.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-04-26T04:05:19.688 INFO:teuthology.orchestra.run.smithi063.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-04-26T04:05:19.688 INFO:teuthology.orchestra.run.smithi063.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-04-26T04:05:19.725 INFO:teuthology.orchestra.run.smithi063.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-04-26T04:05:19.726 INFO:teuthology.orchestra.run.smithi186.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-04-26T04:05:19.864 INFO:teuthology.orchestra.run.smithi063.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,610 kB] 2024-04-26T04:05:19.865 INFO:teuthology.orchestra.run.smithi186.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [618 kB] 2024-04-26T04:05:20.064 INFO:teuthology.orchestra.run.smithi186.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,610 kB] 2024-04-26T04:05:20.176 INFO:teuthology.orchestra.run.smithi063.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [618 kB] 2024-04-26T04:05:20.214 INFO:teuthology.orchestra.run.smithi063.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [303 kB] 2024-04-26T04:05:20.215 INFO:teuthology.orchestra.run.smithi186.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [303 kB] 2024-04-26T04:05:20.229 INFO:teuthology.orchestra.run.smithi063.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-04-26T04:05:20.230 INFO:teuthology.orchestra.run.smithi186.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-04-26T04:05:20.230 INFO:teuthology.orchestra.run.smithi063.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,830 kB] 2024-04-26T04:05:20.231 INFO:teuthology.orchestra.run.smithi186.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,830 kB] 2024-04-26T04:05:20.293 INFO:teuthology.orchestra.run.smithi186.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-04-26T04:05:20.293 INFO:teuthology.orchestra.run.smithi063.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-04-26T04:05:20.293 INFO:teuthology.orchestra.run.smithi063.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [311 kB] 2024-04-26T04:05:20.294 INFO:teuthology.orchestra.run.smithi186.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [311 kB] 2024-04-26T04:05:20.301 INFO:teuthology.orchestra.run.smithi063.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-04-26T04:05:20.301 INFO:teuthology.orchestra.run.smithi186.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-04-26T04:05:20.465 INFO:teuthology.orchestra.run.smithi063.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [700 kB] 2024-04-26T04:05:20.466 INFO:teuthology.orchestra.run.smithi186.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,071 kB] 2024-04-26T04:05:21.047 INFO:teuthology.orchestra.run.smithi063.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,071 kB] 2024-04-26T04:05:21.102 INFO:teuthology.orchestra.run.smithi186.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [700 kB] 2024-04-26T04:05:21.139 INFO:teuthology.orchestra.run.smithi186.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [244 kB] 2024-04-26T04:05:21.157 INFO:teuthology.orchestra.run.smithi186.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-04-26T04:05:21.158 INFO:teuthology.orchestra.run.smithi186.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-04-26T04:05:21.158 INFO:teuthology.orchestra.run.smithi063.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [244 kB] 2024-04-26T04:05:21.159 INFO:teuthology.orchestra.run.smithi186.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-04-26T04:05:21.162 INFO:teuthology.orchestra.run.smithi186.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-04-26T04:05:21.162 INFO:teuthology.orchestra.run.smithi186.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-04-26T04:05:21.162 INFO:teuthology.orchestra.run.smithi186.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-04-26T04:05:21.178 INFO:teuthology.orchestra.run.smithi063.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-04-26T04:05:21.180 INFO:teuthology.orchestra.run.smithi063.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-04-26T04:05:21.181 INFO:teuthology.orchestra.run.smithi063.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-04-26T04:05:21.184 INFO:teuthology.orchestra.run.smithi186.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.3 kB] 2024-04-26T04:05:21.185 INFO:teuthology.orchestra.run.smithi063.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-04-26T04:05:21.185 INFO:teuthology.orchestra.run.smithi063.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-04-26T04:05:21.185 INFO:teuthology.orchestra.run.smithi063.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-04-26T04:05:21.187 INFO:teuthology.orchestra.run.smithi186.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-04-26T04:05:21.189 INFO:teuthology.orchestra.run.smithi063.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.3 kB] 2024-04-26T04:05:21.194 INFO:teuthology.orchestra.run.smithi063.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-04-26T04:05:21.354 INFO:teuthology.orchestra.run.smithi186.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-04-26T04:05:21.358 INFO:teuthology.orchestra.run.smithi063.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-04-26T04:05:21.437 INFO:teuthology.orchestra.run.smithi186.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.2 kB] 2024-04-26T04:05:21.441 INFO:teuthology.orchestra.run.smithi063.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.2 kB] 2024-04-26T04:05:21.544 INFO:teuthology.orchestra.run.smithi186.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.0 kB] 2024-04-26T04:05:21.549 INFO:teuthology.orchestra.run.smithi063.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.0 kB] 2024-04-26T04:05:21.600 INFO:teuthology.orchestra.run.smithi186.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-04-26T04:05:21.603 INFO:teuthology.orchestra.run.smithi063.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-04-26T04:05:21.631 INFO:teuthology.orchestra.run.smithi186.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-04-26T04:05:21.633 INFO:teuthology.orchestra.run.smithi063.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-04-26T04:05:25.967 INFO:teuthology.orchestra.run.smithi186.stdout:Fetched 13.2 MB in 3s (5,248 kB/s) 2024-04-26T04:05:26.510 INFO:teuthology.orchestra.run.smithi063.stdout:Fetched 13.2 MB in 3s (5,220 kB/s) 2024-04-26T04:05:27.371 INFO:teuthology.orchestra.run.smithi186.stdout:Reading package lists... 2024-04-26T04:05:27.388 DEBUG:teuthology.orchestra.run.smithi186:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-26T04:05:27.440 INFO:teuthology.orchestra.run.smithi186.stdout:Reading package lists... 2024-04-26T04:05:27.619 INFO:teuthology.orchestra.run.smithi186.stdout:Building dependency tree... 2024-04-26T04:05:27.619 INFO:teuthology.orchestra.run.smithi186.stdout:Reading state information... 2024-04-26T04:05:27.776 INFO:teuthology.orchestra.run.smithi186.stdout:The following packages were automatically installed and are no longer required: 2024-04-26T04:05:27.777 INFO:teuthology.orchestra.run.smithi186.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-26T04:05:27.777 INFO:teuthology.orchestra.run.smithi186.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-26T04:05:27.777 INFO:teuthology.orchestra.run.smithi186.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-26T04:05:27.777 INFO:teuthology.orchestra.run.smithi186.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-26T04:05:27.778 INFO:teuthology.orchestra.run.smithi186.stdout: python2.7-minimal 2024-04-26T04:05:27.778 INFO:teuthology.orchestra.run.smithi186.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-26T04:05:27.778 INFO:teuthology.orchestra.run.smithi186.stdout:The following additional packages will be installed: 2024-04-26T04:05:27.778 INFO:teuthology.orchestra.run.smithi186.stdout: linux-generic linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-26T04:05:27.778 INFO:teuthology.orchestra.run.smithi186.stdout: linux-headers-generic linux-image-5.15.0-105-generic 2024-04-26T04:05:27.779 INFO:teuthology.orchestra.run.smithi186.stdout: linux-modules-5.15.0-105-generic linux-modules-extra-5.15.0-105-generic 2024-04-26T04:05:27.780 INFO:teuthology.orchestra.run.smithi186.stdout:Suggested packages: 2024-04-26T04:05:27.780 INFO:teuthology.orchestra.run.smithi186.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-04-26T04:05:27.780 INFO:teuthology.orchestra.run.smithi186.stdout:Recommended packages: 2024-04-26T04:05:27.780 INFO:teuthology.orchestra.run.smithi186.stdout: thermald 2024-04-26T04:05:27.834 INFO:teuthology.orchestra.run.smithi186.stdout:The following NEW packages will be installed: 2024-04-26T04:05:27.834 INFO:teuthology.orchestra.run.smithi186.stdout: linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-26T04:05:27.834 INFO:teuthology.orchestra.run.smithi186.stdout: linux-image-5.15.0-105-generic linux-modules-5.15.0-105-generic 2024-04-26T04:05:27.834 INFO:teuthology.orchestra.run.smithi186.stdout: linux-modules-extra-5.15.0-105-generic 2024-04-26T04:05:27.835 INFO:teuthology.orchestra.run.smithi186.stdout:The following packages will be upgraded: 2024-04-26T04:05:27.836 INFO:teuthology.orchestra.run.smithi186.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-26T04:05:27.873 INFO:teuthology.orchestra.run.smithi063.stdout:Reading package lists... 2024-04-26T04:05:27.890 DEBUG:teuthology.orchestra.run.smithi063:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-26T04:05:27.940 INFO:teuthology.orchestra.run.smithi063.stdout:Reading package lists... 2024-04-26T04:05:28.015 INFO:teuthology.orchestra.run.smithi186.stdout:3 upgraded, 5 newly installed, 0 to remove and 315 not upgraded. 2024-04-26T04:05:28.015 INFO:teuthology.orchestra.run.smithi186.stdout:Need to get 113 MB of archives. 2024-04-26T04:05:28.015 INFO:teuthology.orchestra.run.smithi186.stdout:After this operation, 583 MB of additional disk space will be used. 2024-04-26T04:05:28.015 INFO:teuthology.orchestra.run.smithi186.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-105-generic amd64 5.15.0-105.115 [22.7 MB] 2024-04-26T04:05:28.122 INFO:teuthology.orchestra.run.smithi063.stdout:Building dependency tree... 2024-04-26T04:05:28.126 INFO:teuthology.orchestra.run.smithi063.stdout:Reading state information... 2024-04-26T04:05:28.286 INFO:teuthology.orchestra.run.smithi063.stdout:The following packages were automatically installed and are no longer required: 2024-04-26T04:05:28.286 INFO:teuthology.orchestra.run.smithi063.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-26T04:05:28.286 INFO:teuthology.orchestra.run.smithi063.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-26T04:05:28.286 INFO:teuthology.orchestra.run.smithi063.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-26T04:05:28.287 INFO:teuthology.orchestra.run.smithi063.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-26T04:05:28.287 INFO:teuthology.orchestra.run.smithi063.stdout: python2.7-minimal 2024-04-26T04:05:28.287 INFO:teuthology.orchestra.run.smithi063.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-26T04:05:28.287 INFO:teuthology.orchestra.run.smithi063.stdout:The following additional packages will be installed: 2024-04-26T04:05:28.287 INFO:teuthology.orchestra.run.smithi063.stdout: linux-generic linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-26T04:05:28.287 INFO:teuthology.orchestra.run.smithi063.stdout: linux-headers-generic linux-image-5.15.0-105-generic 2024-04-26T04:05:28.287 INFO:teuthology.orchestra.run.smithi063.stdout: linux-modules-5.15.0-105-generic linux-modules-extra-5.15.0-105-generic 2024-04-26T04:05:28.287 INFO:teuthology.orchestra.run.smithi063.stdout:Suggested packages: 2024-04-26T04:05:28.287 INFO:teuthology.orchestra.run.smithi063.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-04-26T04:05:28.287 INFO:teuthology.orchestra.run.smithi063.stdout:Recommended packages: 2024-04-26T04:05:28.288 INFO:teuthology.orchestra.run.smithi063.stdout: thermald 2024-04-26T04:05:28.332 INFO:teuthology.orchestra.run.smithi063.stdout:The following NEW packages will be installed: 2024-04-26T04:05:28.333 INFO:teuthology.orchestra.run.smithi063.stdout: linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-26T04:05:28.333 INFO:teuthology.orchestra.run.smithi063.stdout: linux-image-5.15.0-105-generic linux-modules-5.15.0-105-generic 2024-04-26T04:05:28.333 INFO:teuthology.orchestra.run.smithi063.stdout: linux-modules-extra-5.15.0-105-generic 2024-04-26T04:05:28.334 INFO:teuthology.orchestra.run.smithi063.stdout:The following packages will be upgraded: 2024-04-26T04:05:28.335 INFO:teuthology.orchestra.run.smithi063.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-26T04:05:28.396 INFO:teuthology.orchestra.run.smithi063.stdout:3 upgraded, 5 newly installed, 0 to remove and 315 not upgraded. 2024-04-26T04:05:28.397 INFO:teuthology.orchestra.run.smithi063.stdout:Need to get 113 MB of archives. 2024-04-26T04:05:28.397 INFO:teuthology.orchestra.run.smithi063.stdout:After this operation, 583 MB of additional disk space will be used. 2024-04-26T04:05:28.397 INFO:teuthology.orchestra.run.smithi063.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-105-generic amd64 5.15.0-105.115 [22.7 MB] 2024-04-26T04:05:28.875 INFO:teuthology.orchestra.run.smithi063.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-105-generic amd64 5.15.0-105.115 [11.5 MB] 2024-04-26T04:05:29.007 INFO:teuthology.orchestra.run.smithi063.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-105-generic amd64 5.15.0-105.115 [63.9 MB] 2024-04-26T04:05:29.449 INFO:teuthology.orchestra.run.smithi186.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-105-generic amd64 5.15.0-105.115 [11.5 MB] 2024-04-26T04:05:29.843 INFO:teuthology.orchestra.run.smithi186.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-105-generic amd64 5.15.0-105.115 [63.9 MB] 2024-04-26T04:05:30.307 INFO:teuthology.orchestra.run.smithi063.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.105.102 [1,694 B] 2024-04-26T04:05:30.307 INFO:teuthology.orchestra.run.smithi063.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.105.102 [2,530 B] 2024-04-26T04:05:30.308 INFO:teuthology.orchestra.run.smithi063.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105 all 5.15.0-105.115 [12.3 MB] 2024-04-26T04:05:30.622 INFO:teuthology.orchestra.run.smithi063.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105-generic amd64 5.15.0-105.115 [2,857 kB] 2024-04-26T04:05:30.698 INFO:teuthology.orchestra.run.smithi063.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.105.102 [2,382 B] 2024-04-26T04:05:31.236 INFO:teuthology.orchestra.run.smithi063.stdout:Fetched 113 MB in 2s (48.1 MB/s) 2024-04-26T04:05:31.415 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package linux-modules-5.15.0-105-generic. 2024-04-26T04:05:32.023 INFO:teuthology.orchestra.run.smithi186.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.105.102 [1,694 B] 2024-04-26T04:05:32.023 INFO:teuthology.orchestra.run.smithi186.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.105.102 [2,530 B] 2024-04-26T04:05:32.023 INFO:teuthology.orchestra.run.smithi186.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105 all 5.15.0-105.115 [12.3 MB] 2024-04-26T04:05:32.577 INFO:teuthology.orchestra.run.smithi186.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105-generic amd64 5.15.0-105.115 [2,857 kB] 2024-04-26T04:05:32.679 INFO:teuthology.orchestra.run.smithi186.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.105.102 [2,382 B] 2024-04-26T04:05:33.060 INFO:teuthology.orchestra.run.smithi186.stdout:Fetched 113 MB in 5s (23.4 MB/s) 2024-04-26T04:05:33.232 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package linux-modules-5.15.0-105-generic. 2024-04-26T04:05:33.349 INFO:teuthology.orchestra.run.smithi063.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-04-26T04:05:33.354 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../0-linux-modules-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-26T04:05:33.562 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T04:05:35.220 INFO:teuthology.orchestra.run.smithi186.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-04-26T04:05:35.225 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../0-linux-modules-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-26T04:05:35.312 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T04:05:35.773 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package linux-image-5.15.0-105-generic. 2024-04-26T04:05:35.795 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../1-linux-image-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-26T04:05:35.885 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T04:05:36.499 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-105-generic. 2024-04-26T04:05:36.520 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-26T04:05:36.559 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T04:05:37.429 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package linux-image-5.15.0-105-generic. 2024-04-26T04:05:37.450 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../1-linux-image-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-26T04:05:37.543 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T04:05:38.041 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-105-generic. 2024-04-26T04:05:38.061 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-26T04:05:38.099 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T04:05:43.176 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../3-linux-generic_5.15.0.105.102_amd64.deb ... 2024-04-26T04:05:43.319 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking linux-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-26T04:05:43.644 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.105.102_amd64.deb ... 2024-04-26T04:05:43.772 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking linux-image-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-26T04:05:44.092 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package linux-headers-5.15.0-105. 2024-04-26T04:05:44.115 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../5-linux-headers-5.15.0-105_5.15.0-105.115_all.deb ... 2024-04-26T04:05:44.158 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-26T04:05:45.065 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../3-linux-generic_5.15.0.105.102_amd64.deb ... 2024-04-26T04:05:45.190 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking linux-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-26T04:05:45.540 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.105.102_amd64.deb ... 2024-04-26T04:05:45.669 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking linux-image-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-26T04:05:45.988 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package linux-headers-5.15.0-105. 2024-04-26T04:05:46.010 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../5-linux-headers-5.15.0-105_5.15.0-105.115_all.deb ... 2024-04-26T04:05:46.054 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-26T04:05:50.214 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package linux-headers-5.15.0-105-generic. 2024-04-26T04:05:50.252 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../6-linux-headers-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-26T04:05:50.289 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T04:05:52.102 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package linux-headers-5.15.0-105-generic. 2024-04-26T04:05:52.138 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../6-linux-headers-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-26T04:05:52.176 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T04:05:53.034 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.105.102_amd64.deb ... 2024-04-26T04:05:53.159 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking linux-headers-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-26T04:05:53.500 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-26T04:05:53.596 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T04:05:53.706 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up linux-headers-generic (5.15.0.105.102) ... 2024-04-26T04:05:53.851 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T04:05:55.259 INFO:teuthology.orchestra.run.smithi063.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-04-26T04:05:55.259 INFO:teuthology.orchestra.run.smithi063.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-04-26T04:05:55.259 INFO:teuthology.orchestra.run.smithi063.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-105-generic 2024-04-26T04:05:55.259 INFO:teuthology.orchestra.run.smithi063.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-105-generic 2024-04-26T04:05:55.569 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T04:05:55.712 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.105.102_amd64.deb ... 2024-04-26T04:05:55.863 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking linux-headers-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-26T04:05:56.221 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-26T04:05:56.342 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T04:05:56.468 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up linux-headers-generic (5.15.0.105.102) ... 2024-04-26T04:05:56.597 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T04:05:56.996 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T04:05:57.933 INFO:teuthology.orchestra.run.smithi186.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-04-26T04:05:57.933 INFO:teuthology.orchestra.run.smithi186.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-04-26T04:05:57.933 INFO:teuthology.orchestra.run.smithi186.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-105-generic 2024-04-26T04:05:57.933 INFO:teuthology.orchestra.run.smithi186.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-105-generic 2024-04-26T04:05:58.091 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T04:05:58.438 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up linux-image-generic (5.15.0.105.102) ... 2024-04-26T04:05:58.564 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up linux-generic (5.15.0.105.102) ... 2024-04-26T04:05:58.716 INFO:teuthology.orchestra.run.smithi063.stdout:Processing triggers for linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T04:05:58.779 INFO:teuthology.orchestra.run.smithi063.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-26T04:05:58.779 INFO:teuthology.orchestra.run.smithi063.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-105-generic 2024-04-26T04:05:59.484 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T04:06:01.244 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up linux-image-generic (5.15.0.105.102) ... 2024-04-26T04:06:01.369 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up linux-generic (5.15.0.105.102) ... 2024-04-26T04:06:01.521 INFO:teuthology.orchestra.run.smithi186.stdout:Processing triggers for linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-26T04:06:01.591 INFO:teuthology.orchestra.run.smithi186.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-26T04:06:01.592 INFO:teuthology.orchestra.run.smithi186.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-105-generic 2024-04-26T04:06:13.444 INFO:teuthology.orchestra.run.smithi063.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-26T04:06:13.444 INFO:teuthology.orchestra.run.smithi063.stdout:Sourcing file `/etc/default/grub' 2024-04-26T04:06:13.460 INFO:teuthology.orchestra.run.smithi063.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-26T04:06:13.485 INFO:teuthology.orchestra.run.smithi063.stdout:Generating grub configuration file ... 2024-04-26T04:06:13.830 INFO:teuthology.orchestra.run.smithi063.stdout:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-26T04:06:13.845 INFO:teuthology.orchestra.run.smithi063.stdout:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-26T04:06:14.214 INFO:teuthology.orchestra.run.smithi063.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-26T04:06:14.220 INFO:teuthology.orchestra.run.smithi063.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-26T04:06:14.306 INFO:teuthology.orchestra.run.smithi063.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-26T04:06:14.311 INFO:teuthology.orchestra.run.smithi063.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-26T04:06:14.473 INFO:teuthology.orchestra.run.smithi063.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-26T04:06:14.473 INFO:teuthology.orchestra.run.smithi063.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-04-26T04:06:14.473 INFO:teuthology.orchestra.run.smithi063.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-26T04:06:14.511 INFO:teuthology.orchestra.run.smithi063.stdout:done 2024-04-26T04:06:14.998 DEBUG:teuthology.orchestra.run.smithi063:> dpkg -s linux-image-generic 2024-04-26T04:06:15.021 INFO:teuthology.orchestra.run.smithi063.stdout:Package: linux-image-generic 2024-04-26T04:06:15.021 INFO:teuthology.orchestra.run.smithi063.stdout:Status: install ok installed 2024-04-26T04:06:15.022 INFO:teuthology.orchestra.run.smithi063.stdout:Priority: optional 2024-04-26T04:06:15.022 INFO:teuthology.orchestra.run.smithi063.stdout:Section: kernel 2024-04-26T04:06:15.022 INFO:teuthology.orchestra.run.smithi063.stdout:Installed-Size: 21 2024-04-26T04:06:15.022 INFO:teuthology.orchestra.run.smithi063.stdout:Maintainer: Ubuntu Kernel Team 2024-04-26T04:06:15.022 INFO:teuthology.orchestra.run.smithi063.stdout:Architecture: amd64 2024-04-26T04:06:15.022 INFO:teuthology.orchestra.run.smithi063.stdout:Source: linux-meta 2024-04-26T04:06:15.022 INFO:teuthology.orchestra.run.smithi063.stdout:Version: 5.15.0.105.102 2024-04-26T04:06:15.022 INFO:teuthology.orchestra.run.smithi063.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-26T04:06:15.022 INFO:teuthology.orchestra.run.smithi063.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-26T04:06:15.022 INFO:teuthology.orchestra.run.smithi063.stdout:Recommends: thermald 2024-04-26T04:06:15.022 INFO:teuthology.orchestra.run.smithi063.stdout:Description: Generic Linux kernel image 2024-04-26T04:06:15.022 INFO:teuthology.orchestra.run.smithi063.stdout: This package will always depend on the latest generic kernel image 2024-04-26T04:06:15.023 INFO:teuthology.orchestra.run.smithi063.stdout: available. 2024-04-26T04:06:15.023 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-105-generic 2024-04-26T04:06:15.023 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-26T04:06:15.023 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-26T04:06:15.023 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-26T04:06:15.023 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi063.front.sepia.ceph.com, path=None, version=distro) 2024-04-26T04:06:15.023 DEBUG:teuthology.orchestra.run.smithi063:> sudo apt-get clean 2024-04-26T04:06:15.180 DEBUG:teuthology.orchestra.run.smithi063:> sudo apt-get update 2024-04-26T04:06:16.060 INFO:teuthology.orchestra.run.smithi186.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-26T04:06:16.060 INFO:teuthology.orchestra.run.smithi186.stdout:Sourcing file `/etc/default/grub' 2024-04-26T04:06:16.076 INFO:teuthology.orchestra.run.smithi186.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-26T04:06:16.096 INFO:teuthology.orchestra.run.smithi186.stdout:Generating grub configuration file ... 2024-04-26T04:06:16.165 INFO:teuthology.orchestra.run.smithi063.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-26T04:06:16.165 INFO:teuthology.orchestra.run.smithi063.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-26T04:06:16.236 INFO:teuthology.orchestra.run.smithi063.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-26T04:06:16.318 INFO:teuthology.orchestra.run.smithi063.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-26T04:06:16.390 INFO:teuthology.orchestra.run.smithi186.stdout:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-26T04:06:16.411 INFO:teuthology.orchestra.run.smithi186.stdout:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-26T04:06:16.739 INFO:teuthology.orchestra.run.smithi186.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-26T04:06:16.745 INFO:teuthology.orchestra.run.smithi186.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-26T04:06:16.814 INFO:teuthology.orchestra.run.smithi186.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-26T04:06:16.820 INFO:teuthology.orchestra.run.smithi186.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-26T04:06:16.976 INFO:teuthology.orchestra.run.smithi186.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-26T04:06:16.976 INFO:teuthology.orchestra.run.smithi186.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-04-26T04:06:16.977 INFO:teuthology.orchestra.run.smithi186.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-26T04:06:17.012 INFO:teuthology.orchestra.run.smithi186.stdout:done 2024-04-26T04:06:17.457 DEBUG:teuthology.orchestra.run.smithi186:> dpkg -s linux-image-generic 2024-04-26T04:06:17.481 INFO:teuthology.orchestra.run.smithi186.stdout:Package: linux-image-generic 2024-04-26T04:06:17.481 INFO:teuthology.orchestra.run.smithi186.stdout:Status: install ok installed 2024-04-26T04:06:17.481 INFO:teuthology.orchestra.run.smithi186.stdout:Priority: optional 2024-04-26T04:06:17.481 INFO:teuthology.orchestra.run.smithi186.stdout:Section: kernel 2024-04-26T04:06:17.482 INFO:teuthology.orchestra.run.smithi186.stdout:Installed-Size: 21 2024-04-26T04:06:17.482 INFO:teuthology.orchestra.run.smithi186.stdout:Maintainer: Ubuntu Kernel Team 2024-04-26T04:06:17.482 INFO:teuthology.orchestra.run.smithi186.stdout:Architecture: amd64 2024-04-26T04:06:17.482 INFO:teuthology.orchestra.run.smithi186.stdout:Source: linux-meta 2024-04-26T04:06:17.482 INFO:teuthology.orchestra.run.smithi186.stdout:Version: 5.15.0.105.102 2024-04-26T04:06:17.482 INFO:teuthology.orchestra.run.smithi186.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-26T04:06:17.482 INFO:teuthology.orchestra.run.smithi186.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-26T04:06:17.482 INFO:teuthology.orchestra.run.smithi186.stdout:Recommends: thermald 2024-04-26T04:06:17.482 INFO:teuthology.orchestra.run.smithi186.stdout:Description: Generic Linux kernel image 2024-04-26T04:06:17.482 INFO:teuthology.orchestra.run.smithi186.stdout: This package will always depend on the latest generic kernel image 2024-04-26T04:06:17.482 INFO:teuthology.orchestra.run.smithi186.stdout: available. 2024-04-26T04:06:17.482 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-105-generic 2024-04-26T04:06:17.483 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-26T04:06:17.483 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-26T04:06:17.483 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-04-26T04:06:17.483 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi186.front.sepia.ceph.com, path=None, version=distro) 2024-04-26T04:06:17.483 DEBUG:teuthology.orchestra.run.smithi186:> sudo apt-get clean 2024-04-26T04:06:17.588 INFO:teuthology.orchestra.run.smithi063.stdout:Reading package lists... 2024-04-26T04:06:17.605 DEBUG:teuthology.orchestra.run.smithi063:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-26T04:06:17.659 INFO:teuthology.orchestra.run.smithi063.stdout:Reading package lists... 2024-04-26T04:06:17.664 DEBUG:teuthology.orchestra.run.smithi186:> sudo apt-get update 2024-04-26T04:06:17.786 INFO:teuthology.orchestra.run.smithi186.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-26T04:06:17.840 INFO:teuthology.orchestra.run.smithi063.stdout:Building dependency tree... 2024-04-26T04:06:17.840 INFO:teuthology.orchestra.run.smithi063.stdout:Reading state information... 2024-04-26T04:06:17.912 INFO:teuthology.orchestra.run.smithi186.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-26T04:06:17.987 INFO:teuthology.orchestra.run.smithi186.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-26T04:06:18.002 INFO:teuthology.orchestra.run.smithi063.stdout:linux-image-generic is already the newest version (5.15.0.105.102). 2024-04-26T04:06:18.002 INFO:teuthology.orchestra.run.smithi063.stdout:The following packages were automatically installed and are no longer required: 2024-04-26T04:06:18.003 INFO:teuthology.orchestra.run.smithi063.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-26T04:06:18.003 INFO:teuthology.orchestra.run.smithi063.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-26T04:06:18.003 INFO:teuthology.orchestra.run.smithi063.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-26T04:06:18.003 INFO:teuthology.orchestra.run.smithi063.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-26T04:06:18.003 INFO:teuthology.orchestra.run.smithi063.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-26T04:06:18.003 INFO:teuthology.orchestra.run.smithi063.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-26T04:06:18.044 INFO:teuthology.orchestra.run.smithi063.stdout:0 upgraded, 0 newly installed, 0 to remove and 315 not upgraded. 2024-04-26T04:06:18.045 DEBUG:teuthology.orchestra.run.smithi063:> dpkg -s linux-image-generic 2024-04-26T04:06:18.069 INFO:teuthology.orchestra.run.smithi186.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-26T04:06:18.102 INFO:teuthology.orchestra.run.smithi063.stdout:Package: linux-image-generic 2024-04-26T04:06:18.102 INFO:teuthology.orchestra.run.smithi063.stdout:Status: install ok installed 2024-04-26T04:06:18.102 INFO:teuthology.orchestra.run.smithi063.stdout:Priority: optional 2024-04-26T04:06:18.102 INFO:teuthology.orchestra.run.smithi063.stdout:Section: kernel 2024-04-26T04:06:18.102 INFO:teuthology.orchestra.run.smithi063.stdout:Installed-Size: 21 2024-04-26T04:06:18.102 INFO:teuthology.orchestra.run.smithi063.stdout:Maintainer: Ubuntu Kernel Team 2024-04-26T04:06:18.102 INFO:teuthology.orchestra.run.smithi063.stdout:Architecture: amd64 2024-04-26T04:06:18.102 INFO:teuthology.orchestra.run.smithi063.stdout:Source: linux-meta 2024-04-26T04:06:18.102 INFO:teuthology.orchestra.run.smithi063.stdout:Version: 5.15.0.105.102 2024-04-26T04:06:18.102 INFO:teuthology.orchestra.run.smithi063.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-26T04:06:18.102 INFO:teuthology.orchestra.run.smithi063.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-26T04:06:18.103 INFO:teuthology.orchestra.run.smithi063.stdout:Recommends: thermald 2024-04-26T04:06:18.103 INFO:teuthology.orchestra.run.smithi063.stdout:Description: Generic Linux kernel image 2024-04-26T04:06:18.103 INFO:teuthology.orchestra.run.smithi063.stdout: This package will always depend on the latest generic kernel image 2024-04-26T04:06:18.103 INFO:teuthology.orchestra.run.smithi063.stdout: available. 2024-04-26T04:06:18.103 DEBUG:teuthology.orchestra.run.smithi063:> mktemp 2024-04-26T04:06:18.146 INFO:teuthology.orchestra.run.smithi063.stdout:/tmp/tmp.nWrRc9BYr1 2024-04-26T04:06:18.147 DEBUG:teuthology.orchestra.run.smithi063:> sudo cp /boot/grub/grub.cfg /tmp/tmp.nWrRc9BYr1 2024-04-26T04:06:18.203 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod 0666 /tmp/tmp.nWrRc9BYr1 2024-04-26T04:06:18.319 DEBUG:teuthology.orchestra.remote:smithi063:/tmp/tmp.nWrRc9BYr1 is 10KB 2024-04-26T04:06:18.376 DEBUG:teuthology.orchestra.run.smithi063:> rm -fr /tmp/tmp.nWrRc9BYr1 2024-04-26T04:06:18.381 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-26T04:06:18.434 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-26T04:06:18.434 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-26T04:06:18.434 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-26T04:06:18.524 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-105-generic 2024-04-26T04:06:18.525 DEBUG:teuthology.orchestra.run.smithi063:> sudo update-grub 2024-04-26T04:06:19.230 INFO:teuthology.orchestra.run.smithi186.stdout:Reading package lists... 2024-04-26T04:06:19.249 DEBUG:teuthology.orchestra.run.smithi186:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-26T04:06:19.310 INFO:teuthology.orchestra.run.smithi186.stdout:Reading package lists... 2024-04-26T04:06:19.510 INFO:teuthology.orchestra.run.smithi186.stdout:Building dependency tree... 2024-04-26T04:06:19.511 INFO:teuthology.orchestra.run.smithi186.stdout:Reading state information... 2024-04-26T04:06:19.685 INFO:teuthology.orchestra.run.smithi186.stdout:linux-image-generic is already the newest version (5.15.0.105.102). 2024-04-26T04:06:19.685 INFO:teuthology.orchestra.run.smithi186.stdout:The following packages were automatically installed and are no longer required: 2024-04-26T04:06:19.685 INFO:teuthology.orchestra.run.smithi186.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-26T04:06:19.685 INFO:teuthology.orchestra.run.smithi186.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-26T04:06:19.685 INFO:teuthology.orchestra.run.smithi186.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-26T04:06:19.686 INFO:teuthology.orchestra.run.smithi186.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-26T04:06:19.686 INFO:teuthology.orchestra.run.smithi186.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-26T04:06:19.686 INFO:teuthology.orchestra.run.smithi186.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-26T04:06:19.725 INFO:teuthology.orchestra.run.smithi186.stdout:0 upgraded, 0 newly installed, 0 to remove and 315 not upgraded. 2024-04-26T04:06:19.727 DEBUG:teuthology.orchestra.run.smithi186:> dpkg -s linux-image-generic 2024-04-26T04:06:19.743 INFO:teuthology.orchestra.run.smithi186.stdout:Package: linux-image-generic 2024-04-26T04:06:19.743 INFO:teuthology.orchestra.run.smithi186.stdout:Status: install ok installed 2024-04-26T04:06:19.743 INFO:teuthology.orchestra.run.smithi186.stdout:Priority: optional 2024-04-26T04:06:19.743 INFO:teuthology.orchestra.run.smithi186.stdout:Section: kernel 2024-04-26T04:06:19.744 INFO:teuthology.orchestra.run.smithi186.stdout:Installed-Size: 21 2024-04-26T04:06:19.744 INFO:teuthology.orchestra.run.smithi186.stdout:Maintainer: Ubuntu Kernel Team 2024-04-26T04:06:19.744 INFO:teuthology.orchestra.run.smithi186.stdout:Architecture: amd64 2024-04-26T04:06:19.744 INFO:teuthology.orchestra.run.smithi186.stdout:Source: linux-meta 2024-04-26T04:06:19.744 INFO:teuthology.orchestra.run.smithi186.stdout:Version: 5.15.0.105.102 2024-04-26T04:06:19.744 INFO:teuthology.orchestra.run.smithi186.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-26T04:06:19.744 INFO:teuthology.orchestra.run.smithi186.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-26T04:06:19.744 INFO:teuthology.orchestra.run.smithi186.stdout:Recommends: thermald 2024-04-26T04:06:19.744 INFO:teuthology.orchestra.run.smithi186.stdout:Description: Generic Linux kernel image 2024-04-26T04:06:19.744 INFO:teuthology.orchestra.run.smithi186.stdout: This package will always depend on the latest generic kernel image 2024-04-26T04:06:19.744 INFO:teuthology.orchestra.run.smithi186.stdout: available. 2024-04-26T04:06:19.745 DEBUG:teuthology.orchestra.run.smithi186:> mktemp 2024-04-26T04:06:19.789 INFO:teuthology.orchestra.run.smithi186.stdout:/tmp/tmp.ig4lZeYLNH 2024-04-26T04:06:19.790 DEBUG:teuthology.orchestra.run.smithi186:> sudo cp /boot/grub/grub.cfg /tmp/tmp.ig4lZeYLNH 2024-04-26T04:06:19.797 INFO:teuthology.orchestra.run.smithi063.stderr:Sourcing file `/etc/default/grub' 2024-04-26T04:06:19.798 INFO:teuthology.orchestra.run.smithi063.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-26T04:06:19.800 INFO:teuthology.orchestra.run.smithi063.stderr:Generating grub configuration file ... 2024-04-26T04:06:19.846 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 0666 /tmp/tmp.ig4lZeYLNH 2024-04-26T04:06:19.971 DEBUG:teuthology.orchestra.remote:smithi186:/tmp/tmp.ig4lZeYLNH is 10KB 2024-04-26T04:06:19.984 DEBUG:teuthology.orchestra.run.smithi186:> rm -fr /tmp/tmp.ig4lZeYLNH 2024-04-26T04:06:19.989 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-26T04:06:20.042 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-26T04:06:20.042 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-26T04:06:20.042 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-26T04:06:20.059 INFO:teuthology.orchestra.run.smithi063.stderr:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-26T04:06:20.064 INFO:teuthology.orchestra.run.smithi063.stderr:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-26T04:06:20.134 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-105-generic 2024-04-26T04:06:20.134 DEBUG:teuthology.orchestra.run.smithi186:> sudo update-grub 2024-04-26T04:06:20.354 INFO:teuthology.orchestra.run.smithi063.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-26T04:06:20.360 INFO:teuthology.orchestra.run.smithi063.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-26T04:06:20.403 INFO:teuthology.orchestra.run.smithi063.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-26T04:06:20.408 INFO:teuthology.orchestra.run.smithi063.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-26T04:06:20.525 INFO:teuthology.orchestra.run.smithi063.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-26T04:06:20.525 INFO:teuthology.orchestra.run.smithi063.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-26T04:06:20.525 INFO:teuthology.orchestra.run.smithi063.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-26T04:06:20.543 INFO:teuthology.orchestra.run.smithi063.stderr:done 2024-04-26T04:06:20.545 DEBUG:teuthology.orchestra.run.smithi063:> sudo shutdown -r now 2024-04-26T04:06:21.404 INFO:teuthology.orchestra.run.smithi186.stderr:Sourcing file `/etc/default/grub' 2024-04-26T04:06:21.406 INFO:teuthology.orchestra.run.smithi186.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-26T04:06:21.409 INFO:teuthology.orchestra.run.smithi186.stderr:Generating grub configuration file ... 2024-04-26T04:06:21.631 INFO:teuthology.orchestra.run.smithi186.stderr:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-26T04:06:21.638 INFO:teuthology.orchestra.run.smithi186.stderr:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-26T04:06:21.883 INFO:teuthology.orchestra.run.smithi186.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-26T04:06:21.889 INFO:teuthology.orchestra.run.smithi186.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-26T04:06:21.929 INFO:teuthology.orchestra.run.smithi186.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-26T04:06:21.934 INFO:teuthology.orchestra.run.smithi186.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-26T04:06:22.050 INFO:teuthology.orchestra.run.smithi186.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-26T04:06:22.050 INFO:teuthology.orchestra.run.smithi186.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-26T04:06:22.050 INFO:teuthology.orchestra.run.smithi186.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-26T04:06:22.084 INFO:teuthology.orchestra.run.smithi186.stderr:done 2024-04-26T04:06:22.093 DEBUG:teuthology.orchestra.run.smithi186:> sudo shutdown -r now 2024-04-26T04:06:50.574 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-26T04:06:50.575 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-04-26T04:06:50.576 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T04:06:52.097 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-04-26T04:06:52.098 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-04-26T04:06:52.098 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T04:07:09.091 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.63 2024-04-26T04:07:18.100 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-04-26T04:07:18.101 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T04:07:21.185 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.63 2024-04-26T04:07:26.750 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-04-26T04:07:33.190 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-04-26T04:07:33.191 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T04:07:35.754 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-04-26T04:07:35.755 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T04:07:51.583 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.63 2024-04-26T04:08:06.598 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-04-26T04:08:06.599 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T04:08:07.615 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.63 2024-04-26T04:08:25.632 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-04-26T04:08:25.633 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T04:08:25.855 DEBUG:teuthology.orchestra.run.smithi063:> true 2024-04-26T04:08:26.425 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-04-26T04:08:26.425 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-105-generic"... 2024-04-26T04:08:26.425 DEBUG:teuthology.orchestra.run.smithi063:> uname -r 2024-04-26T04:08:26.470 INFO:teuthology.orchestra.run.smithi063.stdout:5.15.0-105-generic 2024-04-26T04:08:26.470 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-105-generic vs 5.15.0-105-generic 2024-04-26T04:08:26.470 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-26T04:08:26.470 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-26T04:08:27.472 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-26T04:08:27.472 DEBUG:teuthology.orchestra.run.smithi063:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-26T04:08:27.539 INFO:teuthology.orchestra.run.smithi063.stdout:ttyS1 2024-04-26T04:08:27.564 DEBUG:teuthology.parallel:result is None 2024-04-26T04:08:35.757 DEBUG:teuthology.orchestra.remote:timed out 2024-04-26T04:08:47.759 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-04-26T04:08:47.760 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T04:08:48.051 DEBUG:teuthology.orchestra.run.smithi186:> true 2024-04-26T04:08:48.641 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-04-26T04:08:48.641 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.15.0-105-generic"... 2024-04-26T04:08:48.641 DEBUG:teuthology.orchestra.run.smithi186:> uname -r 2024-04-26T04:08:48.686 INFO:teuthology.orchestra.run.smithi186.stdout:5.15.0-105-generic 2024-04-26T04:08:48.687 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-105-generic vs 5.15.0-105-generic 2024-04-26T04:08:48.687 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-26T04:08:48.687 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-26T04:08:49.687 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-04-26T04:08:49.688 DEBUG:teuthology.orchestra.run.smithi186:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-26T04:08:50.063 INFO:teuthology.orchestra.run.smithi186.stdout:ttyS1 2024-04-26T04:08:50.092 DEBUG:teuthology.parallel:result is None 2024-04-26T04:08:50.093 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-26T04:08:50.100 INFO:teuthology.task.internal:Creating test directory... 2024-04-26T04:08:50.100 DEBUG:teuthology.orchestra.run.smithi063:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-26T04:08:50.103 DEBUG:teuthology.orchestra.run.smithi186:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-26T04:08:50.108 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-26T04:08:50.122 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-26T04:08:50.131 INFO:teuthology.task.internal:Creating archive directory... 2024-04-26T04:08:50.132 DEBUG:teuthology.orchestra.run.smithi063:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-26T04:08:50.151 DEBUG:teuthology.orchestra.run.smithi186:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-26T04:08:50.170 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-26T04:08:50.177 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-26T04:08:50.177 DEBUG:teuthology.orchestra.run.smithi063:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-26T04:08:50.208 DEBUG:teuthology.orchestra.run.smithi186:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-26T04:08:50.227 INFO:teuthology.orchestra.run.smithi063.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T04:08:50.230 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T04:08:50.235 INFO:teuthology.orchestra.run.smithi063.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T04:08:50.239 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-26T04:08:50.240 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-26T04:08:50.247 INFO:teuthology.task.internal:Configuring sudo... 2024-04-26T04:08:50.248 DEBUG:teuthology.orchestra.run.smithi063:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-26T04:08:50.284 DEBUG:teuthology.orchestra.run.smithi186:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-26T04:08:50.295 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-26T04:08:50.305 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-26T04:08:50.305 DEBUG:teuthology.orchestra.run.smithi063:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-26T04:08:50.339 DEBUG:teuthology.orchestra.run.smithi186:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-26T04:08:50.344 DEBUG:teuthology.orchestra.run.smithi063:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T04:08:50.391 DEBUG:teuthology.orchestra.run.smithi063:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T04:08:50.439 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-26T04:08:50.440 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-26T04:08:50.496 DEBUG:teuthology.orchestra.run.smithi186:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-26T04:08:50.502 DEBUG:teuthology.orchestra.run.smithi186:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-26T04:08:50.551 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-26T04:08:50.551 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-26T04:08:50.615 DEBUG:teuthology.orchestra.run.smithi063:> sudo service rsyslog restart 2024-04-26T04:08:50.617 DEBUG:teuthology.orchestra.run.smithi186:> sudo service rsyslog restart 2024-04-26T04:08:50.693 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-26T04:08:50.701 INFO:teuthology.task.internal:Starting timer... 2024-04-26T04:08:50.701 INFO:teuthology.run_tasks:Running task pcp... 2024-04-26T04:08:50.711 INFO:teuthology.run_tasks:Running task selinux... 2024-04-26T04:08:50.720 DEBUG:teuthology.task.selinux:Excluding smithi063: OS 'ubuntu' does not support SELinux 2024-04-26T04:08:50.720 DEBUG:teuthology.task.selinux:Excluding smithi186: OS 'ubuntu' does not support SELinux 2024-04-26T04:08:50.720 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-26T04:08:50.721 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-04-26T04:08:50.721 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-26T04:08:50.721 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-26T04:08:50.730 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-26T04:08:50.735 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-26T04:08:50.863 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-26T04:08:50.884 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-26T04:08:50.886 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi063.front.sepia.ceph.com,smithi186.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-26T04:15:57.263 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi063.front.sepia.ceph.com'), Remote(name='ubuntu@smithi186.front.sepia.ceph.com')] 2024-04-26T04:15:57.264 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-04-26T04:15:57.265 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T04:15:57.349 DEBUG:teuthology.orchestra.run.smithi063:> true 2024-04-26T04:15:57.431 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-04-26T04:15:57.431 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-04-26T04:15:57.432 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-26T04:15:57.520 DEBUG:teuthology.orchestra.run.smithi186:> true 2024-04-26T04:15:57.594 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-04-26T04:15:57.595 INFO:teuthology.run_tasks:Running task clock... 2024-04-26T04:15:57.606 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-26T04:15:57.606 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-26T04:15:57.607 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T04:15:57.609 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-26T04:15:57.610 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T04:15:57.636 INFO:teuthology.orchestra.run.smithi063.stdout:26 Apr 04:15:57 ntpd[17203]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-26T04:15:57.636 INFO:teuthology.orchestra.run.smithi063.stdout:26 Apr 04:15:57 ntpd[17203]: Command line: ntpd -gq 2024-04-26T04:15:57.636 INFO:teuthology.orchestra.run.smithi063.stdout:26 Apr 04:15:57 ntpd[17203]: ---------------------------------------------------- 2024-04-26T04:15:57.636 INFO:teuthology.orchestra.run.smithi063.stdout:26 Apr 04:15:57 ntpd[17203]: ntp-4 is maintained by Network Time Foundation, 2024-04-26T04:15:57.636 INFO:teuthology.orchestra.run.smithi063.stdout:26 Apr 04:15:57 ntpd[17203]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-26T04:15:57.636 INFO:teuthology.orchestra.run.smithi063.stdout:26 Apr 04:15:57 ntpd[17203]: corporation. Support and training for ntp-4 are 2024-04-26T04:15:57.636 INFO:teuthology.orchestra.run.smithi063.stdout:26 Apr 04:15:57 ntpd[17203]: available at https://www.nwtime.org/support 2024-04-26T04:15:57.636 INFO:teuthology.orchestra.run.smithi063.stdout:26 Apr 04:15:57 ntpd[17203]: ---------------------------------------------------- 2024-04-26T04:15:57.636 INFO:teuthology.orchestra.run.smithi063.stdout:26 Apr 04:15:57 ntpd[17203]: proto: precision = 0.047 usec (-24) 2024-04-26T04:15:57.636 INFO:teuthology.orchestra.run.smithi063.stdout:26 Apr 04:15:57 ntpd[17203]: basedate set to 2022-02-04 2024-04-26T04:15:57.636 INFO:teuthology.orchestra.run.smithi063.stdout:26 Apr 04:15:57 ntpd[17203]: gps base set to 2022-02-06 (week 2196) 2024-04-26T04:15:57.637 INFO:teuthology.orchestra.run.smithi063.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-26T04:15:57.638 INFO:teuthology.orchestra.run.smithi063.stdout:26 Apr 04:15:57 ntpd[17203]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-26T04:15:57.638 INFO:teuthology.orchestra.run.smithi063.stdout:26 Apr 04:15:57 ntpd[17203]: restrict ::: KOD does nothing without LIMITED. 2024-04-26T04:15:57.638 INFO:teuthology.orchestra.run.smithi063.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-26T04:15:57.640 INFO:teuthology.orchestra.run.smithi063.stdout:26 Apr 04:15:57 ntpd[17203]: Listen and drop on 0 v6wildcard [::]:123 2024-04-26T04:15:57.640 INFO:teuthology.orchestra.run.smithi063.stdout:26 Apr 04:15:57 ntpd[17203]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-26T04:15:57.640 INFO:teuthology.orchestra.run.smithi063.stdout:26 Apr 04:15:57 ntpd[17203]: Listen normally on 2 lo 127.0.0.1:123 2024-04-26T04:15:57.640 INFO:teuthology.orchestra.run.smithi063.stdout:26 Apr 04:15:57 ntpd[17203]: Listen normally on 3 enp3s0f1 172.21.15.63:123 2024-04-26T04:15:57.640 INFO:teuthology.orchestra.run.smithi063.stdout:26 Apr 04:15:57 ntpd[17203]: Listen normally on 4 lo [::1]:123 2024-04-26T04:15:57.640 INFO:teuthology.orchestra.run.smithi063.stdout:26 Apr 04:15:57 ntpd[17203]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:feb6:f8d7%5]:123 2024-04-26T04:15:57.640 INFO:teuthology.orchestra.run.smithi063.stdout:26 Apr 04:15:57 ntpd[17203]: Listening on routing socket on fd #22 for interface updates 2024-04-26T04:15:57.667 INFO:teuthology.orchestra.run.smithi186.stdout:26 Apr 04:15:57 ntpd[17208]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-26T04:15:57.667 INFO:teuthology.orchestra.run.smithi186.stdout:26 Apr 04:15:57 ntpd[17208]: Command line: ntpd -gq 2024-04-26T04:15:57.668 INFO:teuthology.orchestra.run.smithi186.stdout:26 Apr 04:15:57 ntpd[17208]: ---------------------------------------------------- 2024-04-26T04:15:57.668 INFO:teuthology.orchestra.run.smithi186.stdout:26 Apr 04:15:57 ntpd[17208]: ntp-4 is maintained by Network Time Foundation, 2024-04-26T04:15:57.668 INFO:teuthology.orchestra.run.smithi186.stdout:26 Apr 04:15:57 ntpd[17208]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-26T04:15:57.668 INFO:teuthology.orchestra.run.smithi186.stdout:26 Apr 04:15:57 ntpd[17208]: corporation. Support and training for ntp-4 are 2024-04-26T04:15:57.668 INFO:teuthology.orchestra.run.smithi186.stdout:26 Apr 04:15:57 ntpd[17208]: available at https://www.nwtime.org/support 2024-04-26T04:15:57.668 INFO:teuthology.orchestra.run.smithi186.stdout:26 Apr 04:15:57 ntpd[17208]: ---------------------------------------------------- 2024-04-26T04:15:57.669 INFO:teuthology.orchestra.run.smithi186.stdout:26 Apr 04:15:57 ntpd[17208]: proto: precision = 0.043 usec (-24) 2024-04-26T04:15:57.669 INFO:teuthology.orchestra.run.smithi186.stdout:26 Apr 04:15:57 ntpd[17208]: basedate set to 2022-02-04 2024-04-26T04:15:57.669 INFO:teuthology.orchestra.run.smithi186.stdout:26 Apr 04:15:57 ntpd[17208]: gps base set to 2022-02-06 (week 2196) 2024-04-26T04:15:57.669 INFO:teuthology.orchestra.run.smithi186.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-26T04:15:57.670 INFO:teuthology.orchestra.run.smithi186.stdout:26 Apr 04:15:57 ntpd[17208]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-26T04:15:57.670 INFO:teuthology.orchestra.run.smithi186.stdout:26 Apr 04:15:57 ntpd[17208]: restrict ::: KOD does nothing without LIMITED. 2024-04-26T04:15:57.671 INFO:teuthology.orchestra.run.smithi186.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-26T04:15:57.672 INFO:teuthology.orchestra.run.smithi186.stdout:26 Apr 04:15:57 ntpd[17208]: Listen and drop on 0 v6wildcard [::]:123 2024-04-26T04:15:57.673 INFO:teuthology.orchestra.run.smithi186.stdout:26 Apr 04:15:57 ntpd[17208]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-26T04:15:57.673 INFO:teuthology.orchestra.run.smithi186.stdout:26 Apr 04:15:57 ntpd[17208]: Listen normally on 2 lo 127.0.0.1:123 2024-04-26T04:15:57.673 INFO:teuthology.orchestra.run.smithi186.stdout:26 Apr 04:15:57 ntpd[17208]: Listen normally on 3 enp3s0f1 172.21.15.186:123 2024-04-26T04:15:57.673 INFO:teuthology.orchestra.run.smithi186.stdout:26 Apr 04:15:57 ntpd[17208]: Listen normally on 4 lo [::1]:123 2024-04-26T04:15:57.673 INFO:teuthology.orchestra.run.smithi186.stdout:26 Apr 04:15:57 ntpd[17208]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:6df5%5]:123 2024-04-26T04:15:57.673 INFO:teuthology.orchestra.run.smithi186.stdout:26 Apr 04:15:57 ntpd[17208]: Listening on routing socket on fd #22 for interface updates 2024-04-26T04:15:58.636 INFO:teuthology.orchestra.run.smithi063.stderr:26 Apr 04:15:58 ntpd[17203]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T04:15:58.636 INFO:teuthology.orchestra.run.smithi063.stderr:26 Apr 04:15:58 ntpd[17203]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T04:15:58.636 INFO:teuthology.orchestra.run.smithi063.stderr:26 Apr 04:15:58 ntpd[17203]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T04:15:58.636 INFO:teuthology.orchestra.run.smithi063.stderr:26 Apr 04:15:58 ntpd[17203]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T04:15:58.636 INFO:teuthology.orchestra.run.smithi063.stderr:26 Apr 04:15:58 ntpd[17203]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T04:15:58.637 INFO:teuthology.orchestra.run.smithi063.stderr:26 Apr 04:15:58 ntpd[17203]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T04:15:59.667 INFO:teuthology.orchestra.run.smithi186.stderr:26 Apr 04:15:59 ntpd[17208]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T04:15:59.667 INFO:teuthology.orchestra.run.smithi186.stderr:26 Apr 04:15:59 ntpd[17208]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T04:15:59.667 INFO:teuthology.orchestra.run.smithi186.stderr:26 Apr 04:15:59 ntpd[17208]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T04:15:59.667 INFO:teuthology.orchestra.run.smithi186.stderr:26 Apr 04:15:59 ntpd[17208]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T04:15:59.667 INFO:teuthology.orchestra.run.smithi186.stderr:26 Apr 04:15:59 ntpd[17208]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T04:15:59.667 INFO:teuthology.orchestra.run.smithi186.stderr:26 Apr 04:15:59 ntpd[17208]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T04:16:00.635 INFO:teuthology.orchestra.run.smithi063.stderr:26 Apr 04:16:00 ntpd[17203]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T04:16:00.636 INFO:teuthology.orchestra.run.smithi063.stderr:26 Apr 04:16:00 ntpd[17203]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T04:16:00.636 INFO:teuthology.orchestra.run.smithi063.stderr:26 Apr 04:16:00 ntpd[17203]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T04:16:00.636 INFO:teuthology.orchestra.run.smithi063.stderr:26 Apr 04:16:00 ntpd[17203]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T04:16:00.636 INFO:teuthology.orchestra.run.smithi063.stderr:26 Apr 04:16:00 ntpd[17203]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T04:16:00.636 INFO:teuthology.orchestra.run.smithi063.stderr:26 Apr 04:16:00 ntpd[17203]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T04:16:00.636 INFO:teuthology.orchestra.run.smithi063.stderr:26 Apr 04:16:00 ntpd[17203]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T04:16:00.636 INFO:teuthology.orchestra.run.smithi063.stderr:26 Apr 04:16:00 ntpd[17203]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T04:16:01.635 INFO:teuthology.orchestra.run.smithi063.stderr:26 Apr 04:16:01 ntpd[17203]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T04:16:01.635 INFO:teuthology.orchestra.run.smithi063.stderr:26 Apr 04:16:01 ntpd[17203]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T04:16:01.635 INFO:teuthology.orchestra.run.smithi063.stderr:26 Apr 04:16:01 ntpd[17203]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T04:16:01.635 INFO:teuthology.orchestra.run.smithi063.stderr:26 Apr 04:16:01 ntpd[17203]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T04:16:01.635 INFO:teuthology.orchestra.run.smithi063.stderr:26 Apr 04:16:01 ntpd[17203]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T04:16:01.636 INFO:teuthology.orchestra.run.smithi063.stderr:26 Apr 04:16:01 ntpd[17203]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T04:16:01.667 INFO:teuthology.orchestra.run.smithi186.stderr:26 Apr 04:16:01 ntpd[17208]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T04:16:01.667 INFO:teuthology.orchestra.run.smithi186.stderr:26 Apr 04:16:01 ntpd[17208]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T04:16:01.667 INFO:teuthology.orchestra.run.smithi186.stderr:26 Apr 04:16:01 ntpd[17208]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T04:16:01.667 INFO:teuthology.orchestra.run.smithi186.stderr:26 Apr 04:16:01 ntpd[17208]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T04:16:01.667 INFO:teuthology.orchestra.run.smithi186.stderr:26 Apr 04:16:01 ntpd[17208]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T04:16:01.667 INFO:teuthology.orchestra.run.smithi186.stderr:26 Apr 04:16:01 ntpd[17208]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T04:16:02.637 INFO:teuthology.orchestra.run.smithi063.stderr:26 Apr 04:16:02 ntpd[17203]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T04:16:02.637 INFO:teuthology.orchestra.run.smithi063.stderr:26 Apr 04:16:02 ntpd[17203]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T04:16:02.637 INFO:teuthology.orchestra.run.smithi063.stderr:26 Apr 04:16:02 ntpd[17203]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T04:16:02.637 INFO:teuthology.orchestra.run.smithi063.stderr:26 Apr 04:16:02 ntpd[17203]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T04:16:02.638 INFO:teuthology.orchestra.run.smithi063.stderr:26 Apr 04:16:02 ntpd[17203]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T04:16:02.638 INFO:teuthology.orchestra.run.smithi063.stderr:26 Apr 04:16:02 ntpd[17203]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T04:16:02.638 INFO:teuthology.orchestra.run.smithi063.stderr:26 Apr 04:16:02 ntpd[17203]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T04:16:02.638 INFO:teuthology.orchestra.run.smithi063.stderr:26 Apr 04:16:02 ntpd[17203]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T04:16:02.638 INFO:teuthology.orchestra.run.smithi063.stderr:26 Apr 04:16:02 ntpd[17203]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T04:16:02.638 INFO:teuthology.orchestra.run.smithi063.stderr:26 Apr 04:16:02 ntpd[17203]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T04:16:02.639 INFO:teuthology.orchestra.run.smithi063.stderr:26 Apr 04:16:02 ntpd[17203]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T04:16:02.639 INFO:teuthology.orchestra.run.smithi063.stderr:26 Apr 04:16:02 ntpd[17203]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T04:16:02.667 INFO:teuthology.orchestra.run.smithi186.stderr:26 Apr 04:16:02 ntpd[17208]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T04:16:02.667 INFO:teuthology.orchestra.run.smithi186.stderr:26 Apr 04:16:02 ntpd[17208]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T04:16:02.667 INFO:teuthology.orchestra.run.smithi186.stderr:26 Apr 04:16:02 ntpd[17208]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T04:16:02.668 INFO:teuthology.orchestra.run.smithi186.stderr:26 Apr 04:16:02 ntpd[17208]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T04:16:02.668 INFO:teuthology.orchestra.run.smithi186.stderr:26 Apr 04:16:02 ntpd[17208]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T04:16:02.668 INFO:teuthology.orchestra.run.smithi186.stderr:26 Apr 04:16:02 ntpd[17208]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T04:16:02.668 INFO:teuthology.orchestra.run.smithi186.stderr:26 Apr 04:16:02 ntpd[17208]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T04:16:02.668 INFO:teuthology.orchestra.run.smithi186.stderr:26 Apr 04:16:02 ntpd[17208]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T04:16:02.668 INFO:teuthology.orchestra.run.smithi186.stderr:26 Apr 04:16:02 ntpd[17208]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T04:16:02.668 INFO:teuthology.orchestra.run.smithi186.stderr:26 Apr 04:16:02 ntpd[17208]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T04:16:02.668 INFO:teuthology.orchestra.run.smithi186.stderr:26 Apr 04:16:02 ntpd[17208]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T04:16:02.668 INFO:teuthology.orchestra.run.smithi186.stderr:26 Apr 04:16:02 ntpd[17208]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T04:16:02.668 INFO:teuthology.orchestra.run.smithi186.stderr:26 Apr 04:16:02 ntpd[17208]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T04:16:02.668 INFO:teuthology.orchestra.run.smithi186.stderr:26 Apr 04:16:02 ntpd[17208]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T04:16:03.635 INFO:teuthology.orchestra.run.smithi063.stderr:26 Apr 04:16:03 ntpd[17203]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T04:16:03.635 INFO:teuthology.orchestra.run.smithi063.stderr:26 Apr 04:16:03 ntpd[17203]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T04:16:03.635 INFO:teuthology.orchestra.run.smithi063.stderr:26 Apr 04:16:03 ntpd[17203]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T04:16:03.635 INFO:teuthology.orchestra.run.smithi063.stderr:26 Apr 04:16:03 ntpd[17203]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T04:16:03.635 INFO:teuthology.orchestra.run.smithi063.stderr:26 Apr 04:16:03 ntpd[17203]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T04:16:03.636 INFO:teuthology.orchestra.run.smithi063.stderr:26 Apr 04:16:03 ntpd[17203]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T04:16:03.667 INFO:teuthology.orchestra.run.smithi186.stderr:26 Apr 04:16:03 ntpd[17208]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T04:16:03.667 INFO:teuthology.orchestra.run.smithi186.stderr:26 Apr 04:16:03 ntpd[17208]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T04:16:03.667 INFO:teuthology.orchestra.run.smithi186.stderr:26 Apr 04:16:03 ntpd[17208]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T04:16:03.667 INFO:teuthology.orchestra.run.smithi186.stderr:26 Apr 04:16:03 ntpd[17208]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T04:16:03.667 INFO:teuthology.orchestra.run.smithi186.stderr:26 Apr 04:16:03 ntpd[17208]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T04:16:03.667 INFO:teuthology.orchestra.run.smithi186.stderr:26 Apr 04:16:03 ntpd[17208]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T04:16:04.635 INFO:teuthology.orchestra.run.smithi063.stderr:26 Apr 04:16:04 ntpd[17203]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T04:16:04.636 INFO:teuthology.orchestra.run.smithi063.stderr:26 Apr 04:16:04 ntpd[17203]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T04:16:04.636 INFO:teuthology.orchestra.run.smithi063.stderr:26 Apr 04:16:04 ntpd[17203]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T04:16:04.636 INFO:teuthology.orchestra.run.smithi063.stderr:26 Apr 04:16:04 ntpd[17203]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T04:16:04.636 INFO:teuthology.orchestra.run.smithi063.stderr:26 Apr 04:16:04 ntpd[17203]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T04:16:04.636 INFO:teuthology.orchestra.run.smithi063.stderr:26 Apr 04:16:04 ntpd[17203]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T04:16:04.636 INFO:teuthology.orchestra.run.smithi063.stderr:26 Apr 04:16:04 ntpd[17203]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T04:16:04.636 INFO:teuthology.orchestra.run.smithi063.stderr:26 Apr 04:16:04 ntpd[17203]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T04:16:04.636 INFO:teuthology.orchestra.run.smithi063.stdout:26 Apr 04:16:04 ntpd[17203]: ntpd: time slew +0.001115 s 2024-04-26T04:16:04.636 INFO:teuthology.orchestra.run.smithi063.stdout:ntpd: time slew +0.001115s 2024-04-26T04:16:04.637 INFO:teuthology.orchestra.run.smithi063.stderr:26 Apr 04:16:04 ntpd[17203]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-26T04:16:04.637 INFO:teuthology.orchestra.run.smithi063.stderr:26 Apr 04:16:04 ntpd[17203]: can't open /var/log/ntpstats/loopstats.20240426: Permission denied 2024-04-26T04:16:04.667 INFO:teuthology.orchestra.run.smithi186.stderr:26 Apr 04:16:04 ntpd[17208]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T04:16:04.667 INFO:teuthology.orchestra.run.smithi186.stderr:26 Apr 04:16:04 ntpd[17208]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T04:16:04.667 INFO:teuthology.orchestra.run.smithi186.stderr:26 Apr 04:16:04 ntpd[17208]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T04:16:04.667 INFO:teuthology.orchestra.run.smithi186.stderr:26 Apr 04:16:04 ntpd[17208]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T04:16:04.667 INFO:teuthology.orchestra.run.smithi186.stderr:26 Apr 04:16:04 ntpd[17208]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T04:16:04.668 INFO:teuthology.orchestra.run.smithi186.stderr:26 Apr 04:16:04 ntpd[17208]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T04:16:04.668 INFO:teuthology.orchestra.run.smithi186.stderr:26 Apr 04:16:04 ntpd[17208]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T04:16:04.668 INFO:teuthology.orchestra.run.smithi186.stderr:26 Apr 04:16:04 ntpd[17208]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T04:16:04.668 INFO:teuthology.orchestra.run.smithi186.stderr:26 Apr 04:16:04 ntpd[17208]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T04:16:04.668 INFO:teuthology.orchestra.run.smithi186.stderr:26 Apr 04:16:04 ntpd[17208]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T04:16:04.668 INFO:teuthology.orchestra.run.smithi186.stderr:26 Apr 04:16:04 ntpd[17208]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T04:16:04.668 INFO:teuthology.orchestra.run.smithi186.stderr:26 Apr 04:16:04 ntpd[17208]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T04:16:04.685 INFO:teuthology.orchestra.run.smithi063.stdout: remote refid st t when poll reach delay offset jitter 2024-04-26T04:16:04.686 INFO:teuthology.orchestra.run.smithi063.stdout:============================================================================== 2024-04-26T04:16:04.686 INFO:teuthology.orchestra.run.smithi063.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T04:16:04.686 INFO:teuthology.orchestra.run.smithi063.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T04:16:04.686 INFO:teuthology.orchestra.run.smithi063.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T04:16:04.686 INFO:teuthology.orchestra.run.smithi063.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T04:16:05.667 INFO:teuthology.orchestra.run.smithi186.stderr:26 Apr 04:16:05 ntpd[17208]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T04:16:05.667 INFO:teuthology.orchestra.run.smithi186.stderr:26 Apr 04:16:05 ntpd[17208]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T04:16:05.667 INFO:teuthology.orchestra.run.smithi186.stderr:26 Apr 04:16:05 ntpd[17208]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-26T04:16:05.667 INFO:teuthology.orchestra.run.smithi186.stderr:26 Apr 04:16:05 ntpd[17208]: can't open /var/log/ntpstats/rawstats.20240426: Permission denied 2024-04-26T04:16:05.667 INFO:teuthology.orchestra.run.smithi186.stderr:26 Apr 04:16:05 ntpd[17208]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-26T04:16:05.667 INFO:teuthology.orchestra.run.smithi186.stderr:26 Apr 04:16:05 ntpd[17208]: can't open /var/log/ntpstats/peerstats.20240426: Permission denied 2024-04-26T04:16:05.668 INFO:teuthology.orchestra.run.smithi186.stdout:26 Apr 04:16:05 ntpd[17208]: ntpd: time slew +0.000357 s 2024-04-26T04:16:05.668 INFO:teuthology.orchestra.run.smithi186.stdout:ntpd: time slew +0.000357s 2024-04-26T04:16:05.668 INFO:teuthology.orchestra.run.smithi186.stderr:26 Apr 04:16:05 ntpd[17208]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-26T04:16:05.668 INFO:teuthology.orchestra.run.smithi186.stderr:26 Apr 04:16:05 ntpd[17208]: can't open /var/log/ntpstats/loopstats.20240426: Permission denied 2024-04-26T04:16:05.726 INFO:teuthology.orchestra.run.smithi186.stdout: remote refid st t when poll reach delay offset jitter 2024-04-26T04:16:05.726 INFO:teuthology.orchestra.run.smithi186.stdout:============================================================================== 2024-04-26T04:16:05.726 INFO:teuthology.orchestra.run.smithi186.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T04:16:05.727 INFO:teuthology.orchestra.run.smithi186.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T04:16:05.727 INFO:teuthology.orchestra.run.smithi186.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T04:16:05.727 INFO:teuthology.orchestra.run.smithi186.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-26T04:16:05.727 INFO:teuthology.run_tasks:Running task install... 2024-04-26T04:16:05.735 DEBUG:teuthology.task.install:project ceph 2024-04-26T04:16:05.736 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'ddeaa91797339a19172c3036ff48cca58c12f448'}} 2024-04-26T04:16:05.736 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'ddeaa91797339a19172c3036ff48cca58c12f448'} 2024-04-26T04:16:05.736 INFO:teuthology.task.install:Using flavor: default 2024-04-26T04:16:05.742 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-26T04:16:05.742 INFO:teuthology.task.install:extra packages: [] 2024-04-26T04:16:05.742 DEBUG:teuthology.orchestra.run.smithi063:> sudo apt-key list | grep Ceph 2024-04-26T04:16:05.743 DEBUG:teuthology.orchestra.run.smithi186:> sudo apt-key list | grep Ceph 2024-04-26T04:16:05.813 INFO:teuthology.orchestra.run.smithi063.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-26T04:16:05.821 INFO:teuthology.orchestra.run.smithi186.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-26T04:16:05.846 INFO:teuthology.orchestra.run.smithi063.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-26T04:16:05.846 INFO:teuthology.orchestra.run.smithi063.stdout:uid [ unknown] Ceph.com (release key) 2024-04-26T04:16:05.846 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-04-26T04:16:05.847 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=ddeaa91797339a19172c3036ff48cca58c12f448 2024-04-26T04:16:05.852 INFO:teuthology.orchestra.run.smithi186.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-26T04:16:05.852 INFO:teuthology.orchestra.run.smithi186.stdout:uid [ unknown] Ceph.com (release key) 2024-04-26T04:16:05.852 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-04-26T04:16:05.852 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=ddeaa91797339a19172c3036ff48cca58c12f448 2024-04-26T04:16:06.072 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default/ 2024-04-26T04:16:06.072 INFO:teuthology.task.install.deb:Package version is 19.0.0-2457-gddeaa917-1jammy 2024-04-26T04:16:06.076 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default/ 2024-04-26T04:16:06.076 INFO:teuthology.task.install.deb:Package version is 19.0.0-2457-gddeaa917-1jammy 2024-04-26T04:16:06.191 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-26T04:16:06.191 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-26T04:16:06.205 DEBUG:teuthology.orchestra.run.smithi063:> sudo apt-get update 2024-04-26T04:16:06.241 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-26T04:16:06.241 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-26T04:16:06.255 DEBUG:teuthology.orchestra.run.smithi186:> sudo apt-get update 2024-04-26T04:16:06.377 INFO:teuthology.orchestra.run.smithi063.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-26T04:16:06.473 INFO:teuthology.orchestra.run.smithi186.stdout:Ign:1 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy InRelease 2024-04-26T04:16:06.490 INFO:teuthology.orchestra.run.smithi063.stdout:Ign:2 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy InRelease 2024-04-26T04:16:06.492 INFO:teuthology.orchestra.run.smithi063.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-26T04:16:06.494 INFO:teuthology.orchestra.run.smithi186.stdout:Get:2 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-04-26T04:16:06.517 INFO:teuthology.orchestra.run.smithi186.stdout:Ign:3 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-26T04:16:06.539 INFO:teuthology.orchestra.run.smithi063.stdout:Get:4 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-04-26T04:16:06.548 INFO:teuthology.orchestra.run.smithi186.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-26T04:16:06.548 INFO:teuthology.orchestra.run.smithi186.stdout:Hit:5 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-26T04:16:06.548 INFO:teuthology.orchestra.run.smithi186.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,421 B] 2024-04-26T04:16:06.577 INFO:teuthology.orchestra.run.smithi063.stdout:Hit:5 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-26T04:16:06.577 INFO:teuthology.orchestra.run.smithi063.stdout:Ign:6 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-26T04:16:06.579 INFO:teuthology.orchestra.run.smithi186.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-04-26T04:16:06.614 INFO:teuthology.orchestra.run.smithi063.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-04-26T04:16:06.620 INFO:teuthology.orchestra.run.smithi186.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-26T04:16:06.656 INFO:teuthology.orchestra.run.smithi063.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,421 B] 2024-04-26T04:16:06.658 INFO:teuthology.orchestra.run.smithi063.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-26T04:16:06.702 INFO:teuthology.orchestra.run.smithi186.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-26T04:16:06.799 INFO:teuthology.orchestra.run.smithi063.stdout:Fetched 28.9 kB in 0s (70.0 kB/s) 2024-04-26T04:16:06.923 INFO:teuthology.orchestra.run.smithi186.stdout:Fetched 28.9 kB in 0s (59.1 kB/s) 2024-04-26T04:16:07.806 INFO:teuthology.orchestra.run.smithi063.stdout:Reading package lists... 2024-04-26T04:16:07.822 DEBUG:teuthology.orchestra.run.smithi063:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-2457-gddeaa917-1jammy cephadm=19.0.0-2457-gddeaa917-1jammy ceph-mds=19.0.0-2457-gddeaa917-1jammy ceph-mgr=19.0.0-2457-gddeaa917-1jammy ceph-common=19.0.0-2457-gddeaa917-1jammy ceph-fuse=19.0.0-2457-gddeaa917-1jammy ceph-test=19.0.0-2457-gddeaa917-1jammy ceph-volume=19.0.0-2457-gddeaa917-1jammy radosgw=19.0.0-2457-gddeaa917-1jammy python3-rados=19.0.0-2457-gddeaa917-1jammy python3-rgw=19.0.0-2457-gddeaa917-1jammy python3-cephfs=19.0.0-2457-gddeaa917-1jammy python3-rbd=19.0.0-2457-gddeaa917-1jammy libcephfs2=19.0.0-2457-gddeaa917-1jammy libcephfs-dev=19.0.0-2457-gddeaa917-1jammy librados2=19.0.0-2457-gddeaa917-1jammy librbd1=19.0.0-2457-gddeaa917-1jammy rbd-fuse=19.0.0-2457-gddeaa917-1jammy 2024-04-26T04:16:07.876 INFO:teuthology.orchestra.run.smithi063.stdout:Reading package lists... 2024-04-26T04:16:07.920 INFO:teuthology.orchestra.run.smithi186.stdout:Reading package lists... 2024-04-26T04:16:07.937 DEBUG:teuthology.orchestra.run.smithi186:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-2457-gddeaa917-1jammy cephadm=19.0.0-2457-gddeaa917-1jammy ceph-mds=19.0.0-2457-gddeaa917-1jammy ceph-mgr=19.0.0-2457-gddeaa917-1jammy ceph-common=19.0.0-2457-gddeaa917-1jammy ceph-fuse=19.0.0-2457-gddeaa917-1jammy ceph-test=19.0.0-2457-gddeaa917-1jammy ceph-volume=19.0.0-2457-gddeaa917-1jammy radosgw=19.0.0-2457-gddeaa917-1jammy python3-rados=19.0.0-2457-gddeaa917-1jammy python3-rgw=19.0.0-2457-gddeaa917-1jammy python3-cephfs=19.0.0-2457-gddeaa917-1jammy python3-rbd=19.0.0-2457-gddeaa917-1jammy libcephfs2=19.0.0-2457-gddeaa917-1jammy libcephfs-dev=19.0.0-2457-gddeaa917-1jammy librados2=19.0.0-2457-gddeaa917-1jammy librbd1=19.0.0-2457-gddeaa917-1jammy rbd-fuse=19.0.0-2457-gddeaa917-1jammy 2024-04-26T04:16:07.998 INFO:teuthology.orchestra.run.smithi186.stdout:Reading package lists... 2024-04-26T04:16:08.056 INFO:teuthology.orchestra.run.smithi063.stdout:Building dependency tree... 2024-04-26T04:16:08.057 INFO:teuthology.orchestra.run.smithi063.stdout:Reading state information... 2024-04-26T04:16:08.197 INFO:teuthology.orchestra.run.smithi186.stdout:Building dependency tree... 2024-04-26T04:16:08.198 INFO:teuthology.orchestra.run.smithi186.stdout:Reading state information... 2024-04-26T04:16:08.239 INFO:teuthology.orchestra.run.smithi063.stdout:The following packages were automatically installed and are no longer required: 2024-04-26T04:16:08.239 INFO:teuthology.orchestra.run.smithi063.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-26T04:16:08.239 INFO:teuthology.orchestra.run.smithi063.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-26T04:16:08.239 INFO:teuthology.orchestra.run.smithi063.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-26T04:16:08.239 INFO:teuthology.orchestra.run.smithi063.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-26T04:16:08.239 INFO:teuthology.orchestra.run.smithi063.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-26T04:16:08.239 INFO:teuthology.orchestra.run.smithi063.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-26T04:16:08.239 INFO:teuthology.orchestra.run.smithi063.stdout:The following additional packages will be installed: 2024-04-26T04:16:08.240 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-26T04:16:08.240 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-26T04:16:08.240 INFO:teuthology.orchestra.run.smithi063.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-26T04:16:08.240 INFO:teuthology.orchestra.run.smithi063.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-26T04:16:08.240 INFO:teuthology.orchestra.run.smithi063.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-04-26T04:16:08.240 INFO:teuthology.orchestra.run.smithi063.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-04-26T04:16:08.240 INFO:teuthology.orchestra.run.smithi063.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-26T04:16:08.240 INFO:teuthology.orchestra.run.smithi063.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-26T04:16:08.240 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-26T04:16:08.240 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-26T04:16:08.240 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-26T04:16:08.240 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-26T04:16:08.240 INFO:teuthology.orchestra.run.smithi063.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-26T04:16:08.240 INFO:teuthology.orchestra.run.smithi063.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-26T04:16:08.240 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-26T04:16:08.241 INFO:teuthology.orchestra.run.smithi063.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-26T04:16:08.241 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-04-26T04:16:08.241 INFO:teuthology.orchestra.run.smithi063.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-04-26T04:16:08.241 INFO:teuthology.orchestra.run.smithi063.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-26T04:16:08.241 INFO:teuthology.orchestra.run.smithi063.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-26T04:16:08.241 INFO:teuthology.orchestra.run.smithi063.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-26T04:16:08.241 INFO:teuthology.orchestra.run.smithi063.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-26T04:16:08.241 INFO:teuthology.orchestra.run.smithi063.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-04-26T04:16:08.241 INFO:teuthology.orchestra.run.smithi063.stdout: xmlstarlet zip 2024-04-26T04:16:08.241 INFO:teuthology.orchestra.run.smithi063.stdout:Suggested packages: 2024-04-26T04:16:08.242 INFO:teuthology.orchestra.run.smithi063.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-04-26T04:16:08.242 INFO:teuthology.orchestra.run.smithi063.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-04-26T04:16:08.242 INFO:teuthology.orchestra.run.smithi063.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-04-26T04:16:08.242 INFO:teuthology.orchestra.run.smithi063.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-04-26T04:16:08.242 INFO:teuthology.orchestra.run.smithi063.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-04-26T04:16:08.242 INFO:teuthology.orchestra.run.smithi063.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-04-26T04:16:08.242 INFO:teuthology.orchestra.run.smithi063.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-04-26T04:16:08.243 INFO:teuthology.orchestra.run.smithi063.stdout:Recommended packages: 2024-04-26T04:16:08.243 INFO:teuthology.orchestra.run.smithi063.stdout: btrfs-tools 2024-04-26T04:16:08.342 INFO:teuthology.orchestra.run.smithi063.stdout:The following NEW packages will be installed: 2024-04-26T04:16:08.342 INFO:teuthology.orchestra.run.smithi063.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-26T04:16:08.342 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-26T04:16:08.342 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-04-26T04:16:08.342 INFO:teuthology.orchestra.run.smithi063.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-04-26T04:16:08.342 INFO:teuthology.orchestra.run.smithi063.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-04-26T04:16:08.342 INFO:teuthology.orchestra.run.smithi063.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-04-26T04:16:08.344 INFO:teuthology.orchestra.run.smithi063.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-04-26T04:16:08.344 INFO:teuthology.orchestra.run.smithi063.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-04-26T04:16:08.344 INFO:teuthology.orchestra.run.smithi063.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-04-26T04:16:08.344 INFO:teuthology.orchestra.run.smithi063.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-04-26T04:16:08.344 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-04-26T04:16:08.344 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-26T04:16:08.344 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-26T04:16:08.344 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-26T04:16:08.345 INFO:teuthology.orchestra.run.smithi063.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-26T04:16:08.345 INFO:teuthology.orchestra.run.smithi063.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-26T04:16:08.345 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-26T04:16:08.345 INFO:teuthology.orchestra.run.smithi063.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-26T04:16:08.345 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-04-26T04:16:08.345 INFO:teuthology.orchestra.run.smithi063.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-04-26T04:16:08.345 INFO:teuthology.orchestra.run.smithi063.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-04-26T04:16:08.345 INFO:teuthology.orchestra.run.smithi063.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-04-26T04:16:08.345 INFO:teuthology.orchestra.run.smithi063.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-04-26T04:16:08.345 INFO:teuthology.orchestra.run.smithi063.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-04-26T04:16:08.345 INFO:teuthology.orchestra.run.smithi063.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-04-26T04:16:08.345 INFO:teuthology.orchestra.run.smithi063.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-04-26T04:16:08.382 INFO:teuthology.orchestra.run.smithi186.stdout:The following packages were automatically installed and are no longer required: 2024-04-26T04:16:08.382 INFO:teuthology.orchestra.run.smithi186.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-26T04:16:08.382 INFO:teuthology.orchestra.run.smithi186.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-26T04:16:08.382 INFO:teuthology.orchestra.run.smithi186.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-26T04:16:08.382 INFO:teuthology.orchestra.run.smithi186.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-26T04:16:08.382 INFO:teuthology.orchestra.run.smithi186.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-26T04:16:08.382 INFO:teuthology.orchestra.run.smithi186.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-26T04:16:08.385 INFO:teuthology.orchestra.run.smithi186.stdout:The following additional packages will be installed: 2024-04-26T04:16:08.385 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-26T04:16:08.385 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-26T04:16:08.385 INFO:teuthology.orchestra.run.smithi186.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-26T04:16:08.385 INFO:teuthology.orchestra.run.smithi186.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-26T04:16:08.385 INFO:teuthology.orchestra.run.smithi186.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-04-26T04:16:08.385 INFO:teuthology.orchestra.run.smithi186.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-04-26T04:16:08.386 INFO:teuthology.orchestra.run.smithi186.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-26T04:16:08.386 INFO:teuthology.orchestra.run.smithi186.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-26T04:16:08.386 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-26T04:16:08.386 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-26T04:16:08.386 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-26T04:16:08.386 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-26T04:16:08.386 INFO:teuthology.orchestra.run.smithi186.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-26T04:16:08.386 INFO:teuthology.orchestra.run.smithi186.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-26T04:16:08.386 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-26T04:16:08.386 INFO:teuthology.orchestra.run.smithi186.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-26T04:16:08.386 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-04-26T04:16:08.386 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-04-26T04:16:08.386 INFO:teuthology.orchestra.run.smithi186.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-26T04:16:08.386 INFO:teuthology.orchestra.run.smithi186.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-26T04:16:08.387 INFO:teuthology.orchestra.run.smithi186.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-26T04:16:08.387 INFO:teuthology.orchestra.run.smithi186.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-26T04:16:08.387 INFO:teuthology.orchestra.run.smithi186.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-04-26T04:16:08.387 INFO:teuthology.orchestra.run.smithi186.stdout: xmlstarlet zip 2024-04-26T04:16:08.387 INFO:teuthology.orchestra.run.smithi186.stdout:Suggested packages: 2024-04-26T04:16:08.387 INFO:teuthology.orchestra.run.smithi186.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-04-26T04:16:08.387 INFO:teuthology.orchestra.run.smithi186.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-04-26T04:16:08.387 INFO:teuthology.orchestra.run.smithi186.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-04-26T04:16:08.387 INFO:teuthology.orchestra.run.smithi186.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-04-26T04:16:08.387 INFO:teuthology.orchestra.run.smithi186.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-04-26T04:16:08.387 INFO:teuthology.orchestra.run.smithi186.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-04-26T04:16:08.387 INFO:teuthology.orchestra.run.smithi186.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-04-26T04:16:08.387 INFO:teuthology.orchestra.run.smithi186.stdout:Recommended packages: 2024-04-26T04:16:08.387 INFO:teuthology.orchestra.run.smithi186.stdout: btrfs-tools 2024-04-26T04:16:08.411 INFO:teuthology.orchestra.run.smithi063.stdout:0 upgraded, 118 newly installed, 0 to remove and 302 not upgraded. 2024-04-26T04:16:08.411 INFO:teuthology.orchestra.run.smithi063.stdout:Need to get 179 MB of archives. 2024-04-26T04:16:08.411 INFO:teuthology.orchestra.run.smithi063.stdout:After this operation, 822 MB of additional disk space will be used. 2024-04-26T04:16:08.411 INFO:teuthology.orchestra.run.smithi063.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-26T04:16:08.484 INFO:teuthology.orchestra.run.smithi186.stdout:The following NEW packages will be installed: 2024-04-26T04:16:08.484 INFO:teuthology.orchestra.run.smithi186.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-26T04:16:08.484 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-26T04:16:08.485 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-04-26T04:16:08.485 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-04-26T04:16:08.485 INFO:teuthology.orchestra.run.smithi186.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-04-26T04:16:08.485 INFO:teuthology.orchestra.run.smithi186.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-04-26T04:16:08.486 INFO:teuthology.orchestra.run.smithi186.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-04-26T04:16:08.486 INFO:teuthology.orchestra.run.smithi186.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-04-26T04:16:08.486 INFO:teuthology.orchestra.run.smithi186.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-04-26T04:16:08.486 INFO:teuthology.orchestra.run.smithi186.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-04-26T04:16:08.486 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-04-26T04:16:08.486 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-26T04:16:08.487 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-26T04:16:08.487 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-26T04:16:08.487 INFO:teuthology.orchestra.run.smithi186.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-26T04:16:08.487 INFO:teuthology.orchestra.run.smithi186.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-26T04:16:08.487 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-26T04:16:08.487 INFO:teuthology.orchestra.run.smithi186.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-26T04:16:08.487 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-04-26T04:16:08.487 INFO:teuthology.orchestra.run.smithi186.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-04-26T04:16:08.487 INFO:teuthology.orchestra.run.smithi186.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-04-26T04:16:08.487 INFO:teuthology.orchestra.run.smithi186.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-04-26T04:16:08.487 INFO:teuthology.orchestra.run.smithi186.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-04-26T04:16:08.487 INFO:teuthology.orchestra.run.smithi186.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-04-26T04:16:08.487 INFO:teuthology.orchestra.run.smithi186.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-04-26T04:16:08.487 INFO:teuthology.orchestra.run.smithi186.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-04-26T04:16:08.518 INFO:teuthology.orchestra.run.smithi063.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-26T04:16:08.520 INFO:teuthology.orchestra.run.smithi063.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-26T04:16:08.541 INFO:teuthology.orchestra.run.smithi063.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-26T04:16:08.561 INFO:teuthology.orchestra.run.smithi063.stdout:Get:5 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-2457-gddeaa917-1jammy [3,545 kB] 2024-04-26T04:16:08.609 INFO:teuthology.orchestra.run.smithi063.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-26T04:16:08.613 INFO:teuthology.orchestra.run.smithi063.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-26T04:16:08.626 INFO:teuthology.orchestra.run.smithi063.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-26T04:16:08.631 INFO:teuthology.orchestra.run.smithi063.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-26T04:16:08.631 INFO:teuthology.orchestra.run.smithi063.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-26T04:16:08.632 INFO:teuthology.orchestra.run.smithi063.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-26T04:16:08.642 INFO:teuthology.orchestra.run.smithi063.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-04-26T04:16:08.643 INFO:teuthology.orchestra.run.smithi063.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-04-26T04:16:08.646 INFO:teuthology.orchestra.run.smithi063.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-04-26T04:16:08.648 INFO:teuthology.orchestra.run.smithi063.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-04-26T04:16:08.654 INFO:teuthology.orchestra.run.smithi063.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-04-26T04:16:08.657 INFO:teuthology.orchestra.run.smithi063.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-04-26T04:16:08.659 INFO:teuthology.orchestra.run.smithi063.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-04-26T04:16:08.673 INFO:teuthology.orchestra.run.smithi186.stdout:0 upgraded, 118 newly installed, 0 to remove and 302 not upgraded. 2024-04-26T04:16:08.673 INFO:teuthology.orchestra.run.smithi186.stdout:Need to get 179 MB of archives. 2024-04-26T04:16:08.673 INFO:teuthology.orchestra.run.smithi186.stdout:After this operation, 822 MB of additional disk space will be used. 2024-04-26T04:16:08.673 INFO:teuthology.orchestra.run.smithi186.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-26T04:16:08.675 INFO:teuthology.orchestra.run.smithi063.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-26T04:16:08.675 INFO:teuthology.orchestra.run.smithi063.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-26T04:16:08.676 INFO:teuthology.orchestra.run.smithi063.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-26T04:16:08.677 INFO:teuthology.orchestra.run.smithi063.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-26T04:16:08.677 INFO:teuthology.orchestra.run.smithi063.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-26T04:16:08.678 INFO:teuthology.orchestra.run.smithi063.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-26T04:16:08.678 INFO:teuthology.orchestra.run.smithi063.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-26T04:16:08.695 INFO:teuthology.orchestra.run.smithi063.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-26T04:16:08.696 INFO:teuthology.orchestra.run.smithi063.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-26T04:16:08.696 INFO:teuthology.orchestra.run.smithi063.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-26T04:16:08.703 INFO:teuthology.orchestra.run.smithi186.stdout:Get:2 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-2457-gddeaa917-1jammy [3,545 kB] 2024-04-26T04:16:08.716 INFO:teuthology.orchestra.run.smithi063.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-26T04:16:08.717 INFO:teuthology.orchestra.run.smithi063.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-26T04:16:08.717 INFO:teuthology.orchestra.run.smithi063.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-26T04:16:08.718 INFO:teuthology.orchestra.run.smithi063.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-26T04:16:08.719 INFO:teuthology.orchestra.run.smithi063.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-26T04:16:08.719 INFO:teuthology.orchestra.run.smithi063.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-26T04:16:08.719 INFO:teuthology.orchestra.run.smithi063.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-26T04:16:08.736 INFO:teuthology.orchestra.run.smithi063.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-26T04:16:08.737 INFO:teuthology.orchestra.run.smithi063.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-26T04:16:08.739 INFO:teuthology.orchestra.run.smithi063.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-26T04:16:08.760 INFO:teuthology.orchestra.run.smithi063.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-26T04:16:08.761 INFO:teuthology.orchestra.run.smithi063.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-26T04:16:08.761 INFO:teuthology.orchestra.run.smithi063.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-26T04:16:08.762 INFO:teuthology.orchestra.run.smithi063.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-26T04:16:08.763 INFO:teuthology.orchestra.run.smithi063.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-26T04:16:08.765 INFO:teuthology.orchestra.run.smithi063.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-26T04:16:08.766 INFO:teuthology.orchestra.run.smithi063.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-26T04:16:08.781 INFO:teuthology.orchestra.run.smithi063.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-26T04:16:08.782 INFO:teuthology.orchestra.run.smithi063.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-26T04:16:08.785 INFO:teuthology.orchestra.run.smithi063.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-26T04:16:08.802 INFO:teuthology.orchestra.run.smithi063.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-26T04:16:08.803 INFO:teuthology.orchestra.run.smithi063.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-26T04:16:08.807 INFO:teuthology.orchestra.run.smithi063.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-26T04:16:08.864 INFO:teuthology.orchestra.run.smithi063.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-26T04:16:08.865 INFO:teuthology.orchestra.run.smithi063.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-26T04:16:08.866 INFO:teuthology.orchestra.run.smithi063.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-26T04:16:08.867 INFO:teuthology.orchestra.run.smithi063.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-26T04:16:08.868 INFO:teuthology.orchestra.run.smithi063.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-26T04:16:08.890 INFO:teuthology.orchestra.run.smithi063.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-26T04:16:08.892 INFO:teuthology.orchestra.run.smithi063.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-26T04:16:08.892 INFO:teuthology.orchestra.run.smithi063.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-26T04:16:08.896 INFO:teuthology.orchestra.run.smithi063.stdout:Get:60 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-2457-gddeaa917-1jammy [3,218 kB] 2024-04-26T04:16:08.911 INFO:teuthology.orchestra.run.smithi063.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-26T04:16:08.911 INFO:teuthology.orchestra.run.smithi063.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-26T04:16:08.911 INFO:teuthology.orchestra.run.smithi063.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-26T04:16:08.920 INFO:teuthology.orchestra.run.smithi063.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-26T04:16:08.921 INFO:teuthology.orchestra.run.smithi063.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-26T04:16:08.921 INFO:teuthology.orchestra.run.smithi063.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-26T04:16:08.922 INFO:teuthology.orchestra.run.smithi063.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-26T04:16:08.922 INFO:teuthology.orchestra.run.smithi063.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-26T04:16:08.926 INFO:teuthology.orchestra.run.smithi063.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-26T04:16:08.941 INFO:teuthology.orchestra.run.smithi063.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-26T04:16:08.942 INFO:teuthology.orchestra.run.smithi063.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-26T04:16:08.943 INFO:teuthology.orchestra.run.smithi063.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-26T04:16:08.947 INFO:teuthology.orchestra.run.smithi063.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-26T04:16:08.962 INFO:teuthology.orchestra.run.smithi063.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-04-26T04:16:08.962 INFO:teuthology.orchestra.run.smithi063.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-04-26T04:16:08.963 INFO:teuthology.orchestra.run.smithi063.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-26T04:16:08.968 INFO:teuthology.orchestra.run.smithi063.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-04-26T04:16:08.968 INFO:teuthology.orchestra.run.smithi063.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-26T04:16:08.971 INFO:teuthology.orchestra.run.smithi063.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-26T04:16:08.982 INFO:teuthology.orchestra.run.smithi063.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-26T04:16:08.983 INFO:teuthology.orchestra.run.smithi063.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-26T04:16:08.984 INFO:teuthology.orchestra.run.smithi063.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-26T04:16:08.985 INFO:teuthology.orchestra.run.smithi063.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-26T04:16:08.992 INFO:teuthology.orchestra.run.smithi063.stdout:Get:84 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-2457-gddeaa917-1jammy [711 kB] 2024-04-26T04:16:09.003 INFO:teuthology.orchestra.run.smithi063.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-26T04:16:09.011 INFO:teuthology.orchestra.run.smithi063.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-26T04:16:09.012 INFO:teuthology.orchestra.run.smithi063.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-26T04:16:09.012 INFO:teuthology.orchestra.run.smithi063.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-26T04:16:09.015 INFO:teuthology.orchestra.run.smithi063.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-26T04:16:09.016 INFO:teuthology.orchestra.run.smithi063.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-26T04:16:09.039 INFO:teuthology.orchestra.run.smithi063.stdout:Get:91 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-2457-gddeaa917-1jammy [356 kB] 2024-04-26T04:16:09.046 INFO:teuthology.orchestra.run.smithi063.stdout:Get:92 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-2457-gddeaa917-1jammy [32.8 kB] 2024-04-26T04:16:09.047 INFO:teuthology.orchestra.run.smithi063.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-2457-gddeaa917-1jammy [180 kB] 2024-04-26T04:16:09.054 INFO:teuthology.orchestra.run.smithi063.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-2457-gddeaa917-1jammy [65.8 kB] 2024-04-26T04:16:09.054 INFO:teuthology.orchestra.run.smithi063.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-2457-gddeaa917-1jammy [330 kB] 2024-04-26T04:16:09.060 INFO:teuthology.orchestra.run.smithi063.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-2457-gddeaa917-1jammy [6,775 kB] 2024-04-26T04:16:09.099 INFO:teuthology.orchestra.run.smithi186.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-26T04:16:09.112 INFO:teuthology.orchestra.run.smithi186.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-26T04:16:09.193 INFO:teuthology.orchestra.run.smithi186.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-26T04:16:09.317 INFO:teuthology.orchestra.run.smithi063.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-2457-gddeaa917-1jammy [112 kB] 2024-04-26T04:16:09.320 INFO:teuthology.orchestra.run.smithi063.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-2457-gddeaa917-1jammy [466 kB] 2024-04-26T04:16:09.335 INFO:teuthology.orchestra.run.smithi063.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-2457-gddeaa917-1jammy [25.6 MB] 2024-04-26T04:16:09.423 INFO:teuthology.orchestra.run.smithi186.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-26T04:16:09.435 INFO:teuthology.orchestra.run.smithi186.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-26T04:16:09.465 INFO:teuthology.orchestra.run.smithi186.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-26T04:16:09.474 INFO:teuthology.orchestra.run.smithi186.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-26T04:16:09.475 INFO:teuthology.orchestra.run.smithi186.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-26T04:16:09.476 INFO:teuthology.orchestra.run.smithi186.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-26T04:16:09.495 INFO:teuthology.orchestra.run.smithi186.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-04-26T04:16:09.666 INFO:teuthology.orchestra.run.smithi186.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-04-26T04:16:09.741 INFO:teuthology.orchestra.run.smithi186.stdout:Get:14 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-2457-gddeaa917-1jammy [3,218 kB] 2024-04-26T04:16:10.081 INFO:teuthology.orchestra.run.smithi186.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-04-26T04:16:10.114 INFO:teuthology.orchestra.run.smithi186.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-04-26T04:16:10.116 INFO:teuthology.orchestra.run.smithi186.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-04-26T04:16:10.196 INFO:teuthology.orchestra.run.smithi186.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-04-26T04:16:10.196 INFO:teuthology.orchestra.run.smithi186.stdout:Get:19 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-2457-gddeaa917-1jammy [711 kB] 2024-04-26T04:16:10.247 INFO:teuthology.orchestra.run.smithi186.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-04-26T04:16:10.269 INFO:teuthology.orchestra.run.smithi186.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-26T04:16:10.273 INFO:teuthology.orchestra.run.smithi186.stdout:Get:22 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-2457-gddeaa917-1jammy [356 kB] 2024-04-26T04:16:10.276 INFO:teuthology.orchestra.run.smithi186.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-26T04:16:10.276 INFO:teuthology.orchestra.run.smithi186.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-26T04:16:10.286 INFO:teuthology.orchestra.run.smithi186.stdout:Get:25 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-2457-gddeaa917-1jammy [32.8 kB] 2024-04-26T04:16:10.288 INFO:teuthology.orchestra.run.smithi186.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-26T04:16:10.289 INFO:teuthology.orchestra.run.smithi186.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-26T04:16:10.338 INFO:teuthology.orchestra.run.smithi186.stdout:Get:28 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-2457-gddeaa917-1jammy [180 kB] 2024-04-26T04:16:10.345 INFO:teuthology.orchestra.run.smithi186.stdout:Get:29 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-2457-gddeaa917-1jammy [65.8 kB] 2024-04-26T04:16:10.347 INFO:teuthology.orchestra.run.smithi186.stdout:Get:30 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-2457-gddeaa917-1jammy [330 kB] 2024-04-26T04:16:10.358 INFO:teuthology.orchestra.run.smithi186.stdout:Get:31 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-2457-gddeaa917-1jammy [6,775 kB] 2024-04-26T04:16:10.455 INFO:teuthology.orchestra.run.smithi186.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-26T04:16:10.498 INFO:teuthology.orchestra.run.smithi063.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-2457-gddeaa917-1jammy [5,084 kB] 2024-04-26T04:16:10.538 INFO:teuthology.orchestra.run.smithi186.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-26T04:16:10.567 INFO:teuthology.orchestra.run.smithi186.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-26T04:16:10.584 INFO:teuthology.orchestra.run.smithi186.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-26T04:16:10.628 INFO:teuthology.orchestra.run.smithi186.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-26T04:16:10.663 INFO:teuthology.orchestra.run.smithi063.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-2457-gddeaa917-1jammy [244 kB] 2024-04-26T04:16:10.670 INFO:teuthology.orchestra.run.smithi063.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-2457-gddeaa917-1jammy [124 kB] 2024-04-26T04:16:10.673 INFO:teuthology.orchestra.run.smithi063.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-2457-gddeaa917-1jammy [1,527 kB] 2024-04-26T04:16:10.748 INFO:teuthology.orchestra.run.smithi063.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-2457-gddeaa917-1jammy [6,245 kB] 2024-04-26T04:16:10.843 INFO:teuthology.orchestra.run.smithi186.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-26T04:16:10.858 INFO:teuthology.orchestra.run.smithi186.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-26T04:16:10.865 INFO:teuthology.orchestra.run.smithi186.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-26T04:16:10.870 INFO:teuthology.orchestra.run.smithi186.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-26T04:16:10.892 INFO:teuthology.orchestra.run.smithi186.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-26T04:16:10.895 INFO:teuthology.orchestra.run.smithi186.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-26T04:16:11.035 INFO:teuthology.orchestra.run.smithi063.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-2457-gddeaa917-1jammy [22.7 MB] 2024-04-26T04:16:11.063 INFO:teuthology.orchestra.run.smithi186.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-26T04:16:11.081 INFO:teuthology.orchestra.run.smithi186.stdout:Get:44 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-2457-gddeaa917-1jammy [112 kB] 2024-04-26T04:16:11.086 INFO:teuthology.orchestra.run.smithi186.stdout:Get:45 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-2457-gddeaa917-1jammy [466 kB] 2024-04-26T04:16:11.137 INFO:teuthology.orchestra.run.smithi186.stdout:Get:46 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-2457-gddeaa917-1jammy [25.6 MB] 2024-04-26T04:16:11.396 INFO:teuthology.orchestra.run.smithi186.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-26T04:16:11.428 INFO:teuthology.orchestra.run.smithi186.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-26T04:16:11.438 INFO:teuthology.orchestra.run.smithi186.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-26T04:16:11.640 INFO:teuthology.orchestra.run.smithi186.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-26T04:16:11.641 INFO:teuthology.orchestra.run.smithi186.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-26T04:16:11.648 INFO:teuthology.orchestra.run.smithi186.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-26T04:16:11.653 INFO:teuthology.orchestra.run.smithi186.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-26T04:16:11.668 INFO:teuthology.orchestra.run.smithi186.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-26T04:16:11.690 INFO:teuthology.orchestra.run.smithi186.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-26T04:16:11.696 INFO:teuthology.orchestra.run.smithi186.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-26T04:16:11.872 INFO:teuthology.orchestra.run.smithi186.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-26T04:16:12.135 INFO:teuthology.orchestra.run.smithi186.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-26T04:16:12.255 INFO:teuthology.orchestra.run.smithi063.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-2457-gddeaa917-1jammy [14.1 kB] 2024-04-26T04:16:12.255 INFO:teuthology.orchestra.run.smithi063.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-2457-gddeaa917-1jammy [869 kB] 2024-04-26T04:16:12.272 INFO:teuthology.orchestra.run.smithi063.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-2457-gddeaa917-1jammy [2,438 kB] 2024-04-26T04:16:12.313 INFO:teuthology.orchestra.run.smithi186.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-26T04:16:12.326 INFO:teuthology.orchestra.run.smithi186.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-26T04:16:12.328 INFO:teuthology.orchestra.run.smithi063.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-2457-gddeaa917-1jammy [791 kB] 2024-04-26T04:16:12.346 INFO:teuthology.orchestra.run.smithi186.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-26T04:16:12.369 INFO:teuthology.orchestra.run.smithi063.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-2457-gddeaa917-1jammy [147 kB] 2024-04-26T04:16:12.373 INFO:teuthology.orchestra.run.smithi063.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-2457-gddeaa917-1jammy [2,293 kB] 2024-04-26T04:16:12.411 INFO:teuthology.orchestra.run.smithi186.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-26T04:16:12.480 INFO:teuthology.orchestra.run.smithi063.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-2457-gddeaa917-1jammy [8,625 kB] 2024-04-26T04:16:12.700 INFO:teuthology.orchestra.run.smithi186.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-26T04:16:12.702 INFO:teuthology.orchestra.run.smithi186.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-26T04:16:12.705 INFO:teuthology.orchestra.run.smithi186.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-26T04:16:12.705 INFO:teuthology.orchestra.run.smithi186.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-26T04:16:12.707 INFO:teuthology.orchestra.run.smithi186.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-26T04:16:12.928 INFO:teuthology.orchestra.run.smithi186.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-26T04:16:12.977 INFO:teuthology.orchestra.run.smithi063.stdout:Get:113 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-2457-gddeaa917-1jammy [14.2 kB] 2024-04-26T04:16:12.977 INFO:teuthology.orchestra.run.smithi063.stdout:Get:114 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-2457-gddeaa917-1jammy [50.7 MB] 2024-04-26T04:16:13.363 INFO:teuthology.orchestra.run.smithi186.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-26T04:16:13.369 INFO:teuthology.orchestra.run.smithi186.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-26T04:16:13.657 INFO:teuthology.orchestra.run.smithi186.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-26T04:16:13.657 INFO:teuthology.orchestra.run.smithi186.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-26T04:16:13.659 INFO:teuthology.orchestra.run.smithi186.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-26T04:16:13.661 INFO:teuthology.orchestra.run.smithi186.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-26T04:16:13.665 INFO:teuthology.orchestra.run.smithi186.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-26T04:16:13.667 INFO:teuthology.orchestra.run.smithi186.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-26T04:16:13.668 INFO:teuthology.orchestra.run.smithi186.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-26T04:16:13.669 INFO:teuthology.orchestra.run.smithi186.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-26T04:16:13.810 INFO:teuthology.orchestra.run.smithi186.stdout:Get:79 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-2457-gddeaa917-1jammy [5,084 kB] 2024-04-26T04:16:13.906 INFO:teuthology.orchestra.run.smithi186.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-26T04:16:14.159 INFO:teuthology.orchestra.run.smithi186.stdout:Get:81 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-2457-gddeaa917-1jammy [244 kB] 2024-04-26T04:16:14.249 INFO:teuthology.orchestra.run.smithi186.stdout:Get:82 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-2457-gddeaa917-1jammy [124 kB] 2024-04-26T04:16:14.252 INFO:teuthology.orchestra.run.smithi186.stdout:Get:83 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-2457-gddeaa917-1jammy [1,527 kB] 2024-04-26T04:16:14.316 INFO:teuthology.orchestra.run.smithi186.stdout:Get:84 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-2457-gddeaa917-1jammy [6,245 kB] 2024-04-26T04:16:14.323 INFO:teuthology.orchestra.run.smithi186.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-26T04:16:14.370 INFO:teuthology.orchestra.run.smithi186.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-26T04:16:14.405 INFO:teuthology.orchestra.run.smithi186.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-26T04:16:14.503 INFO:teuthology.orchestra.run.smithi186.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-26T04:16:14.543 INFO:teuthology.orchestra.run.smithi186.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-04-26T04:16:14.555 INFO:teuthology.orchestra.run.smithi186.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-04-26T04:16:14.560 INFO:teuthology.orchestra.run.smithi186.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-26T04:16:14.608 INFO:teuthology.orchestra.run.smithi186.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-04-26T04:16:14.611 INFO:teuthology.orchestra.run.smithi186.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-26T04:16:14.632 INFO:teuthology.orchestra.run.smithi186.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-26T04:16:14.798 INFO:teuthology.orchestra.run.smithi186.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-26T04:16:14.847 INFO:teuthology.orchestra.run.smithi186.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-2457-gddeaa917-1jammy [22.7 MB] 2024-04-26T04:16:15.071 INFO:teuthology.orchestra.run.smithi186.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-26T04:16:15.088 INFO:teuthology.orchestra.run.smithi186.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-26T04:16:15.243 INFO:teuthology.orchestra.run.smithi186.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-26T04:16:15.272 INFO:teuthology.orchestra.run.smithi186.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-26T04:16:15.429 INFO:teuthology.orchestra.run.smithi186.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-26T04:16:15.431 INFO:teuthology.orchestra.run.smithi186.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-26T04:16:15.434 INFO:teuthology.orchestra.run.smithi186.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-26T04:16:15.456 INFO:teuthology.orchestra.run.smithi186.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-26T04:16:15.460 INFO:teuthology.orchestra.run.smithi186.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-26T04:16:15.872 INFO:teuthology.orchestra.run.smithi063.stdout:Get:115 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-2457-gddeaa917-1jammy [121 kB] 2024-04-26T04:16:15.874 INFO:teuthology.orchestra.run.smithi063.stdout:Get:116 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-2457-gddeaa917-1jammy [39.0 kB] 2024-04-26T04:16:15.875 INFO:teuthology.orchestra.run.smithi063.stdout:Get:117 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-2457-gddeaa917-1jammy [13.4 MB] 2024-04-26T04:16:16.419 INFO:teuthology.orchestra.run.smithi063.stdout:Get:118 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-2457-gddeaa917-1jammy [91.2 kB] 2024-04-26T04:16:16.797 INFO:teuthology.orchestra.run.smithi186.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-2457-gddeaa917-1jammy [14.1 kB] 2024-04-26T04:16:16.797 INFO:teuthology.orchestra.run.smithi186.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-2457-gddeaa917-1jammy [869 kB] 2024-04-26T04:16:16.830 INFO:teuthology.orchestra.run.smithi186.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-2457-gddeaa917-1jammy [2,438 kB] 2024-04-26T04:16:17.002 INFO:teuthology.orchestra.run.smithi063.stdout:Fetched 179 MB in 8s (22.2 MB/s) 2024-04-26T04:16:17.172 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-26T04:16:17.245 INFO:teuthology.orchestra.run.smithi186.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-2457-gddeaa917-1jammy [791 kB] 2024-04-26T04:16:17.275 INFO:teuthology.orchestra.run.smithi063.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165995 files and directories currently installed.) 2024-04-26T04:16:17.282 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-26T04:16:17.362 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-26T04:16:17.411 INFO:teuthology.orchestra.run.smithi186.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-2457-gddeaa917-1jammy [147 kB] 2024-04-26T04:16:17.414 INFO:teuthology.orchestra.run.smithi186.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-2457-gddeaa917-1jammy [2,293 kB] 2024-04-26T04:16:17.657 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-26T04:16:17.684 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-26T04:16:17.715 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-26T04:16:17.824 INFO:teuthology.orchestra.run.smithi186.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-2457-gddeaa917-1jammy [8,625 kB] 2024-04-26T04:16:18.060 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-26T04:16:18.087 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-26T04:16:18.127 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-26T04:16:18.589 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-26T04:16:18.616 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-26T04:16:18.664 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T04:16:19.000 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-26T04:16:19.027 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-26T04:16:19.058 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T04:16:19.408 INFO:teuthology.orchestra.run.smithi186.stdout:Get:113 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-2457-gddeaa917-1jammy [14.2 kB] 2024-04-26T04:16:19.408 INFO:teuthology.orchestra.run.smithi186.stdout:Get:114 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-2457-gddeaa917-1jammy [50.7 MB] 2024-04-26T04:16:19.437 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-26T04:16:19.464 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-26T04:16:19.503 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T04:16:19.831 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-26T04:16:19.858 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-26T04:16:19.898 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-26T04:16:20.176 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package librados2. 2024-04-26T04:16:20.203 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../007-librados2_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T04:16:20.234 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking librados2 (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:16:20.679 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package librbd1. 2024-04-26T04:16:20.705 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../008-librbd1_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T04:16:20.862 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking librbd1 (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:16:21.307 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package libcephfs2. 2024-04-26T04:16:21.331 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T04:16:21.366 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking libcephfs2 (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:16:21.617 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-rados. 2024-04-26T04:16:21.633 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../010-python3-rados_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T04:16:21.676 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-rados (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:16:22.012 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-26T04:16:22.039 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-2457-gddeaa917-1jammy_all.deb ... 2024-04-26T04:16:22.079 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-ceph-argparse (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:16:22.331 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-cephfs. 2024-04-26T04:16:22.358 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T04:16:22.398 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-cephfs (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:16:22.684 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-26T04:16:22.711 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-2457-gddeaa917-1jammy_all.deb ... 2024-04-26T04:16:22.742 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-ceph-common (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:16:23.012 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-26T04:16:23.039 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-26T04:16:23.079 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-26T04:16:23.398 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-prettytable. 2024-04-26T04:16:23.425 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-04-26T04:16:23.465 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-26T04:16:23.726 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-rbd. 2024-04-26T04:16:23.753 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T04:16:23.793 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-rbd (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:16:24.146 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-26T04:16:24.173 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-26T04:16:24.212 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-26T04:16:24.523 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-04-26T04:16:24.550 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-04-26T04:16:24.590 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-04-26T04:16:24.859 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-04-26T04:16:24.875 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-04-26T04:16:24.909 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-26T04:16:25.248 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package lua5.1. 2024-04-26T04:16:25.274 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-04-26T04:16:25.314 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-04-26T04:16:25.509 INFO:teuthology.orchestra.run.smithi186.stdout:Get:115 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-2457-gddeaa917-1jammy [121 kB] 2024-04-26T04:16:25.580 INFO:teuthology.orchestra.run.smithi186.stdout:Get:116 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-2457-gddeaa917-1jammy [39.0 kB] 2024-04-26T04:16:25.581 INFO:teuthology.orchestra.run.smithi186.stdout:Get:117 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-2457-gddeaa917-1jammy [13.4 MB] 2024-04-26T04:16:25.617 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package lua-any. 2024-04-26T04:16:25.633 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-04-26T04:16:25.670 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking lua-any (27ubuntu1) ... 2024-04-26T04:16:25.886 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package zip. 2024-04-26T04:16:25.913 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-04-26T04:16:25.953 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking zip (3.0-12build2) ... 2024-04-26T04:16:26.381 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package unzip. 2024-04-26T04:16:26.408 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-04-26T04:16:26.436 INFO:teuthology.orchestra.run.smithi186.stdout:Get:118 https://3.chacra.ceph.com/r/ceph/wip-yuri2-testing-2024-04-24-0914-squid/ddeaa91797339a19172c3036ff48cca58c12f448/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-2457-gddeaa917-1jammy [91.2 kB] 2024-04-26T04:16:26.447 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-04-26T04:16:26.826 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package luarocks. 2024-04-26T04:16:26.853 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-04-26T04:16:26.894 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-04-26T04:16:27.009 INFO:teuthology.orchestra.run.smithi186.stdout:Fetched 179 MB in 18s (10.0 MB/s) 2024-04-26T04:16:27.178 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-26T04:16:27.275 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package librgw2. 2024-04-26T04:16:27.276 INFO:teuthology.orchestra.run.smithi186.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165995 files and directories currently installed.) 2024-04-26T04:16:27.282 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-26T04:16:27.302 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../025-librgw2_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T04:16:27.342 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking librgw2 (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:16:27.368 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-26T04:16:27.705 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-26T04:16:27.731 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-26T04:16:27.780 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-26T04:16:27.894 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-rgw. 2024-04-26T04:16:27.922 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T04:16:27.960 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-rgw (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:16:28.116 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-26T04:16:28.143 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-26T04:16:28.182 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-26T04:16:28.322 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-26T04:16:28.348 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-26T04:16:28.388 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-26T04:16:28.569 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-26T04:16:28.596 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-26T04:16:28.644 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T04:16:28.708 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package libradosstriper1. 2024-04-26T04:16:28.734 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T04:16:28.774 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking libradosstriper1 (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:16:29.039 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-26T04:16:29.066 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-26T04:16:29.106 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T04:16:29.177 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package ceph-common. 2024-04-26T04:16:29.204 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../029-ceph-common_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T04:16:29.244 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking ceph-common (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:16:29.450 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-26T04:16:29.477 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-26T04:16:29.517 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T04:16:29.864 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-26T04:16:29.892 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-26T04:16:30.036 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-26T04:16:30.398 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package librados2. 2024-04-26T04:16:30.424 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../007-librados2_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T04:16:30.464 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking librados2 (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:16:30.649 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package ceph-base. 2024-04-26T04:16:30.676 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../030-ceph-base_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T04:16:30.725 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking ceph-base (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:16:30.925 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package librbd1. 2024-04-26T04:16:30.952 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../008-librbd1_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T04:16:30.992 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking librbd1 (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:16:31.419 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libcephfs2. 2024-04-26T04:16:31.435 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T04:16:31.436 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-26T04:16:31.463 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-26T04:16:31.469 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libcephfs2 (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:16:31.503 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-26T04:16:31.714 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-rados. 2024-04-26T04:16:31.740 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../010-python3-rados_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T04:16:31.780 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-rados (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:16:31.814 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-cheroot. 2024-04-26T04:16:31.841 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-26T04:16:31.880 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-26T04:16:32.075 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-26T04:16:32.101 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-2457-gddeaa917-1jammy_all.deb ... 2024-04-26T04:16:32.133 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-ceph-argparse (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:16:32.200 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-26T04:16:32.226 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-26T04:16:32.266 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-26T04:16:32.369 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-cephfs. 2024-04-26T04:16:32.384 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T04:16:32.427 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-cephfs (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:16:32.569 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-26T04:16:32.596 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-26T04:16:32.635 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-26T04:16:32.679 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-26T04:16:32.706 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-2457-gddeaa917-1jammy_all.deb ... 2024-04-26T04:16:32.746 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-ceph-common (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:16:32.947 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-26T04:16:32.974 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-26T04:16:33.013 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-26T04:16:33.058 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-26T04:16:33.084 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-26T04:16:33.124 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-26T04:16:33.266 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-tempora. 2024-04-26T04:16:33.282 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-04-26T04:16:33.316 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-26T04:16:33.435 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-prettytable. 2024-04-26T04:16:33.462 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-04-26T04:16:33.502 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-26T04:16:33.619 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-portend. 2024-04-26T04:16:33.646 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-04-26T04:16:33.685 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-26T04:16:33.788 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-rbd. 2024-04-26T04:16:33.815 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T04:16:33.854 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-rbd (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:16:33.988 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-26T04:16:34.015 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-26T04:16:34.055 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-26T04:16:34.149 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-26T04:16:34.176 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-26T04:16:34.207 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-26T04:16:34.391 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-26T04:16:34.418 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-26T04:16:34.466 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-26T04:16:34.502 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-04-26T04:16:34.528 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-04-26T04:16:34.568 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-04-26T04:16:34.836 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-natsort. 2024-04-26T04:16:34.862 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-04-26T04:16:34.871 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-04-26T04:16:34.886 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-04-26T04:16:34.902 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-26T04:16:34.933 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-26T04:16:35.239 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-logutils. 2024-04-26T04:16:35.265 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-04-26T04:16:35.305 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-26T04:16:35.366 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package lua5.1. 2024-04-26T04:16:35.392 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-04-26T04:16:35.433 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-04-26T04:16:35.641 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-26T04:16:35.668 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-26T04:16:35.716 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-26T04:16:35.744 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package lua-any. 2024-04-26T04:16:35.759 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-04-26T04:16:35.802 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking lua-any (27ubuntu1) ... 2024-04-26T04:16:36.046 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package zip. 2024-04-26T04:16:36.069 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-mako. 2024-04-26T04:16:36.072 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-04-26T04:16:36.096 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-26T04:16:36.113 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking zip (3.0-12build2) ... 2024-04-26T04:16:36.144 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-26T04:16:36.425 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package unzip. 2024-04-26T04:16:36.451 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-04-26T04:16:36.499 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-04-26T04:16:36.589 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-26T04:16:36.616 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-26T04:16:36.705 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-26T04:16:36.862 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package luarocks. 2024-04-26T04:16:36.888 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-04-26T04:16:36.920 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-04-26T04:16:37.008 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-26T04:16:37.035 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-26T04:16:37.074 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-26T04:16:37.319 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package librgw2. 2024-04-26T04:16:37.346 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../025-librgw2_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T04:16:37.385 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-webob. 2024-04-26T04:16:37.386 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking librgw2 (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:16:37.412 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-26T04:16:37.452 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-26T04:16:37.797 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-waitress. 2024-04-26T04:16:37.824 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-26T04:16:37.866 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-26T04:16:37.913 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-rgw. 2024-04-26T04:16:37.941 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T04:16:37.980 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-rgw (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:16:38.149 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-tempita. 2024-04-26T04:16:38.176 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-26T04:16:38.207 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-26T04:16:38.291 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-26T04:16:38.318 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-26T04:16:38.349 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-26T04:16:38.493 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-paste. 2024-04-26T04:16:38.520 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-26T04:16:38.560 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-26T04:16:38.627 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libradosstriper1. 2024-04-26T04:16:38.654 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T04:16:38.694 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libradosstriper1 (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:16:38.880 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-26T04:16:38.896 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-26T04:16:38.938 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-26T04:16:39.130 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph-common. 2024-04-26T04:16:39.157 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../029-ceph-common_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T04:16:39.205 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph-common (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:16:39.207 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-26T04:16:39.234 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-26T04:16:39.274 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-26T04:16:39.560 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-webtest. 2024-04-26T04:16:39.587 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-04-26T04:16:39.626 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-26T04:16:39.962 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-pecan. 2024-04-26T04:16:39.990 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-26T04:16:40.029 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-26T04:16:40.392 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-certifi. 2024-04-26T04:16:40.419 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-04-26T04:16:40.619 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph-base. 2024-04-26T04:16:40.645 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../030-ceph-base_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T04:16:40.695 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph-base (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:16:40.700 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-26T04:16:41.340 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-26T04:16:41.366 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-26T04:16:41.406 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-26T04:16:41.503 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-idna. 2024-04-26T04:16:41.530 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-04-26T04:16:41.570 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-26T04:16:41.898 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-urllib3. 2024-04-26T04:16:41.925 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-26T04:16:41.964 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-26T04:16:42.284 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-requests. 2024-04-26T04:16:42.311 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-26T04:16:42.350 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-26T04:16:42.368 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-cheroot. 2024-04-26T04:16:42.395 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-26T04:16:42.434 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-26T04:16:42.662 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-26T04:16:42.689 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-26T04:16:42.728 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-26T04:16:42.754 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-26T04:16:42.780 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-26T04:16:42.821 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-26T04:16:43.031 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-26T04:16:43.047 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-2457-gddeaa917-1jammy_all.deb ... 2024-04-26T04:16:43.089 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking ceph-mgr-modules-core (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:16:43.132 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-26T04:16:43.158 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-26T04:16:43.198 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-26T04:16:43.477 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-26T04:16:43.501 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-26T04:16:43.504 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T04:16:43.528 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-26T04:16:43.543 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:16:43.568 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-26T04:16:43.813 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-26T04:16:43.837 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-tempora. 2024-04-26T04:16:43.840 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-26T04:16:43.853 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-04-26T04:16:43.871 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-26T04:16:43.896 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-26T04:16:44.182 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-openssl. 2024-04-26T04:16:44.207 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-portend. 2024-04-26T04:16:44.209 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-04-26T04:16:44.232 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-04-26T04:16:44.249 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-26T04:16:44.274 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-26T04:16:44.577 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package ceph-mgr. 2024-04-26T04:16:44.593 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-26T04:16:44.604 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T04:16:44.620 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-26T04:16:44.643 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking ceph-mgr (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:16:44.659 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-26T04:16:44.954 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-26T04:16:44.981 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-26T04:16:45.012 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-26T04:16:45.055 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package ceph-mon. 2024-04-26T04:16:45.082 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T04:16:45.121 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking ceph-mon (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:16:45.574 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-natsort. 2024-04-26T04:16:45.601 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-04-26T04:16:45.641 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-26T04:16:45.691 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package ceph-osd. 2024-04-26T04:16:45.718 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T04:16:45.758 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking ceph-osd (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:16:45.952 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-logutils. 2024-04-26T04:16:45.979 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-04-26T04:16:46.019 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-26T04:16:46.280 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-26T04:16:46.296 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-26T04:16:46.338 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-26T04:16:46.633 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-mako. 2024-04-26T04:16:46.659 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-26T04:16:46.699 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-26T04:16:46.827 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package ceph. 2024-04-26T04:16:46.854 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../066-ceph_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T04:16:47.036 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-26T04:16:47.044 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking ceph (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:16:47.062 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-26T04:16:47.102 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-26T04:16:47.363 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package ceph-fuse. 2024-04-26T04:16:47.390 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T04:16:47.405 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-26T04:16:47.429 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking ceph-fuse (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:16:47.431 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-26T04:16:47.472 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-26T04:16:47.724 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-webob. 2024-04-26T04:16:47.740 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-26T04:16:47.783 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-26T04:16:47.783 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package ceph-mds. 2024-04-26T04:16:47.810 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T04:16:47.841 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking ceph-mds (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:16:48.144 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-waitress. 2024-04-26T04:16:48.173 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-26T04:16:48.213 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-26T04:16:48.227 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package cephadm. 2024-04-26T04:16:48.254 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../069-cephadm_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T04:16:48.293 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking cephadm (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:16:48.522 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-tempita. 2024-04-26T04:16:48.548 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-26T04:16:48.588 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-26T04:16:48.613 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-26T04:16:48.640 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-26T04:16:48.679 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-26T04:16:48.900 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-paste. 2024-04-26T04:16:48.927 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-26T04:16:48.966 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-26T04:16:49.007 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-26T04:16:49.034 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-26T04:16:49.073 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-26T04:16:49.278 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-26T04:16:49.293 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-26T04:16:49.336 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-26T04:16:49.639 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-26T04:16:49.665 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-26T04:16:49.705 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-26T04:16:49.983 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-webtest. 2024-04-26T04:16:49.999 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-04-26T04:16:50.033 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-26T04:16:50.162 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-babel. 2024-04-26T04:16:50.189 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-26T04:16:50.228 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-26T04:16:50.311 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-pecan. 2024-04-26T04:16:50.327 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-26T04:16:50.361 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-26T04:16:50.556 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-jinja2. 2024-04-26T04:16:50.583 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-26T04:16:50.622 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-26T04:16:50.708 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-certifi. 2024-04-26T04:16:50.734 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-04-26T04:16:50.766 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-26T04:16:50.975 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-26T04:16:51.002 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-2457-gddeaa917-1jammy_all.deb ... 2024-04-26T04:16:51.042 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking ceph-mgr-cephadm (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:16:51.077 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-idna. 2024-04-26T04:16:51.104 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-04-26T04:16:51.143 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-26T04:16:51.378 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-26T04:16:51.405 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-04-26T04:16:51.429 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-urllib3. 2024-04-26T04:16:51.456 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-26T04:16:51.468 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-26T04:16:51.496 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-26T04:16:51.781 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-routes. 2024-04-26T04:16:51.808 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-26T04:16:51.816 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-requests. 2024-04-26T04:16:51.843 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-26T04:16:51.847 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-26T04:16:51.883 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-26T04:16:52.184 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-26T04:16:52.194 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-26T04:16:52.211 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-2457-gddeaa917-1jammy_all.deb ... 2024-04-26T04:16:52.221 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-26T04:16:52.250 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking ceph-mgr-dashboard (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:16:52.261 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-26T04:16:52.521 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-26T04:16:52.537 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-2457-gddeaa917-1jammy_all.deb ... 2024-04-26T04:16:52.580 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph-mgr-modules-core (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:16:52.968 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-26T04:16:52.995 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T04:16:53.034 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:16:53.337 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-26T04:16:53.364 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-26T04:16:53.404 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-26T04:16:53.488 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-26T04:16:53.505 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-26T04:16:53.546 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-26T04:16:53.715 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-openssl. 2024-04-26T04:16:53.741 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-04-26T04:16:53.782 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-26T04:16:54.042 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-joblib. 2024-04-26T04:16:54.060 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph-mgr. 2024-04-26T04:16:54.069 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-26T04:16:54.086 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T04:16:54.108 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-26T04:16:54.126 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph-mgr (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:16:54.470 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-26T04:16:54.497 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-26T04:16:54.521 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph-mon. 2024-04-26T04:16:54.536 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-26T04:16:54.548 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T04:16:54.587 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph-mon (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:16:54.839 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-sklearn. 2024-04-26T04:16:54.866 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-26T04:16:54.914 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-26T04:16:55.283 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph-osd. 2024-04-26T04:16:55.310 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T04:16:55.349 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph-osd (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:16:55.672 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-26T04:16:55.700 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-2457-gddeaa917-1jammy_all.deb ... 2024-04-26T04:16:55.832 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:16:56.402 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph. 2024-04-26T04:16:56.429 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../066-ceph_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T04:16:56.469 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:16:56.777 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-cachetools. 2024-04-26T04:16:56.788 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph-fuse. 2024-04-26T04:16:56.804 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-04-26T04:16:56.815 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T04:16:56.843 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-26T04:16:56.855 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph-fuse (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:16:57.209 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph-mds. 2024-04-26T04:16:57.235 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T04:16:57.275 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph-mds (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:16:57.346 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-26T04:16:57.374 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-26T04:16:57.421 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-26T04:16:57.669 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package cephadm. 2024-04-26T04:16:57.696 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../069-cephadm_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T04:16:57.744 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking cephadm (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:16:57.750 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-26T04:16:57.778 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-26T04:16:57.816 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-26T04:16:58.064 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-26T04:16:58.090 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-26T04:16:58.130 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-26T04:16:58.135 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-rsa. 2024-04-26T04:16:58.163 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-04-26T04:16:58.202 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-26T04:16:58.450 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-26T04:16:58.476 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-26T04:16:58.489 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-google-auth. 2024-04-26T04:16:58.516 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-26T04:16:58.517 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-04-26T04:16:58.547 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-26T04:16:58.842 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-26T04:16:58.869 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-26T04:16:58.908 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-26T04:16:59.219 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-websocket. 2024-04-26T04:16:59.247 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-04-26T04:16:59.286 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-26T04:16:59.538 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-babel. 2024-04-26T04:16:59.565 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-26T04:16:59.604 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-26T04:16:59.605 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-26T04:16:59.634 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-26T04:16:59.672 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-26T04:16:59.909 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-jinja2. 2024-04-26T04:16:59.936 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-26T04:16:59.967 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-26T04:17:00.270 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-26T04:17:00.297 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-2457-gddeaa917-1jammy_all.deb ... 2024-04-26T04:17:00.328 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph-mgr-cephadm (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:00.386 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-26T04:17:00.403 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-2457-gddeaa917-1jammy_all.deb ... 2024-04-26T04:17:00.444 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:00.690 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-26T04:17:00.717 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-04-26T04:17:00.756 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-26T04:17:00.763 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-26T04:17:00.791 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-26T04:17:00.830 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-26T04:17:01.051 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-routes. 2024-04-26T04:17:01.078 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-26T04:17:01.117 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-26T04:17:01.175 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-26T04:17:01.203 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-26T04:17:01.241 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-26T04:17:01.462 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-26T04:17:01.477 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package jq. 2024-04-26T04:17:01.490 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-2457-gddeaa917-1jammy_all.deb ... 2024-04-26T04:17:01.493 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-26T04:17:01.526 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-26T04:17:01.529 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph-mgr-dashboard (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:01.746 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package socat. 2024-04-26T04:17:01.762 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-26T04:17:01.804 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-26T04:17:02.308 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package xmlstarlet. 2024-04-26T04:17:02.335 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-26T04:17:02.374 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-26T04:17:02.728 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package ceph-test. 2024-04-26T04:17:02.745 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../097-ceph-test_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T04:17:02.778 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking ceph-test (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:02.809 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-26T04:17:02.825 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-26T04:17:02.867 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-26T04:17:03.313 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-joblib. 2024-04-26T04:17:03.328 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-26T04:17:03.362 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-26T04:17:03.724 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-26T04:17:03.751 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-26T04:17:03.790 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-26T04:17:04.101 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-sklearn. 2024-04-26T04:17:04.129 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-26T04:17:04.168 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-26T04:17:04.902 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-26T04:17:04.929 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-2457-gddeaa917-1jammy_all.deb ... 2024-04-26T04:17:05.054 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:05.159 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package ceph-volume. 2024-04-26T04:17:05.187 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-2457-gddeaa917-1jammy_all.deb ... 2024-04-26T04:17:05.225 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking ceph-volume (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:05.554 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-26T04:17:05.582 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T04:17:05.620 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking libcephfs-dev (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:05.956 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package lua-socket:amd64. 2024-04-26T04:17:05.982 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-cachetools. 2024-04-26T04:17:05.984 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-04-26T04:17:06.009 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-04-26T04:17:06.023 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-26T04:17:06.040 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-26T04:17:06.318 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-26T04:17:06.344 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package lua-sec:amd64. 2024-04-26T04:17:06.346 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-26T04:17:06.372 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-04-26T04:17:06.385 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-26T04:17:06.410 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-04-26T04:17:06.764 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package nvme-cli. 2024-04-26T04:17:06.792 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-26T04:17:06.830 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-26T04:17:06.905 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-26T04:17:06.921 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-26T04:17:06.963 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-26T04:17:07.257 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-rsa. 2024-04-26T04:17:07.269 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package pkg-config. 2024-04-26T04:17:07.285 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-04-26T04:17:07.297 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-04-26T04:17:07.324 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-26T04:17:07.335 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-04-26T04:17:07.644 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-google-auth. 2024-04-26T04:17:07.672 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-04-26T04:17:07.709 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-26T04:17:07.711 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-26T04:17:07.737 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-26T04:17:07.838 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-26T04:17:08.022 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-26T04:17:08.050 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-26T04:17:08.089 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-26T04:17:08.274 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-26T04:17:08.302 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-26T04:17:08.357 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-26T04:17:08.392 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-websocket. 2024-04-26T04:17:08.419 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-04-26T04:17:08.458 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-26T04:17:08.660 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-pastescript. 2024-04-26T04:17:08.688 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-04-26T04:17:08.726 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-26T04:17:08.778 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-26T04:17:08.806 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-26T04:17:08.845 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-26T04:17:09.055 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-pluggy. 2024-04-26T04:17:09.083 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-26T04:17:09.121 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-26T04:17:09.374 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-psutil. 2024-04-26T04:17:09.402 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-26T04:17:09.432 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-26T04:17:09.433 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-26T04:17:09.462 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-2457-gddeaa917-1jammy_all.deb ... 2024-04-26T04:17:09.520 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:09.760 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-py. 2024-04-26T04:17:09.788 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-04-26T04:17:09.826 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-26T04:17:09.870 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-26T04:17:09.897 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-26T04:17:09.936 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-26T04:17:10.113 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-pygments. 2024-04-26T04:17:10.130 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-26T04:17:10.162 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-26T04:17:10.239 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-26T04:17:10.266 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-26T04:17:10.297 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-26T04:17:10.541 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package jq. 2024-04-26T04:17:10.566 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-26T04:17:10.608 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-26T04:17:10.633 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-26T04:17:10.662 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-26T04:17:10.699 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-26T04:17:10.869 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package socat. 2024-04-26T04:17:10.893 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-26T04:17:10.935 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-26T04:17:11.019 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-toml. 2024-04-26T04:17:11.047 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-04-26T04:17:11.086 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-26T04:17:11.238 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package xmlstarlet. 2024-04-26T04:17:11.255 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-26T04:17:11.297 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-26T04:17:11.363 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-pytest. 2024-04-26T04:17:11.391 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-26T04:17:11.430 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-26T04:17:11.642 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph-test. 2024-04-26T04:17:11.658 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../097-ceph-test_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T04:17:11.701 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph-test (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:11.775 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-simplejson. 2024-04-26T04:17:11.803 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-26T04:17:11.841 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-26T04:17:12.118 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-26T04:17:12.136 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-26T04:17:12.177 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-26T04:17:12.872 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package radosgw. 2024-04-26T04:17:12.901 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../116-radosgw_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T04:17:12.930 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking radosgw (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:13.725 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package rbd-fuse. 2024-04-26T04:17:13.741 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T04:17:13.774 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking rbd-fuse (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:13.872 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package ceph-volume. 2024-04-26T04:17:13.900 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-2457-gddeaa917-1jammy_all.deb ... 2024-04-26T04:17:13.939 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking ceph-volume (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:14.047 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-26T04:17:14.225 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-26T04:17:14.241 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T04:17:14.283 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking libcephfs-dev (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:14.296 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-26T04:17:14.422 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-26T04:17:14.578 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package lua-socket:amd64. 2024-04-26T04:17:14.606 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-04-26T04:17:14.636 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-26T04:17:14.685 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-26T04:17:14.949 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package lua-sec:amd64. 2024-04-26T04:17:14.977 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-04-26T04:17:15.011 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-26T04:17:15.015 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-04-26T04:17:15.370 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package nvme-cli. 2024-04-26T04:17:15.398 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-26T04:17:15.431 INFO:teuthology.orchestra.run.smithi063.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-26T04:17:15.436 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-26T04:17:15.498 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up cephadm (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:15.849 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package pkg-config. 2024-04-26T04:17:15.877 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-04-26T04:17:15.916 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-04-26T04:17:16.023 INFO:teuthology.orchestra.run.smithi063.stdout:Adding system user cephadm....done 2024-04-26T04:17:16.161 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-26T04:17:16.171 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-26T04:17:16.188 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-26T04:17:16.219 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-26T04:17:16.447 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-26T04:17:16.622 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-26T04:17:16.650 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-26T04:17:16.705 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-26T04:17:16.723 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-26T04:17:16.874 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-26T04:17:17.142 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-26T04:17:17.216 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-pastescript. 2024-04-26T04:17:17.243 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-04-26T04:17:17.282 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-26T04:17:17.393 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-26T04:17:17.511 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-26T04:17:17.594 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-pluggy. 2024-04-26T04:17:17.622 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-26T04:17:17.661 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-26T04:17:17.823 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-26T04:17:17.956 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-psutil. 2024-04-26T04:17:17.983 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-26T04:17:18.022 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-26T04:17:18.198 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-26T04:17:18.375 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-py. 2024-04-26T04:17:18.403 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-04-26T04:17:18.449 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-26T04:17:18.490 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-26T04:17:18.778 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-pygments. 2024-04-26T04:17:18.783 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-04-26T04:17:18.806 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-26T04:17:18.844 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-26T04:17:19.052 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-26T04:17:19.282 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-26T04:17:19.309 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-26T04:17:19.311 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-26T04:17:19.340 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-26T04:17:19.570 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-26T04:17:19.651 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-toml. 2024-04-26T04:17:19.678 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-04-26T04:17:19.717 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-26T04:17:19.838 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-ceph-argparse (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:20.020 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-pytest. 2024-04-26T04:17:20.048 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-26T04:17:20.087 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-26T04:17:20.130 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-26T04:17:20.282 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-26T04:17:20.433 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-04-26T04:17:20.457 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package python3-simplejson. 2024-04-26T04:17:20.484 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-26T04:17:20.523 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-26T04:17:20.592 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-26T04:17:20.792 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-26T04:17:20.808 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-26T04:17:20.850 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-26T04:17:20.876 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-04-26T04:17:21.263 INFO:teuthology.orchestra.run.smithi063.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-04-26T04:17:21.291 INFO:teuthology.orchestra.run.smithi063.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-04-26T04:17:21.354 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-26T04:17:21.480 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-26T04:17:21.513 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package radosgw. 2024-04-26T04:17:21.529 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../116-radosgw_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T04:17:21.563 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking radosgw (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:21.856 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-26T04:17:21.982 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-26T04:17:22.267 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-26T04:17:22.366 INFO:teuthology.orchestra.run.smithi186.stdout:Selecting previously unselected package rbd-fuse. 2024-04-26T04:17:22.394 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-2457-gddeaa917-1jammy_amd64.deb ... 2024-04-26T04:17:22.432 INFO:teuthology.orchestra.run.smithi186.stdout:Unpacking rbd-fuse (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:22.551 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-26T04:17:22.720 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-26T04:17:22.844 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up zip (3.0-12build2) ... 2024-04-26T04:17:22.970 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-26T04:17:22.971 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-26T04:17:23.096 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-26T04:17:23.357 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-26T04:17:23.664 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-26T04:17:23.675 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-26T04:17:23.932 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-26T04:17:24.082 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-26T04:17:24.089 INFO:teuthology.orchestra.run.smithi186.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-26T04:17:24.157 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up cephadm (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:24.242 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-26T04:17:24.559 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-26T04:17:24.673 INFO:teuthology.orchestra.run.smithi186.stdout:Adding system user cephadm....done 2024-04-26T04:17:24.829 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-26T04:17:24.961 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-26T04:17:25.097 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-26T04:17:25.212 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-26T04:17:25.348 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-26T04:17:25.474 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-26T04:17:25.605 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-26T04:17:25.751 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-26T04:17:25.906 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-26T04:17:26.019 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-26T04:17:26.145 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-26T04:17:26.291 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-26T04:17:26.446 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-26T04:17:26.567 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-26T04:17:26.668 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-ceph-common (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:26.790 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-26T04:17:26.952 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-idna (3.3-1) ... 2024-04-26T04:17:27.057 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-26T04:17:27.270 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-26T04:17:27.325 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-04-26T04:17:27.919 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-26T04:17:28.237 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-26T04:17:28.488 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-26T04:17:28.501 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-04-26T04:17:28.730 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-ceph-argparse (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:28.760 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T04:17:28.895 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-26T04:17:28.981 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-26T04:17:29.107 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-26T04:17:29.212 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-26T04:17:29.233 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-04-26T04:17:29.338 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-26T04:17:29.359 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-26T04:17:29.465 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-26T04:17:29.610 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-04-26T04:17:29.782 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-26T04:17:30.075 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-26T04:17:30.142 INFO:teuthology.orchestra.run.smithi186.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-04-26T04:17:30.166 INFO:teuthology.orchestra.run.smithi186.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-04-26T04:17:30.213 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-26T04:17:30.331 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-26T04:17:30.343 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T04:17:30.444 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-26T04:17:30.699 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-26T04:17:30.745 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-26T04:17:30.825 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-26T04:17:30.996 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-26T04:17:31.084 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-26T04:17:31.264 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-26T04:17:31.344 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-26T04:17:31.540 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-26T04:17:31.653 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up zip (3.0-12build2) ... 2024-04-26T04:17:31.763 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-26T04:17:31.807 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-26T04:17:32.092 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-26T04:17:32.218 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-26T04:17:32.440 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-26T04:17:32.511 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-26T04:17:32.628 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-26T04:17:32.716 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-26T04:17:32.833 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-26T04:17:32.913 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-26T04:17:32.951 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-26T04:17:33.239 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-26T04:17:33.243 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-26T04:17:33.498 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-26T04:17:33.637 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-26T04:17:33.649 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-04-26T04:17:33.809 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-26T04:17:33.888 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-26T04:17:33.906 INFO:teuthology.orchestra.run.smithi063.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-26T04:17:34.144 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-26T04:17:34.245 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-26T04:17:34.264 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-26T04:17:34.557 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-26T04:17:34.621 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-26T04:17:34.889 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up lua-any (27ubuntu1) ... 2024-04-26T04:17:34.942 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-26T04:17:35.015 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-26T04:17:35.184 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-26T04:17:35.282 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T04:17:35.319 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-ceph-common (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:35.433 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-26T04:17:35.620 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-idna (3.3-1) ... 2024-04-26T04:17:35.768 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-26T04:17:35.921 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-26T04:17:36.077 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-26T04:17:36.345 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-26T04:17:36.446 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-26T04:17:36.713 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-26T04:17:37.006 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-26T04:17:37.060 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-04-26T04:17:37.312 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T04:17:37.368 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-26T04:17:37.429 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-26T04:17:37.680 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-26T04:17:37.795 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-26T04:17:37.806 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-26T04:17:37.932 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-26T04:17:38.096 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-26T04:17:38.217 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-26T04:17:38.440 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-26T04:17:38.451 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-26T04:17:38.591 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up librados2 (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:38.703 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T04:17:38.742 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up libsqlite3-mod-ceph (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:38.829 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-26T04:17:38.876 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-04-26T04:17:39.122 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-26T04:17:39.229 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up libcephfs2 (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:39.355 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up libradosstriper1 (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:39.372 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-26T04:17:39.506 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-26T04:17:39.624 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-26T04:17:39.808 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up librbd1 (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:39.875 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-26T04:17:39.933 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up ceph-mgr-modules-core (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:40.060 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up ceph-fuse (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:40.117 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-26T04:17:40.277 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-26T04:17:40.277 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-26T04:17:40.377 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-26T04:17:40.503 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-26T04:17:40.769 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up libcephfs-dev (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:40.779 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-26T04:17:40.880 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-26T04:17:40.895 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-rados (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:41.030 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up librgw2 (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:41.165 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-26T04:17:41.180 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-26T04:17:41.457 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-26T04:17:41.708 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-26T04:17:41.842 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-04-26T04:17:41.960 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-26T04:17:42.049 INFO:teuthology.orchestra.run.smithi186.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-26T04:17:42.287 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-26T04:17:42.405 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-26T04:17:42.501 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-rbd (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:42.627 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up rbd-fuse (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:42.728 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-rgw (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:42.773 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-26T04:17:42.828 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-cephfs (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:42.930 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up ceph-common (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:43.049 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up lua-any (27ubuntu1) ... 2024-04-26T04:17:43.175 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-26T04:17:43.434 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-26T04:17:43.472 INFO:teuthology.orchestra.run.smithi063.stdout:Adding group ceph....done 2024-04-26T04:17:43.560 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-26T04:17:43.865 INFO:teuthology.orchestra.run.smithi063.stdout:Adding system user ceph....done 2024-04-26T04:17:43.886 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-26T04:17:43.975 INFO:teuthology.orchestra.run.smithi063.stdout:Setting system user ceph properties....done 2024-04-26T04:17:43.988 INFO:teuthology.orchestra.run.smithi063.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-26T04:17:44.143 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-26T04:17:44.171 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-26T04:17:44.422 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-26T04:17:44.510 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-26T04:17:44.548 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-26T04:17:44.815 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-26T04:17:44.947 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up ceph-test (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:45.073 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up radosgw (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:45.101 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-26T04:17:45.495 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-26T04:17:45.539 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-26T04:17:45.539 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-26T04:17:45.880 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-26T04:17:46.033 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up ceph-base (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:46.165 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-26T04:17:46.402 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-26T04:17:46.500 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-26T04:17:46.629 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up librados2 (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:46.730 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libsqlite3-mod-ceph (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:46.856 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-04-26T04:17:46.885 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up ceph-mds (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:47.133 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-26T04:17:47.133 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-26T04:17:47.192 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libcephfs2 (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:47.326 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libradosstriper1 (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:47.452 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-26T04:17:47.562 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up ceph-mgr (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:47.737 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up librbd1 (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:47.788 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-26T04:17:47.789 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-26T04:17:47.846 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph-mgr-modules-core (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:47.964 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph-fuse (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:48.178 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-26T04:17:48.179 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-26T04:17:48.246 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up ceph-osd (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:48.533 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-26T04:17:48.533 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-26T04:17:48.699 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up libcephfs-dev (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:48.850 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-rados (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:49.001 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up librgw2 (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:49.040 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up ceph-mgr-k8sevents (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:49.127 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-26T04:17:49.141 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:49.292 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up ceph-mon (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:49.509 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-26T04:17:49.510 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-26T04:17:49.952 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up ceph-mgr-cephadm (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:50.103 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up ceph (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:50.229 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up ceph-mgr-dashboard (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:50.380 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up ceph-volume (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:50.416 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-rbd (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:50.542 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up rbd-fuse (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:50.668 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-rgw (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:50.741 INFO:teuthology.orchestra.run.smithi063.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-26T04:17:50.794 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up python3-cephfs (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:50.833 INFO:teuthology.orchestra.run.smithi063.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-26T04:17:50.920 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph-common (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:50.994 INFO:teuthology.orchestra.run.smithi063.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-26T04:17:51.421 INFO:teuthology.orchestra.run.smithi186.stdout:Adding group ceph....done 2024-04-26T04:17:51.815 INFO:teuthology.orchestra.run.smithi186.stdout:Adding system user ceph....done 2024-04-26T04:17:51.974 INFO:teuthology.orchestra.run.smithi186.stdout:Setting system user ceph properties....done 2024-04-26T04:17:51.987 INFO:teuthology.orchestra.run.smithi186.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-26T04:17:52.125 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-26T04:17:52.153 INFO:teuthology.orchestra.run.smithi063.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-26T04:17:52.467 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-26T04:17:52.797 INFO:teuthology.orchestra.run.smithi063.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-26T04:17:52.802 DEBUG:teuthology.parallel:result is None 2024-04-26T04:17:52.938 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph-test (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:53.055 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up radosgw (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:53.522 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-26T04:17:53.522 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-26T04:17:53.982 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph-base (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:54.327 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-26T04:17:54.817 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph-mds (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:55.041 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-26T04:17:55.042 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-26T04:17:55.502 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph-mgr (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:55.717 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-26T04:17:55.717 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-26T04:17:56.195 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph-osd (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:56.474 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-26T04:17:56.475 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-26T04:17:56.972 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph-mgr-k8sevents (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:57.090 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:57.224 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph-mon (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:57.421 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-26T04:17:57.421 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-26T04:17:57.892 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph-mgr-cephadm (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:58.034 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:58.282 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph-mgr-dashboard (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:58.712 INFO:teuthology.orchestra.run.smithi186.stdout:Setting up ceph-volume (19.0.0-2457-gddeaa917-1jammy) ... 2024-04-26T04:17:59.056 INFO:teuthology.orchestra.run.smithi186.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-26T04:17:59.166 INFO:teuthology.orchestra.run.smithi186.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-26T04:17:59.301 INFO:teuthology.orchestra.run.smithi186.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-26T04:18:00.118 INFO:teuthology.orchestra.run.smithi186.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-26T04:18:00.765 INFO:teuthology.orchestra.run.smithi186.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-26T04:18:00.770 DEBUG:teuthology.parallel:result is None 2024-04-26T04:18:00.771 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=ddeaa91797339a19172c3036ff48cca58c12f448 2024-04-26T04:18:00.910 DEBUG:teuthology.orchestra.run.smithi063:> dpkg-query -W -f '${Version}' ceph 2024-04-26T04:18:00.931 INFO:teuthology.orchestra.run.smithi063.stdout:19.0.0-2457-gddeaa917-1jammy 2024-04-26T04:18:00.931 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2457-gddeaa917-1jammy 2024-04-26T04:18:00.931 INFO:teuthology.task.install:The correct ceph version 19.0.0-2457-gddeaa917-1jammy is installed. 2024-04-26T04:18:00.933 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=ddeaa91797339a19172c3036ff48cca58c12f448 2024-04-26T04:18:01.309 DEBUG:teuthology.orchestra.run.smithi186:> dpkg-query -W -f '${Version}' ceph 2024-04-26T04:18:01.328 INFO:teuthology.orchestra.run.smithi186.stdout:19.0.0-2457-gddeaa917-1jammy 2024-04-26T04:18:01.328 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2457-gddeaa917-1jammy 2024-04-26T04:18:01.328 INFO:teuthology.task.install:The correct ceph version 19.0.0-2457-gddeaa917-1jammy is installed. 2024-04-26T04:18:01.330 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-26T04:18:01.330 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-26T04:18:01.330 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-26T04:18:01.345 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-26T04:18:01.345 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-26T04:18:01.384 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-26T04:18:01.384 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-26T04:18:01.384 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/usr/bin/daemon-helper 2024-04-26T04:18:01.402 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-26T04:18:01.458 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-26T04:18:01.459 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/usr/bin/daemon-helper 2024-04-26T04:18:01.472 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-26T04:18:01.527 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-26T04:18:01.527 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-26T04:18:01.527 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-26T04:18:01.541 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-26T04:18:01.594 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-26T04:18:01.594 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-26T04:18:01.606 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-26T04:18:01.659 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-26T04:18:01.660 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-26T04:18:01.660 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/usr/bin/stdin-killer 2024-04-26T04:18:01.673 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-26T04:18:01.730 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-26T04:18:01.730 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/usr/bin/stdin-killer 2024-04-26T04:18:01.743 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-26T04:18:01.799 INFO:teuthology.run_tasks:Running task ceph... 2024-04-26T04:18:01.886 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-26T04:18:01.886 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod 777 /var/log/ceph 2024-04-26T04:18:01.889 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 777 /var/log/ceph 2024-04-26T04:18:01.901 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-26T04:18:01.901 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-26T04:18:01.943 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-26T04:18:01.955 INFO:tasks.ceph:Creating extra log directories... 2024-04-26T04:18:01.955 DEBUG:teuthology.orchestra.run.smithi063:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-26T04:18:01.999 DEBUG:teuthology.orchestra.run.smithi186:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-26T04:18:02.021 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-26T04:18:02.021 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos min': 10, 'paxos service trim min': 5, 'paxos trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'slow request', 'overall HEALTH_'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-26T04:18:02.022 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2024-04-26_01:23:56-rados-wip-yuri2-testing-2024-04-24-0914-squid-distro-default-smithi/7674515', 'branch': 'wip-yuri2-testing-2024-04-24-0914-squid', 'description': 'rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/sync-many workloads/pool-create-delete}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7674515', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2024-04-26_01:23:56-rados-wip-yuri2-testing-2024-04-24-0914-squid-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-yuri2-testing-2024-04-24-0914-squid'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos min': 10, 'paxos service trim min': 5, 'paxos trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'slow request', 'overall HEALTH_'], 'sha1': 'ddeaa91797339a19172c3036ff48cca58c12f448'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'ddeaa91797339a19172c3036ff48cca58c12f448'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri2-testing-2024-04-24-0914-squid', 'sha1': 'ddeaa91797339a19172c3036ff48cca58c12f448'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 99, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'], ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0']], 'seed': 1848, 'sha1': 'ddeaa91797339a19172c3036ff48cca58c12f448', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-yuri2-testing-2024-04-24-0914-squid', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_ddeaa91797339a19172c3036ff48cca58c12f448/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'ddeaa91797339a19172c3036ff48cca58c12f448', 'targets': {'smithi063.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMPjy2nGYyGpIxXaEuFqjiO8JiR0ZipLFjyWlr1wVxKA4aYKg0370mBvmLKfaTm2JcV/i8NkujatTa7hECUx7YM=', 'smithi186.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH8Pvzwaklc3FDhfrA4zSkFmK1Fjomd9nV+PpvdXoNCYyIKQ1F5pwid2lnW8OR0aEiG3ibraLMdukNIoa52S1/w='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'mon_thrash': {'revive_delay': 90, 'thrash_delay': 1, 'thrash_many': True}}, {'exec': {'client.0': ['ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17', 'timestamp': '2024-04-26_01:23:56', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-26T04:18:02.022 DEBUG:teuthology.orchestra.run.smithi063:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-26T04:18:02.056 DEBUG:teuthology.orchestra.run.smithi186:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-26T04:18:02.061 DEBUG:teuthology.orchestra.run.smithi063:> sudo install -d -m0777 -- /var/run/ceph 2024-04-26T04:18:02.103 DEBUG:teuthology.orchestra.run.smithi186:> sudo install -d -m0777 -- /var/run/ceph 2024-04-26T04:18:02.116 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-26T04:18:02.116 DEBUG:teuthology.orchestra.run.smithi063:> dd if=/scratch_devs of=/dev/stdout 2024-04-26T04:18:02.163 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-26T04:18:02.163 DEBUG:teuthology.orchestra.run.smithi063:> stat /dev/vg_nvme/lv_1 2024-04-26T04:18:02.211 INFO:teuthology.orchestra.run.smithi063.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-26T04:18:02.211 INFO:teuthology.orchestra.run.smithi063.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T04:18:02.211 INFO:teuthology.orchestra.run.smithi063.stdout:Device: 5h/5d Inode: 886 Links: 1 2024-04-26T04:18:02.211 INFO:teuthology.orchestra.run.smithi063.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T04:18:02.211 INFO:teuthology.orchestra.run.smithi063.stdout:Access: 2024-04-26 04:12:39.294167444 +0000 2024-04-26T04:18:02.211 INFO:teuthology.orchestra.run.smithi063.stdout:Modify: 2024-04-26 04:12:39.042169696 +0000 2024-04-26T04:18:02.211 INFO:teuthology.orchestra.run.smithi063.stdout:Change: 2024-04-26 04:12:39.042169696 +0000 2024-04-26T04:18:02.211 INFO:teuthology.orchestra.run.smithi063.stdout: Birth: - 2024-04-26T04:18:02.212 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-26T04:18:02.266 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records in 2024-04-26T04:18:02.266 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records out 2024-04-26T04:18:02.266 INFO:teuthology.orchestra.run.smithi063.stderr:512 bytes copied, 0.000368535 s, 1.4 MB/s 2024-04-26T04:18:02.267 DEBUG:teuthology.orchestra.run.smithi063:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-26T04:18:02.316 DEBUG:teuthology.orchestra.run.smithi063:> stat /dev/vg_nvme/lv_2 2024-04-26T04:18:02.362 INFO:teuthology.orchestra.run.smithi063.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-26T04:18:02.362 INFO:teuthology.orchestra.run.smithi063.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T04:18:02.362 INFO:teuthology.orchestra.run.smithi063.stdout:Device: 5h/5d Inode: 906 Links: 1 2024-04-26T04:18:02.362 INFO:teuthology.orchestra.run.smithi063.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T04:18:02.363 INFO:teuthology.orchestra.run.smithi063.stdout:Access: 2024-04-26 04:12:39.534165301 +0000 2024-04-26T04:18:02.363 INFO:teuthology.orchestra.run.smithi063.stdout:Modify: 2024-04-26 04:12:39.530165336 +0000 2024-04-26T04:18:02.363 INFO:teuthology.orchestra.run.smithi063.stdout:Change: 2024-04-26 04:12:39.530165336 +0000 2024-04-26T04:18:02.363 INFO:teuthology.orchestra.run.smithi063.stdout: Birth: - 2024-04-26T04:18:02.363 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-26T04:18:02.416 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records in 2024-04-26T04:18:02.416 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records out 2024-04-26T04:18:02.417 INFO:teuthology.orchestra.run.smithi063.stderr:512 bytes copied, 0.000310105 s, 1.7 MB/s 2024-04-26T04:18:02.418 DEBUG:teuthology.orchestra.run.smithi063:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-26T04:18:02.467 DEBUG:teuthology.orchestra.run.smithi063:> stat /dev/vg_nvme/lv_3 2024-04-26T04:18:02.514 INFO:teuthology.orchestra.run.smithi063.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-26T04:18:02.514 INFO:teuthology.orchestra.run.smithi063.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T04:18:02.514 INFO:teuthology.orchestra.run.smithi063.stdout:Device: 5h/5d Inode: 923 Links: 1 2024-04-26T04:18:02.515 INFO:teuthology.orchestra.run.smithi063.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T04:18:02.515 INFO:teuthology.orchestra.run.smithi063.stdout:Access: 2024-04-26 04:12:40.326158227 +0000 2024-04-26T04:18:02.515 INFO:teuthology.orchestra.run.smithi063.stdout:Modify: 2024-04-26 04:12:40.050160692 +0000 2024-04-26T04:18:02.515 INFO:teuthology.orchestra.run.smithi063.stdout:Change: 2024-04-26 04:12:40.050160692 +0000 2024-04-26T04:18:02.515 INFO:teuthology.orchestra.run.smithi063.stdout: Birth: - 2024-04-26T04:18:02.515 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-26T04:18:02.569 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records in 2024-04-26T04:18:02.569 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records out 2024-04-26T04:18:02.569 INFO:teuthology.orchestra.run.smithi063.stderr:512 bytes copied, 0.000432573 s, 1.2 MB/s 2024-04-26T04:18:02.570 DEBUG:teuthology.orchestra.run.smithi063:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-26T04:18:02.619 DEBUG:teuthology.orchestra.run.smithi063:> stat /dev/vg_nvme/lv_4 2024-04-26T04:18:02.666 INFO:teuthology.orchestra.run.smithi063.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-26T04:18:02.666 INFO:teuthology.orchestra.run.smithi063.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T04:18:02.666 INFO:teuthology.orchestra.run.smithi063.stdout:Device: 5h/5d Inode: 938 Links: 1 2024-04-26T04:18:02.667 INFO:teuthology.orchestra.run.smithi063.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T04:18:02.667 INFO:teuthology.orchestra.run.smithi063.stdout:Access: 2024-04-26 04:12:40.966152511 +0000 2024-04-26T04:18:02.667 INFO:teuthology.orchestra.run.smithi063.stdout:Modify: 2024-04-26 04:12:40.694154940 +0000 2024-04-26T04:18:02.667 INFO:teuthology.orchestra.run.smithi063.stdout:Change: 2024-04-26 04:12:40.694154940 +0000 2024-04-26T04:18:02.667 INFO:teuthology.orchestra.run.smithi063.stdout: Birth: - 2024-04-26T04:18:02.667 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-26T04:18:02.721 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records in 2024-04-26T04:18:02.721 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records out 2024-04-26T04:18:02.721 INFO:teuthology.orchestra.run.smithi063.stderr:512 bytes copied, 0.000377144 s, 1.4 MB/s 2024-04-26T04:18:02.722 DEBUG:teuthology.orchestra.run.smithi063:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-26T04:18:02.772 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-26T04:18:02.772 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-26T04:18:02.772 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/scratch_devs of=/dev/stdout 2024-04-26T04:18:02.778 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-26T04:18:02.778 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_1 2024-04-26T04:18:02.824 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-26T04:18:02.824 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T04:18:02.825 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 883 Links: 1 2024-04-26T04:18:02.825 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T04:18:02.825 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-04-26 04:12:36.457745761 +0000 2024-04-26T04:18:02.825 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-04-26 04:12:36.453745795 +0000 2024-04-26T04:18:02.825 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-04-26 04:12:36.453745795 +0000 2024-04-26T04:18:02.825 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: - 2024-04-26T04:18:02.825 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-26T04:18:02.879 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-04-26T04:18:02.879 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-04-26T04:18:02.879 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000448119 s, 1.1 MB/s 2024-04-26T04:18:02.880 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-26T04:18:02.929 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_2 2024-04-26T04:18:02.976 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-26T04:18:02.976 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T04:18:02.976 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 905 Links: 1 2024-04-26T04:18:02.976 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T04:18:02.976 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-04-26 04:12:37.229738865 +0000 2024-04-26T04:18:02.976 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-04-26 04:12:36.973741152 +0000 2024-04-26T04:18:02.977 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-04-26 04:12:36.973741152 +0000 2024-04-26T04:18:02.977 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: - 2024-04-26T04:18:02.977 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-26T04:18:03.030 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-04-26T04:18:03.031 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-04-26T04:18:03.031 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000342751 s, 1.5 MB/s 2024-04-26T04:18:03.032 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-26T04:18:03.082 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_3 2024-04-26T04:18:03.128 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-26T04:18:03.128 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T04:18:03.128 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 920 Links: 1 2024-04-26T04:18:03.128 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T04:18:03.128 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-04-26 04:12:37.725734435 +0000 2024-04-26T04:18:03.128 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-04-26 04:12:37.461736792 +0000 2024-04-26T04:18:03.128 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-04-26 04:12:37.461736792 +0000 2024-04-26T04:18:03.128 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: - 2024-04-26T04:18:03.129 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-26T04:18:03.183 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-04-26T04:18:03.183 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-04-26T04:18:03.183 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.00043077 s, 1.2 MB/s 2024-04-26T04:18:03.184 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-26T04:18:03.234 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_4 2024-04-26T04:18:03.280 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-26T04:18:03.281 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-26T04:18:03.281 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 939 Links: 1 2024-04-26T04:18:03.281 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-26T04:18:03.281 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-04-26 04:12:38.209730112 +0000 2024-04-26T04:18:03.281 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-04-26 04:12:37.961732327 +0000 2024-04-26T04:18:03.281 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-04-26 04:12:37.961732327 +0000 2024-04-26T04:18:03.281 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: - 2024-04-26T04:18:03.281 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-26T04:18:03.334 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-04-26T04:18:03.335 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-04-26T04:18:03.335 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000365641 s, 1.4 MB/s 2024-04-26T04:18:03.336 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-26T04:18:03.386 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-04-26T04:18:03.386 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi063.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi186.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-04-26T04:18:03.386 INFO:tasks.ceph:Generating config... 2024-04-26T04:18:03.387 INFO:tasks.ceph:[client] debug monc = 20 2024-04-26T04:18:03.387 INFO:tasks.ceph:[client] debug ms = 1 2024-04-26T04:18:03.388 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-04-26T04:18:03.388 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-04-26T04:18:03.388 INFO:tasks.ceph:[global] ms inject delay max = 1 2024-04-26T04:18:03.388 INFO:tasks.ceph:[global] ms inject delay probability = 0.005 2024-04-26T04:18:03.388 INFO:tasks.ceph:[global] ms inject delay type = mon 2024-04-26T04:18:03.388 INFO:tasks.ceph:[global] ms inject internal delays = 0.002 2024-04-26T04:18:03.388 INFO:tasks.ceph:[global] ms inject socket failures = 2500 2024-04-26T04:18:03.388 INFO:tasks.ceph:[global] ms type = async 2024-04-26T04:18:03.388 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-26T04:18:03.388 INFO:tasks.ceph:[mgr] debug monc = 10 2024-04-26T04:18:03.388 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-26T04:18:03.388 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-26T04:18:03.388 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-26T04:18:03.388 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-26T04:18:03.389 INFO:tasks.ceph:[mon] mon min osdmap epochs = 25 2024-04-26T04:18:03.389 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-04-26T04:18:03.389 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-04-26T04:18:03.389 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-04-26T04:18:03.389 INFO:tasks.ceph:[mon] mon scrub inject crc mismatch = 0.01 2024-04-26T04:18:03.389 INFO:tasks.ceph:[mon] mon scrub inject missing keys = 0.05 2024-04-26T04:18:03.389 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-04-26T04:18:03.389 INFO:tasks.ceph:[mon] paxos min = 10 2024-04-26T04:18:03.389 INFO:tasks.ceph:[mon] paxos service trim min = 5 2024-04-26T04:18:03.389 INFO:tasks.ceph:[mon] paxos trim min = 10 2024-04-26T04:18:03.389 INFO:tasks.ceph:[osd] bdev async discard = True 2024-04-26T04:18:03.389 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-04-26T04:18:03.389 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-04-26T04:18:03.389 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-26T04:18:03.390 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-26T04:18:03.390 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-04-26T04:18:03.390 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-04-26T04:18:03.390 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-04-26T04:18:03.390 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-26T04:18:03.390 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-26T04:18:03.390 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-04-26T04:18:03.390 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-26T04:18:03.390 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-26T04:18:03.390 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-26T04:18:03.390 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-04-26T04:18:03.390 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-04-26T04:18:03.390 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-26T04:18:03.391 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-04-26T04:18:03.391 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-04-26T04:18:03.391 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-26T04:18:03.391 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-04-26T04:18:03.391 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-04-26T04:18:03.391 INFO:tasks.ceph:Setting up mon.a... 2024-04-26T04:18:03.391 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-26T04:18:03.426 INFO:teuthology.orchestra.run.smithi063.stdout:creating /etc/ceph/ceph.keyring 2024-04-26T04:18:03.436 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-26T04:18:03.516 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-26T04:18:03.569 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.63'), ('mon.c', '[v2:172.21.15.63:3301,v1:172.21.15.63:6790]'), ('mon.b', '172.21.15.186')] 2024-04-26T04:18:03.569 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.63,[v2:172.21.15.63:3301,v1:172.21.15.63:6790],172.21.15.186', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug monc': 10}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos min': 10, 'paxos service trim min': 5, 'paxos trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-04-26T04:18:03.570 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-26T04:18:03.570 DEBUG:teuthology.orchestra.run.smithi063:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-26T04:18:03.614 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.63 --addv c '[v2:172.21.15.63:3301,v1:172.21.15.63:6790]' --add b 172.21.15.186 --print /home/ubuntu/cephtest/ceph.monmap 2024-04-26T04:18:03.702 INFO:teuthology.orchestra.run.smithi063.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-26T04:18:03.702 INFO:teuthology.orchestra.run.smithi063.stdout:monmaptool: generated fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:03.702 INFO:teuthology.orchestra.run.smithi063.stdout:setting min_mon_release = quincy 2024-04-26T04:18:03.702 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 0 2024-04-26T04:18:03.702 INFO:teuthology.orchestra.run.smithi063.stdout:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:03.703 INFO:teuthology.orchestra.run.smithi063.stdout:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:03.703 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:03.703 INFO:teuthology.orchestra.run.smithi063.stdout:min_mon_release 17 (quincy) 2024-04-26T04:18:03.703 INFO:teuthology.orchestra.run.smithi063.stdout:election_strategy: 1 2024-04-26T04:18:03.703 INFO:teuthology.orchestra.run.smithi063.stdout:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:03.703 INFO:teuthology.orchestra.run.smithi063.stdout:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:03.703 INFO:teuthology.orchestra.run.smithi063.stdout:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:03.703 INFO:teuthology.orchestra.run.smithi063.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-04-26T04:18:03.704 DEBUG:teuthology.orchestra.run.smithi063:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-26T04:18:03.709 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID cd1c6a0c-4e4b-4014-89c0-2e535479ef39... 2024-04-26T04:18:03.710 DEBUG:teuthology.orchestra.run.smithi063:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-26T04:18:03.755 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-26T04:18:03.779 INFO:teuthology.orchestra.run.smithi063.stdout:[global] 2024-04-26T04:18:03.779 INFO:teuthology.orchestra.run.smithi063.stdout: chdir = "" 2024-04-26T04:18:03.779 INFO:teuthology.orchestra.run.smithi063.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-26T04:18:03.779 INFO:teuthology.orchestra.run.smithi063.stdout: auth supported = cephx 2024-04-26T04:18:03.779 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-26T04:18:03.779 INFO:teuthology.orchestra.run.smithi063.stdout: filestore xattr use omap = true 2024-04-26T04:18:03.779 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-26T04:18:03.779 INFO:teuthology.orchestra.run.smithi063.stdout: mon clock drift allowed = 1.000 2024-04-26T04:18:03.780 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-26T04:18:03.780 INFO:teuthology.orchestra.run.smithi063.stdout: osd crush chooseleaf type = 0 2024-04-26T04:18:03.780 INFO:teuthology.orchestra.run.smithi063.stdout: auth debug = true 2024-04-26T04:18:03.780 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-26T04:18:03.780 INFO:teuthology.orchestra.run.smithi063.stdout: ms die on old message = true 2024-04-26T04:18:03.780 INFO:teuthology.orchestra.run.smithi063.stdout: ms die on bug = true 2024-04-26T04:18:03.780 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-26T04:18:03.780 INFO:teuthology.orchestra.run.smithi063.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-26T04:18:03.780 INFO:teuthology.orchestra.run.smithi063.stdout: mon pg warn max object skew = 0 2024-04-26T04:18:03.780 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-26T04:18:03.780 INFO:teuthology.orchestra.run.smithi063.stdout: # disable pg_autoscaler by default for new pools 2024-04-26T04:18:03.780 INFO:teuthology.orchestra.run.smithi063.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-26T04:18:03.780 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-26T04:18:03.780 INFO:teuthology.orchestra.run.smithi063.stdout: osd pool default size = 2 2024-04-26T04:18:03.781 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-26T04:18:03.781 INFO:teuthology.orchestra.run.smithi063.stdout: mon osd allow primary affinity = true 2024-04-26T04:18:03.781 INFO:teuthology.orchestra.run.smithi063.stdout: mon osd allow pg remap = true 2024-04-26T04:18:03.781 INFO:teuthology.orchestra.run.smithi063.stdout: mon warn on legacy crush tunables = false 2024-04-26T04:18:03.781 INFO:teuthology.orchestra.run.smithi063.stdout: mon warn on crush straw calc version zero = false 2024-04-26T04:18:03.781 INFO:teuthology.orchestra.run.smithi063.stdout: mon warn on no sortbitwise = false 2024-04-26T04:18:03.781 INFO:teuthology.orchestra.run.smithi063.stdout: mon warn on osd down out interval zero = false 2024-04-26T04:18:03.781 INFO:teuthology.orchestra.run.smithi063.stdout: mon warn on too few osds = false 2024-04-26T04:18:03.781 INFO:teuthology.orchestra.run.smithi063.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-26T04:18:03.781 INFO:teuthology.orchestra.run.smithi063.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-26T04:18:03.781 INFO:teuthology.orchestra.run.smithi063.stdout: mon_allow_pool_size_one = true 2024-04-26T04:18:03.781 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-26T04:18:03.781 INFO:teuthology.orchestra.run.smithi063.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-26T04:18:03.781 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-26T04:18:03.782 INFO:teuthology.orchestra.run.smithi063.stdout: osd default data pool replay window = 5 2024-04-26T04:18:03.782 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-26T04:18:03.782 INFO:teuthology.orchestra.run.smithi063.stdout: mon allow pool delete = true 2024-04-26T04:18:03.782 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-26T04:18:03.782 INFO:teuthology.orchestra.run.smithi063.stdout: mon cluster log file level = debug 2024-04-26T04:18:03.782 INFO:teuthology.orchestra.run.smithi063.stdout: debug asserts on shutdown = true 2024-04-26T04:18:03.782 INFO:teuthology.orchestra.run.smithi063.stdout: mon health detail to clog = false 2024-04-26T04:18:03.782 INFO:teuthology.orchestra.run.smithi063.stdout: mon host = "172.21.15.63,[v2:172.21.15.63:3301,v1:172.21.15.63:6790],172.21.15.186" 2024-04-26T04:18:03.782 INFO:teuthology.orchestra.run.smithi063.stdout: mon client directed command retry = 5 2024-04-26T04:18:03.782 INFO:teuthology.orchestra.run.smithi063.stdout: mon election default strategy = 1 2024-04-26T04:18:03.782 INFO:teuthology.orchestra.run.smithi063.stdout: ms inject delay max = 1 2024-04-26T04:18:03.782 INFO:teuthology.orchestra.run.smithi063.stdout: ms inject delay probability = 0.005 2024-04-26T04:18:03.782 INFO:teuthology.orchestra.run.smithi063.stdout: ms inject delay type = mon 2024-04-26T04:18:03.782 INFO:teuthology.orchestra.run.smithi063.stdout: ms inject internal delays = 0.002 2024-04-26T04:18:03.783 INFO:teuthology.orchestra.run.smithi063.stdout: ms inject socket failures = 2500 2024-04-26T04:18:03.783 INFO:teuthology.orchestra.run.smithi063.stdout: ms type = async 2024-04-26T04:18:03.783 INFO:teuthology.orchestra.run.smithi063.stdout: fsid = cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:03.783 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-26T04:18:03.783 INFO:teuthology.orchestra.run.smithi063.stdout:[osd] 2024-04-26T04:18:03.783 INFO:teuthology.orchestra.run.smithi063.stdout: osd journal size = 100 2024-04-26T04:18:03.783 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-26T04:18:03.783 INFO:teuthology.orchestra.run.smithi063.stdout: osd scrub load threshold = 5.0 2024-04-26T04:18:03.783 INFO:teuthology.orchestra.run.smithi063.stdout: osd scrub max interval = 600 2024-04-26T04:18:03.783 INFO:teuthology.orchestra.run.smithi063.stdout: osd mclock profile = high_recovery_ops 2024-04-26T04:18:03.783 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-26T04:18:03.783 INFO:teuthology.orchestra.run.smithi063.stdout: osd recover clone overlap = true 2024-04-26T04:18:03.783 INFO:teuthology.orchestra.run.smithi063.stdout: osd recovery max chunk = 1048576 2024-04-26T04:18:03.783 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-26T04:18:03.784 INFO:teuthology.orchestra.run.smithi063.stdout: osd debug shutdown = true 2024-04-26T04:18:03.784 INFO:teuthology.orchestra.run.smithi063.stdout: osd debug op order = true 2024-04-26T04:18:03.784 INFO:teuthology.orchestra.run.smithi063.stdout: osd debug verify stray on activate = true 2024-04-26T04:18:03.784 INFO:teuthology.orchestra.run.smithi063.stdout: osd debug trim objects = true 2024-04-26T04:18:03.784 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-26T04:18:03.784 INFO:teuthology.orchestra.run.smithi063.stdout: osd open classes on start = true 2024-04-26T04:18:03.784 INFO:teuthology.orchestra.run.smithi063.stdout: osd debug pg log writeout = true 2024-04-26T04:18:03.784 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-26T04:18:03.784 INFO:teuthology.orchestra.run.smithi063.stdout: osd deep scrub update digest min age = 30 2024-04-26T04:18:03.784 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-26T04:18:03.784 INFO:teuthology.orchestra.run.smithi063.stdout: osd map max advance = 10 2024-04-26T04:18:03.784 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-26T04:18:03.784 INFO:teuthology.orchestra.run.smithi063.stdout: journal zero on create = true 2024-04-26T04:18:03.784 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-26T04:18:03.784 INFO:teuthology.orchestra.run.smithi063.stdout: filestore ondisk finisher threads = 3 2024-04-26T04:18:03.785 INFO:teuthology.orchestra.run.smithi063.stdout: filestore apply finisher threads = 3 2024-04-26T04:18:03.785 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-26T04:18:03.785 INFO:teuthology.orchestra.run.smithi063.stdout: bdev debug aio = true 2024-04-26T04:18:03.785 INFO:teuthology.orchestra.run.smithi063.stdout: osd debug misdirected ops = true 2024-04-26T04:18:03.785 INFO:teuthology.orchestra.run.smithi063.stdout: bdev async discard = True 2024-04-26T04:18:03.785 INFO:teuthology.orchestra.run.smithi063.stdout: bdev enable discard = True 2024-04-26T04:18:03.785 INFO:teuthology.orchestra.run.smithi063.stdout: bluestore allocator = bitmap 2024-04-26T04:18:03.785 INFO:teuthology.orchestra.run.smithi063.stdout: bluestore block size = 96636764160 2024-04-26T04:18:03.785 INFO:teuthology.orchestra.run.smithi063.stdout: bluestore fsck on mount = True 2024-04-26T04:18:03.785 INFO:teuthology.orchestra.run.smithi063.stdout: bluestore zero block detection = True 2024-04-26T04:18:03.785 INFO:teuthology.orchestra.run.smithi063.stdout: debug bluefs = 20 2024-04-26T04:18:03.785 INFO:teuthology.orchestra.run.smithi063.stdout: debug bluestore = 20 2024-04-26T04:18:03.785 INFO:teuthology.orchestra.run.smithi063.stdout: debug ms = 1 2024-04-26T04:18:03.785 INFO:teuthology.orchestra.run.smithi063.stdout: debug osd = 20 2024-04-26T04:18:03.786 INFO:teuthology.orchestra.run.smithi063.stdout: debug rocksdb = 10 2024-04-26T04:18:03.786 INFO:teuthology.orchestra.run.smithi063.stdout: mon osd backfillfull_ratio = 0.85 2024-04-26T04:18:03.786 INFO:teuthology.orchestra.run.smithi063.stdout: mon osd full ratio = 0.9 2024-04-26T04:18:03.786 INFO:teuthology.orchestra.run.smithi063.stdout: mon osd nearfull ratio = 0.8 2024-04-26T04:18:03.786 INFO:teuthology.orchestra.run.smithi063.stdout: osd debug verify cached snaps = True 2024-04-26T04:18:03.786 INFO:teuthology.orchestra.run.smithi063.stdout: osd debug verify missing on start = True 2024-04-26T04:18:03.786 INFO:teuthology.orchestra.run.smithi063.stdout: osd failsafe full ratio = 0.95 2024-04-26T04:18:03.786 INFO:teuthology.orchestra.run.smithi063.stdout: osd mclock override recovery settings = True 2024-04-26T04:18:03.786 INFO:teuthology.orchestra.run.smithi063.stdout: osd objectstore = bluestore 2024-04-26T04:18:03.786 INFO:teuthology.orchestra.run.smithi063.stdout: osd op queue = debug_random 2024-04-26T04:18:03.786 INFO:teuthology.orchestra.run.smithi063.stdout: osd op queue cut off = debug_random 2024-04-26T04:18:03.786 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-26T04:18:03.786 INFO:teuthology.orchestra.run.smithi063.stdout:[mgr] 2024-04-26T04:18:03.786 INFO:teuthology.orchestra.run.smithi063.stdout: debug ms = 1 2024-04-26T04:18:03.787 INFO:teuthology.orchestra.run.smithi063.stdout: debug mgr = 20 2024-04-26T04:18:03.787 INFO:teuthology.orchestra.run.smithi063.stdout: debug mon = 20 2024-04-26T04:18:03.787 INFO:teuthology.orchestra.run.smithi063.stdout: debug auth = 20 2024-04-26T04:18:03.787 INFO:teuthology.orchestra.run.smithi063.stdout: mon reweight min pgs per osd = 4 2024-04-26T04:18:03.787 INFO:teuthology.orchestra.run.smithi063.stdout: mon reweight min bytes per osd = 10 2024-04-26T04:18:03.787 INFO:teuthology.orchestra.run.smithi063.stdout: mgr/telemetry/nag = false 2024-04-26T04:18:03.787 INFO:teuthology.orchestra.run.smithi063.stdout: debug monc = 10 2024-04-26T04:18:03.787 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-26T04:18:03.787 INFO:teuthology.orchestra.run.smithi063.stdout:[mon] 2024-04-26T04:18:03.787 INFO:teuthology.orchestra.run.smithi063.stdout: debug ms = 1 2024-04-26T04:18:03.787 INFO:teuthology.orchestra.run.smithi063.stdout: debug mon = 20 2024-04-26T04:18:03.787 INFO:teuthology.orchestra.run.smithi063.stdout: debug paxos = 20 2024-04-26T04:18:03.787 INFO:teuthology.orchestra.run.smithi063.stdout: debug auth = 20 2024-04-26T04:18:03.787 INFO:teuthology.orchestra.run.smithi063.stdout: mon data avail warn = 5 2024-04-26T04:18:03.787 INFO:teuthology.orchestra.run.smithi063.stdout: mon mgr mkfs grace = 240 2024-04-26T04:18:03.788 INFO:teuthology.orchestra.run.smithi063.stdout: mon reweight min pgs per osd = 4 2024-04-26T04:18:03.788 INFO:teuthology.orchestra.run.smithi063.stdout: mon osd reporter subtree level = osd 2024-04-26T04:18:03.788 INFO:teuthology.orchestra.run.smithi063.stdout: mon osd prime pg temp = true 2024-04-26T04:18:03.788 INFO:teuthology.orchestra.run.smithi063.stdout: mon reweight min bytes per osd = 10 2024-04-26T04:18:03.788 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-26T04:18:03.788 INFO:teuthology.orchestra.run.smithi063.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-26T04:18:03.788 INFO:teuthology.orchestra.run.smithi063.stdout: auth mon ticket ttl = 660 # 11m 2024-04-26T04:18:03.788 INFO:teuthology.orchestra.run.smithi063.stdout: auth service ticket ttl = 240 # 4m 2024-04-26T04:18:03.788 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-26T04:18:03.788 INFO:teuthology.orchestra.run.smithi063.stdout: # don't complain about insecure global_id in the test suite 2024-04-26T04:18:03.788 INFO:teuthology.orchestra.run.smithi063.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-26T04:18:03.788 INFO:teuthology.orchestra.run.smithi063.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-26T04:18:03.788 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-26T04:18:03.788 INFO:teuthology.orchestra.run.smithi063.stdout: # 1m isn't quite enough 2024-04-26T04:18:03.789 INFO:teuthology.orchestra.run.smithi063.stdout: mon_down_mkfs_grace = 2m 2024-04-26T04:18:03.789 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-26T04:18:03.789 INFO:teuthology.orchestra.run.smithi063.stdout: mon_warn_on_filestore_osds = false 2024-04-26T04:18:03.789 INFO:teuthology.orchestra.run.smithi063.stdout: mon min osdmap epochs = 25 2024-04-26T04:18:03.789 INFO:teuthology.orchestra.run.smithi063.stdout: mon osdmap full prune interval = 2 2024-04-26T04:18:03.789 INFO:teuthology.orchestra.run.smithi063.stdout: mon osdmap full prune min = 15 2024-04-26T04:18:03.789 INFO:teuthology.orchestra.run.smithi063.stdout: mon osdmap full prune txsize = 2 2024-04-26T04:18:03.789 INFO:teuthology.orchestra.run.smithi063.stdout: mon scrub inject crc mismatch = 0.01 2024-04-26T04:18:03.789 INFO:teuthology.orchestra.run.smithi063.stdout: mon scrub inject missing keys = 0.05 2024-04-26T04:18:03.789 INFO:teuthology.orchestra.run.smithi063.stdout: mon scrub interval = 300 2024-04-26T04:18:03.789 INFO:teuthology.orchestra.run.smithi063.stdout: paxos min = 10 2024-04-26T04:18:03.789 INFO:teuthology.orchestra.run.smithi063.stdout: paxos service trim min = 5 2024-04-26T04:18:03.789 INFO:teuthology.orchestra.run.smithi063.stdout: paxos trim min = 10 2024-04-26T04:18:03.789 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-26T04:18:03.789 INFO:teuthology.orchestra.run.smithi063.stdout:[client] 2024-04-26T04:18:03.790 INFO:teuthology.orchestra.run.smithi063.stdout: rgw cache enabled = true 2024-04-26T04:18:03.790 INFO:teuthology.orchestra.run.smithi063.stdout: rgw enable ops log = true 2024-04-26T04:18:03.790 INFO:teuthology.orchestra.run.smithi063.stdout: rgw enable usage log = true 2024-04-26T04:18:03.790 INFO:teuthology.orchestra.run.smithi063.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-26T04:18:03.790 INFO:teuthology.orchestra.run.smithi063.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-26T04:18:03.790 INFO:teuthology.orchestra.run.smithi063.stdout: debug monc = 20 2024-04-26T04:18:03.790 INFO:teuthology.orchestra.run.smithi063.stdout: debug ms = 1 2024-04-26T04:18:03.790 INFO:teuthology.orchestra.run.smithi063.stdout:[mon.a] 2024-04-26T04:18:03.790 INFO:teuthology.orchestra.run.smithi063.stdout:[mon.c] 2024-04-26T04:18:03.790 INFO:teuthology.orchestra.run.smithi063.stdout:[mon.b] 2024-04-26T04:18:03.791 INFO:teuthology.orchestra.run.smithi186.stdout:[global] 2024-04-26T04:18:03.791 INFO:teuthology.orchestra.run.smithi186.stdout: chdir = "" 2024-04-26T04:18:03.791 INFO:teuthology.orchestra.run.smithi186.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-26T04:18:03.791 INFO:teuthology.orchestra.run.smithi186.stdout: auth supported = cephx 2024-04-26T04:18:03.791 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T04:18:03.791 INFO:teuthology.orchestra.run.smithi186.stdout: filestore xattr use omap = true 2024-04-26T04:18:03.791 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T04:18:03.791 INFO:teuthology.orchestra.run.smithi186.stdout: mon clock drift allowed = 1.000 2024-04-26T04:18:03.792 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T04:18:03.792 INFO:teuthology.orchestra.run.smithi186.stdout: osd crush chooseleaf type = 0 2024-04-26T04:18:03.792 INFO:teuthology.orchestra.run.smithi186.stdout: auth debug = true 2024-04-26T04:18:03.792 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T04:18:03.792 INFO:teuthology.orchestra.run.smithi186.stdout: ms die on old message = true 2024-04-26T04:18:03.792 INFO:teuthology.orchestra.run.smithi186.stdout: ms die on bug = true 2024-04-26T04:18:03.792 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T04:18:03.792 INFO:teuthology.orchestra.run.smithi186.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-26T04:18:03.792 INFO:teuthology.orchestra.run.smithi186.stdout: mon pg warn max object skew = 0 2024-04-26T04:18:03.792 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T04:18:03.792 INFO:teuthology.orchestra.run.smithi186.stdout: # disable pg_autoscaler by default for new pools 2024-04-26T04:18:03.792 INFO:teuthology.orchestra.run.smithi186.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-26T04:18:03.792 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T04:18:03.792 INFO:teuthology.orchestra.run.smithi186.stdout: osd pool default size = 2 2024-04-26T04:18:03.793 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T04:18:03.793 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd allow primary affinity = true 2024-04-26T04:18:03.793 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd allow pg remap = true 2024-04-26T04:18:03.793 INFO:teuthology.orchestra.run.smithi186.stdout: mon warn on legacy crush tunables = false 2024-04-26T04:18:03.793 INFO:teuthology.orchestra.run.smithi186.stdout: mon warn on crush straw calc version zero = false 2024-04-26T04:18:03.793 INFO:teuthology.orchestra.run.smithi186.stdout: mon warn on no sortbitwise = false 2024-04-26T04:18:03.793 INFO:teuthology.orchestra.run.smithi186.stdout: mon warn on osd down out interval zero = false 2024-04-26T04:18:03.793 INFO:teuthology.orchestra.run.smithi186.stdout: mon warn on too few osds = false 2024-04-26T04:18:03.793 INFO:teuthology.orchestra.run.smithi186.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-26T04:18:03.793 INFO:teuthology.orchestra.run.smithi186.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-26T04:18:03.793 INFO:teuthology.orchestra.run.smithi186.stdout: mon_allow_pool_size_one = true 2024-04-26T04:18:03.793 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T04:18:03.793 INFO:teuthology.orchestra.run.smithi186.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-26T04:18:03.793 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T04:18:03.794 INFO:teuthology.orchestra.run.smithi186.stdout: osd default data pool replay window = 5 2024-04-26T04:18:03.794 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T04:18:03.794 INFO:teuthology.orchestra.run.smithi186.stdout: mon allow pool delete = true 2024-04-26T04:18:03.794 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T04:18:03.794 INFO:teuthology.orchestra.run.smithi186.stdout: mon cluster log file level = debug 2024-04-26T04:18:03.794 INFO:teuthology.orchestra.run.smithi186.stdout: debug asserts on shutdown = true 2024-04-26T04:18:03.794 INFO:teuthology.orchestra.run.smithi186.stdout: mon health detail to clog = false 2024-04-26T04:18:03.794 INFO:teuthology.orchestra.run.smithi186.stdout: mon host = "172.21.15.63,[v2:172.21.15.63:3301,v1:172.21.15.63:6790],172.21.15.186" 2024-04-26T04:18:03.794 INFO:teuthology.orchestra.run.smithi186.stdout: mon client directed command retry = 5 2024-04-26T04:18:03.794 INFO:teuthology.orchestra.run.smithi186.stdout: mon election default strategy = 1 2024-04-26T04:18:03.794 INFO:teuthology.orchestra.run.smithi186.stdout: ms inject delay max = 1 2024-04-26T04:18:03.794 INFO:teuthology.orchestra.run.smithi186.stdout: ms inject delay probability = 0.005 2024-04-26T04:18:03.794 INFO:teuthology.orchestra.run.smithi186.stdout: ms inject delay type = mon 2024-04-26T04:18:03.795 INFO:teuthology.orchestra.run.smithi186.stdout: ms inject internal delays = 0.002 2024-04-26T04:18:03.795 INFO:teuthology.orchestra.run.smithi186.stdout: ms inject socket failures = 2500 2024-04-26T04:18:03.795 INFO:teuthology.orchestra.run.smithi186.stdout: ms type = async 2024-04-26T04:18:03.795 INFO:teuthology.orchestra.run.smithi186.stdout: fsid = cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:03.795 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T04:18:03.795 INFO:teuthology.orchestra.run.smithi186.stdout:[osd] 2024-04-26T04:18:03.795 INFO:teuthology.orchestra.run.smithi186.stdout: osd journal size = 100 2024-04-26T04:18:03.795 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T04:18:03.795 INFO:teuthology.orchestra.run.smithi186.stdout: osd scrub load threshold = 5.0 2024-04-26T04:18:03.795 INFO:teuthology.orchestra.run.smithi186.stdout: osd scrub max interval = 600 2024-04-26T04:18:03.795 INFO:teuthology.orchestra.run.smithi186.stdout: osd mclock profile = high_recovery_ops 2024-04-26T04:18:03.795 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T04:18:03.795 INFO:teuthology.orchestra.run.smithi186.stdout: osd recover clone overlap = true 2024-04-26T04:18:03.795 INFO:teuthology.orchestra.run.smithi186.stdout: osd recovery max chunk = 1048576 2024-04-26T04:18:03.796 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T04:18:03.796 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug shutdown = true 2024-04-26T04:18:03.796 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug op order = true 2024-04-26T04:18:03.796 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug verify stray on activate = true 2024-04-26T04:18:03.796 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug trim objects = true 2024-04-26T04:18:03.796 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T04:18:03.796 INFO:teuthology.orchestra.run.smithi186.stdout: osd open classes on start = true 2024-04-26T04:18:03.796 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug pg log writeout = true 2024-04-26T04:18:03.796 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T04:18:03.796 INFO:teuthology.orchestra.run.smithi186.stdout: osd deep scrub update digest min age = 30 2024-04-26T04:18:03.796 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T04:18:03.796 INFO:teuthology.orchestra.run.smithi186.stdout: osd map max advance = 10 2024-04-26T04:18:03.796 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T04:18:03.796 INFO:teuthology.orchestra.run.smithi186.stdout: journal zero on create = true 2024-04-26T04:18:03.796 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T04:18:03.797 INFO:teuthology.orchestra.run.smithi186.stdout: filestore ondisk finisher threads = 3 2024-04-26T04:18:03.797 INFO:teuthology.orchestra.run.smithi186.stdout: filestore apply finisher threads = 3 2024-04-26T04:18:03.797 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T04:18:03.797 INFO:teuthology.orchestra.run.smithi186.stdout: bdev debug aio = true 2024-04-26T04:18:03.797 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug misdirected ops = true 2024-04-26T04:18:03.797 INFO:teuthology.orchestra.run.smithi186.stdout: bdev async discard = True 2024-04-26T04:18:03.797 INFO:teuthology.orchestra.run.smithi186.stdout: bdev enable discard = True 2024-04-26T04:18:03.797 INFO:teuthology.orchestra.run.smithi186.stdout: bluestore allocator = bitmap 2024-04-26T04:18:03.797 INFO:teuthology.orchestra.run.smithi186.stdout: bluestore block size = 96636764160 2024-04-26T04:18:03.797 INFO:teuthology.orchestra.run.smithi186.stdout: bluestore fsck on mount = True 2024-04-26T04:18:03.797 INFO:teuthology.orchestra.run.smithi186.stdout: bluestore zero block detection = True 2024-04-26T04:18:03.797 INFO:teuthology.orchestra.run.smithi186.stdout: debug bluefs = 20 2024-04-26T04:18:03.797 INFO:teuthology.orchestra.run.smithi186.stdout: debug bluestore = 20 2024-04-26T04:18:03.797 INFO:teuthology.orchestra.run.smithi186.stdout: debug ms = 1 2024-04-26T04:18:03.797 INFO:teuthology.orchestra.run.smithi186.stdout: debug osd = 20 2024-04-26T04:18:03.798 INFO:teuthology.orchestra.run.smithi186.stdout: debug rocksdb = 10 2024-04-26T04:18:03.798 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd backfillfull_ratio = 0.85 2024-04-26T04:18:03.798 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd full ratio = 0.9 2024-04-26T04:18:03.798 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd nearfull ratio = 0.8 2024-04-26T04:18:03.798 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug verify cached snaps = True 2024-04-26T04:18:03.798 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug verify missing on start = True 2024-04-26T04:18:03.798 INFO:teuthology.orchestra.run.smithi186.stdout: osd failsafe full ratio = 0.95 2024-04-26T04:18:03.798 INFO:teuthology.orchestra.run.smithi186.stdout: osd mclock override recovery settings = True 2024-04-26T04:18:03.798 INFO:teuthology.orchestra.run.smithi186.stdout: osd objectstore = bluestore 2024-04-26T04:18:03.798 INFO:teuthology.orchestra.run.smithi186.stdout: osd op queue = debug_random 2024-04-26T04:18:03.798 INFO:teuthology.orchestra.run.smithi186.stdout: osd op queue cut off = debug_random 2024-04-26T04:18:03.798 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T04:18:03.798 INFO:teuthology.orchestra.run.smithi186.stdout:[mgr] 2024-04-26T04:18:03.798 INFO:teuthology.orchestra.run.smithi186.stdout: debug ms = 1 2024-04-26T04:18:03.799 INFO:teuthology.orchestra.run.smithi186.stdout: debug mgr = 20 2024-04-26T04:18:03.799 INFO:teuthology.orchestra.run.smithi186.stdout: debug mon = 20 2024-04-26T04:18:03.799 INFO:teuthology.orchestra.run.smithi186.stdout: debug auth = 20 2024-04-26T04:18:03.799 INFO:teuthology.orchestra.run.smithi186.stdout: mon reweight min pgs per osd = 4 2024-04-26T04:18:03.799 INFO:teuthology.orchestra.run.smithi186.stdout: mon reweight min bytes per osd = 10 2024-04-26T04:18:03.799 INFO:teuthology.orchestra.run.smithi186.stdout: mgr/telemetry/nag = false 2024-04-26T04:18:03.799 INFO:teuthology.orchestra.run.smithi186.stdout: debug monc = 10 2024-04-26T04:18:03.799 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T04:18:03.799 INFO:teuthology.orchestra.run.smithi186.stdout:[mon] 2024-04-26T04:18:03.799 INFO:teuthology.orchestra.run.smithi186.stdout: debug ms = 1 2024-04-26T04:18:03.799 INFO:teuthology.orchestra.run.smithi186.stdout: debug mon = 20 2024-04-26T04:18:03.799 INFO:teuthology.orchestra.run.smithi186.stdout: debug paxos = 20 2024-04-26T04:18:03.799 INFO:teuthology.orchestra.run.smithi186.stdout: debug auth = 20 2024-04-26T04:18:03.800 INFO:teuthology.orchestra.run.smithi186.stdout: mon data avail warn = 5 2024-04-26T04:18:03.800 INFO:teuthology.orchestra.run.smithi186.stdout: mon mgr mkfs grace = 240 2024-04-26T04:18:03.800 INFO:teuthology.orchestra.run.smithi186.stdout: mon reweight min pgs per osd = 4 2024-04-26T04:18:03.800 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd reporter subtree level = osd 2024-04-26T04:18:03.800 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd prime pg temp = true 2024-04-26T04:18:03.800 INFO:teuthology.orchestra.run.smithi186.stdout: mon reweight min bytes per osd = 10 2024-04-26T04:18:03.800 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T04:18:03.800 INFO:teuthology.orchestra.run.smithi186.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-26T04:18:03.800 INFO:teuthology.orchestra.run.smithi186.stdout: auth mon ticket ttl = 660 # 11m 2024-04-26T04:18:03.800 INFO:teuthology.orchestra.run.smithi186.stdout: auth service ticket ttl = 240 # 4m 2024-04-26T04:18:03.800 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T04:18:03.800 INFO:teuthology.orchestra.run.smithi186.stdout: # don't complain about insecure global_id in the test suite 2024-04-26T04:18:03.800 INFO:teuthology.orchestra.run.smithi186.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-26T04:18:03.800 INFO:teuthology.orchestra.run.smithi186.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-26T04:18:03.801 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T04:18:03.801 INFO:teuthology.orchestra.run.smithi186.stdout: # 1m isn't quite enough 2024-04-26T04:18:03.801 INFO:teuthology.orchestra.run.smithi186.stdout: mon_down_mkfs_grace = 2m 2024-04-26T04:18:03.801 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T04:18:03.801 INFO:teuthology.orchestra.run.smithi186.stdout: mon_warn_on_filestore_osds = false 2024-04-26T04:18:03.801 INFO:teuthology.orchestra.run.smithi186.stdout: mon min osdmap epochs = 25 2024-04-26T04:18:03.801 INFO:teuthology.orchestra.run.smithi186.stdout: mon osdmap full prune interval = 2 2024-04-26T04:18:03.801 INFO:teuthology.orchestra.run.smithi186.stdout: mon osdmap full prune min = 15 2024-04-26T04:18:03.801 INFO:teuthology.orchestra.run.smithi186.stdout: mon osdmap full prune txsize = 2 2024-04-26T04:18:03.801 INFO:teuthology.orchestra.run.smithi186.stdout: mon scrub inject crc mismatch = 0.01 2024-04-26T04:18:03.801 INFO:teuthology.orchestra.run.smithi186.stdout: mon scrub inject missing keys = 0.05 2024-04-26T04:18:03.801 INFO:teuthology.orchestra.run.smithi186.stdout: mon scrub interval = 300 2024-04-26T04:18:03.801 INFO:teuthology.orchestra.run.smithi186.stdout: paxos min = 10 2024-04-26T04:18:03.801 INFO:teuthology.orchestra.run.smithi186.stdout: paxos service trim min = 5 2024-04-26T04:18:03.801 INFO:teuthology.orchestra.run.smithi186.stdout: paxos trim min = 10 2024-04-26T04:18:03.802 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-26T04:18:03.802 INFO:teuthology.orchestra.run.smithi186.stdout:[client] 2024-04-26T04:18:03.802 INFO:teuthology.orchestra.run.smithi186.stdout: rgw cache enabled = true 2024-04-26T04:18:03.802 INFO:teuthology.orchestra.run.smithi186.stdout: rgw enable ops log = true 2024-04-26T04:18:03.802 INFO:teuthology.orchestra.run.smithi186.stdout: rgw enable usage log = true 2024-04-26T04:18:03.802 INFO:teuthology.orchestra.run.smithi186.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-26T04:18:03.802 INFO:teuthology.orchestra.run.smithi186.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-26T04:18:03.802 INFO:teuthology.orchestra.run.smithi186.stdout: debug monc = 20 2024-04-26T04:18:03.802 INFO:teuthology.orchestra.run.smithi186.stdout: debug ms = 1 2024-04-26T04:18:03.802 INFO:teuthology.orchestra.run.smithi186.stdout:[mon.a] 2024-04-26T04:18:03.802 INFO:teuthology.orchestra.run.smithi186.stdout:[mon.c] 2024-04-26T04:18:03.802 INFO:teuthology.orchestra.run.smithi186.stdout:[mon.b] 2024-04-26T04:18:03.804 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-26T04:18:03.804 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-26T04:18:03.897 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-26T04:18:03.898 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-26T04:18:03.898 DEBUG:teuthology.orchestra.run.smithi063:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-26T04:18:03.903 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-26T04:18:03.903 DEBUG:teuthology.orchestra.run.smithi063:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-26T04:18:03.950 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi063.front.sepia.ceph.com 2024-04-26T04:18:03.950 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-26T04:18:03.951 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-26T04:18:03.951 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-26T04:18:04.015 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-26T04:18:04.015 DEBUG:teuthology.orchestra.run.smithi063:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-26T04:18:04.062 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi186.front.sepia.ceph.com 2024-04-26T04:18:04.062 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-26T04:18:04.062 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-26T04:18:04.063 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-26T04:18:04.085 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-26T04:18:04.086 DEBUG:teuthology.orchestra.run.smithi186:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-26T04:18:04.133 INFO:tasks.ceph:Setting up mon nodes... 2024-04-26T04:18:04.133 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-26T04:18:04.133 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-26T04:18:04.217 INFO:teuthology.orchestra.run.smithi186.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-26T04:18:04.229 INFO:tasks.ceph:Setting up mds nodes... 2024-04-26T04:18:04.229 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-26T04:18:04.229 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-26T04:18:04.309 INFO:teuthology.orchestra.run.smithi186.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-26T04:18:04.319 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-26T04:18:04.320 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi063.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi186.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-04-26T04:18:04.320 DEBUG:teuthology.orchestra.run.smithi063:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-26T04:18:04.334 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-26T04:18:04.334 INFO:tasks.ceph:role: osd.0 2024-04-26T04:18:04.334 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi063.front.sepia.ceph.com 2024-04-26T04:18:04.334 DEBUG:teuthology.orchestra.run.smithi063:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-26T04:18:04.398 INFO:teuthology.orchestra.run.smithi063.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-26T04:18:04.398 INFO:teuthology.orchestra.run.smithi063.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-26T04:18:04.398 INFO:teuthology.orchestra.run.smithi063.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-26T04:18:04.399 INFO:teuthology.orchestra.run.smithi063.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-26T04:18:04.399 INFO:teuthology.orchestra.run.smithi063.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-26T04:18:04.399 INFO:teuthology.orchestra.run.smithi063.stdout: = sunit=0 swidth=0 blks 2024-04-26T04:18:04.399 INFO:teuthology.orchestra.run.smithi063.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-26T04:18:04.399 INFO:teuthology.orchestra.run.smithi063.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-26T04:18:04.399 INFO:teuthology.orchestra.run.smithi063.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-26T04:18:04.399 INFO:teuthology.orchestra.run.smithi063.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-26T04:18:04.851 INFO:teuthology.orchestra.run.smithi063.stdout:Discarding blocks...Done. 2024-04-26T04:18:04.853 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi063.front.sepia.ceph.com -o noatime 2024-04-26T04:18:04.853 DEBUG:teuthology.orchestra.run.smithi063:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-26T04:18:04.978 DEBUG:teuthology.orchestra.run.smithi063:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-26T04:18:04.987 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: /sbin/restorecon: command not found 2024-04-26T04:18:04.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T04:18:04.988 DEBUG:teuthology.orchestra.run.smithi063:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-26T04:18:05.040 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-26T04:18:05.040 INFO:tasks.ceph:role: osd.1 2024-04-26T04:18:05.040 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi063.front.sepia.ceph.com 2024-04-26T04:18:05.040 DEBUG:teuthology.orchestra.run.smithi063:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-26T04:18:05.102 INFO:teuthology.orchestra.run.smithi063.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-26T04:18:05.102 INFO:teuthology.orchestra.run.smithi063.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-26T04:18:05.102 INFO:teuthology.orchestra.run.smithi063.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-26T04:18:05.102 INFO:teuthology.orchestra.run.smithi063.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-26T04:18:05.102 INFO:teuthology.orchestra.run.smithi063.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-26T04:18:05.102 INFO:teuthology.orchestra.run.smithi063.stdout: = sunit=0 swidth=0 blks 2024-04-26T04:18:05.102 INFO:teuthology.orchestra.run.smithi063.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-26T04:18:05.102 INFO:teuthology.orchestra.run.smithi063.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-26T04:18:05.102 INFO:teuthology.orchestra.run.smithi063.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-26T04:18:05.103 INFO:teuthology.orchestra.run.smithi063.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-26T04:18:05.533 INFO:teuthology.orchestra.run.smithi063.stdout:Discarding blocks...Done. 2024-04-26T04:18:05.536 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi063.front.sepia.ceph.com -o noatime 2024-04-26T04:18:05.536 DEBUG:teuthology.orchestra.run.smithi063:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-26T04:18:05.608 DEBUG:teuthology.orchestra.run.smithi063:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-26T04:18:05.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T04:18:05.659 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: /sbin/restorecon: command not found 2024-04-26T04:18:05.659 DEBUG:teuthology.orchestra.run.smithi063:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-26T04:18:05.715 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-26T04:18:05.716 INFO:tasks.ceph:role: osd.2 2024-04-26T04:18:05.716 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi063.front.sepia.ceph.com 2024-04-26T04:18:05.716 DEBUG:teuthology.orchestra.run.smithi063:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-26T04:18:05.784 INFO:teuthology.orchestra.run.smithi063.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-26T04:18:05.784 INFO:teuthology.orchestra.run.smithi063.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-26T04:18:05.784 INFO:teuthology.orchestra.run.smithi063.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-26T04:18:05.785 INFO:teuthology.orchestra.run.smithi063.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-26T04:18:05.785 INFO:teuthology.orchestra.run.smithi063.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-26T04:18:05.785 INFO:teuthology.orchestra.run.smithi063.stdout: = sunit=0 swidth=0 blks 2024-04-26T04:18:05.785 INFO:teuthology.orchestra.run.smithi063.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-26T04:18:05.785 INFO:teuthology.orchestra.run.smithi063.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-26T04:18:05.785 INFO:teuthology.orchestra.run.smithi063.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-26T04:18:05.785 INFO:teuthology.orchestra.run.smithi063.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-26T04:18:06.226 INFO:teuthology.orchestra.run.smithi063.stdout:Discarding blocks...Done. 2024-04-26T04:18:06.229 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi063.front.sepia.ceph.com -o noatime 2024-04-26T04:18:06.229 DEBUG:teuthology.orchestra.run.smithi063:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-04-26T04:18:06.324 DEBUG:teuthology.orchestra.run.smithi063:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-26T04:18:06.335 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: /sbin/restorecon: command not found 2024-04-26T04:18:06.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T04:18:06.336 DEBUG:teuthology.orchestra.run.smithi063:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-26T04:18:06.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:06.443+0000 7f4f48f507c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-04-26T04:18:06.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:06.443+0000 7f4f48f507c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-04-26T04:18:06.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:06.443+0000 7f4f48f507c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-26T04:18:06.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:06.443+0000 7f4f48f507c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-26T04:18:06.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:06.443+0000 7f4f48f507c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-26T04:18:06.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:06.451+0000 7f4f48f507c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-04-26T04:18:09.443 DEBUG:teuthology.orchestra.run.smithi063:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-26T04:18:09.458 DEBUG:teuthology.orchestra.run.smithi063:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-26T04:18:09.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:09.538+0000 7efd139307c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-04-26T04:18:09.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:09.538+0000 7efd139307c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-04-26T04:18:09.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:09.538+0000 7efd139307c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-26T04:18:09.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:09.538+0000 7efd139307c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-26T04:18:09.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:09.538+0000 7efd139307c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-26T04:18:09.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:09.538+0000 7efd139307c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-04-26T04:18:12.525 DEBUG:teuthology.orchestra.run.smithi063:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-26T04:18:12.583 DEBUG:teuthology.orchestra.run.smithi063:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-26T04:18:12.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:12.662+0000 7f7ee2feb7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-04-26T04:18:12.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:12.662+0000 7f7ee2feb7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-04-26T04:18:12.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:12.662+0000 7f7ee2feb7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-26T04:18:12.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:12.662+0000 7f7ee2feb7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-26T04:18:12.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:12.662+0000 7f7ee2feb7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-26T04:18:12.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:12.662+0000 7f7ee2feb7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-04-26T04:18:15.646 DEBUG:teuthology.orchestra.run.smithi063:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-26T04:18:15.705 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-04-26T04:18:15.721 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-04-26T04:18:15.722 INFO:tasks.ceph:role: osd.3 2024-04-26T04:18:15.722 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi186.front.sepia.ceph.com 2024-04-26T04:18:15.722 DEBUG:teuthology.orchestra.run.smithi186:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-26T04:18:15.783 INFO:teuthology.orchestra.run.smithi186.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-26T04:18:15.783 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-26T04:18:15.783 INFO:teuthology.orchestra.run.smithi186.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-26T04:18:15.783 INFO:teuthology.orchestra.run.smithi186.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-26T04:18:15.783 INFO:teuthology.orchestra.run.smithi186.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-26T04:18:15.784 INFO:teuthology.orchestra.run.smithi186.stdout: = sunit=0 swidth=0 blks 2024-04-26T04:18:15.784 INFO:teuthology.orchestra.run.smithi186.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-26T04:18:15.784 INFO:teuthology.orchestra.run.smithi186.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-26T04:18:15.784 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-26T04:18:15.784 INFO:teuthology.orchestra.run.smithi186.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-26T04:18:16.238 INFO:teuthology.orchestra.run.smithi186.stdout:Discarding blocks...Done. 2024-04-26T04:18:16.240 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi186.front.sepia.ceph.com -o noatime 2024-04-26T04:18:16.241 DEBUG:teuthology.orchestra.run.smithi186:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2024-04-26T04:18:16.335 DEBUG:teuthology.orchestra.run.smithi186:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-04-26T04:18:16.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T04:18:16.345 INFO:teuthology.orchestra.run.smithi186.stderr:sudo: /sbin/restorecon: command not found 2024-04-26T04:18:16.345 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-04-26T04:18:16.397 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-04-26T04:18:16.397 INFO:tasks.ceph:role: osd.4 2024-04-26T04:18:16.397 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi186.front.sepia.ceph.com 2024-04-26T04:18:16.397 DEBUG:teuthology.orchestra.run.smithi186:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-26T04:18:16.459 INFO:teuthology.orchestra.run.smithi186.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-26T04:18:16.460 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-26T04:18:16.460 INFO:teuthology.orchestra.run.smithi186.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-26T04:18:16.460 INFO:teuthology.orchestra.run.smithi186.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-26T04:18:16.460 INFO:teuthology.orchestra.run.smithi186.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-26T04:18:16.460 INFO:teuthology.orchestra.run.smithi186.stdout: = sunit=0 swidth=0 blks 2024-04-26T04:18:16.460 INFO:teuthology.orchestra.run.smithi186.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-26T04:18:16.460 INFO:teuthology.orchestra.run.smithi186.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-26T04:18:16.460 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-26T04:18:16.460 INFO:teuthology.orchestra.run.smithi186.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-26T04:18:16.941 INFO:teuthology.orchestra.run.smithi186.stdout:Discarding blocks...Done. 2024-04-26T04:18:16.943 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi186.front.sepia.ceph.com -o noatime 2024-04-26T04:18:16.943 DEBUG:teuthology.orchestra.run.smithi186:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2024-04-26T04:18:17.075 DEBUG:teuthology.orchestra.run.smithi186:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-04-26T04:18:17.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T04:18:17.086 INFO:teuthology.orchestra.run.smithi186.stderr:sudo: /sbin/restorecon: command not found 2024-04-26T04:18:17.087 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-04-26T04:18:17.139 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-04-26T04:18:17.139 INFO:tasks.ceph:role: osd.5 2024-04-26T04:18:17.139 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi186.front.sepia.ceph.com 2024-04-26T04:18:17.140 DEBUG:teuthology.orchestra.run.smithi186:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-26T04:18:17.204 INFO:teuthology.orchestra.run.smithi186.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-26T04:18:17.204 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-26T04:18:17.204 INFO:teuthology.orchestra.run.smithi186.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-26T04:18:17.204 INFO:teuthology.orchestra.run.smithi186.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-26T04:18:17.204 INFO:teuthology.orchestra.run.smithi186.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-26T04:18:17.204 INFO:teuthology.orchestra.run.smithi186.stdout: = sunit=0 swidth=0 blks 2024-04-26T04:18:17.204 INFO:teuthology.orchestra.run.smithi186.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-26T04:18:17.205 INFO:teuthology.orchestra.run.smithi186.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-26T04:18:17.205 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-26T04:18:17.205 INFO:teuthology.orchestra.run.smithi186.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-26T04:18:17.700 INFO:teuthology.orchestra.run.smithi186.stdout:Discarding blocks...Done. 2024-04-26T04:18:17.702 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi186.front.sepia.ceph.com -o noatime 2024-04-26T04:18:17.702 DEBUG:teuthology.orchestra.run.smithi186:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5 2024-04-26T04:18:17.863 DEBUG:teuthology.orchestra.run.smithi186:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-04-26T04:18:17.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T04:18:17.875 INFO:teuthology.orchestra.run.smithi186.stderr:sudo: /sbin/restorecon: command not found 2024-04-26T04:18:17.876 DEBUG:teuthology.orchestra.run.smithi186:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-26T04:18:17.987 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:17.984+0000 7f2e60ec77c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-04-26T04:18:17.988 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:17.988+0000 7f2e60ec77c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-04-26T04:18:17.988 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:17.988+0000 7f2e60ec77c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-26T04:18:17.988 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:17.988+0000 7f2e60ec77c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-26T04:18:17.988 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:17.988+0000 7f2e60ec77c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-26T04:18:17.989 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:17.988+0000 7f2e60ec77c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-04-26T04:18:20.964 DEBUG:teuthology.orchestra.run.smithi186:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-26T04:18:20.977 DEBUG:teuthology.orchestra.run.smithi186:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-26T04:18:21.057 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:21.056+0000 7f010d2837c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-04-26T04:18:21.058 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:21.056+0000 7f010d2837c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-04-26T04:18:21.058 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:21.056+0000 7f010d2837c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-26T04:18:21.058 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:21.056+0000 7f010d2837c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-26T04:18:21.058 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:21.056+0000 7f010d2837c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-26T04:18:21.058 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:21.056+0000 7f010d2837c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-04-26T04:18:24.063 DEBUG:teuthology.orchestra.run.smithi186:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-26T04:18:24.121 DEBUG:teuthology.orchestra.run.smithi186:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-26T04:18:24.202 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:24.200+0000 7fb8dd6207c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-04-26T04:18:24.203 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:24.200+0000 7fb8dd6207c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-04-26T04:18:24.203 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:24.200+0000 7fb8dd6207c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-26T04:18:24.203 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:24.200+0000 7fb8dd6207c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-26T04:18:24.203 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:24.200+0000 7fb8dd6207c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-26T04:18:24.204 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:24.204+0000 7fb8dd6207c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-04-26T04:18:27.200 DEBUG:teuthology.orchestra.run.smithi186:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-26T04:18:27.257 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-26T04:18:27.257 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-26T04:18:27.258 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-26T04:18:27.274 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-26T04:18:27.274 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-26T04:18:27.331 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-26T04:18:27.331 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-26T04:18:27.390 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-26T04:18:27.390 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-26T04:18:27.404 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-26T04:18:27.404 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-04-26T04:18:27.459 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-26T04:18:27.459 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-04-26T04:18:27.515 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-26T04:18:27.515 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-04-26T04:18:27.573 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-26T04:18:27.573 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-26T04:18:27.623 INFO:tasks.ceph:Adding keys to all mons... 2024-04-26T04:18:27.623 DEBUG:teuthology.orchestra.run.smithi063:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-26T04:18:27.625 DEBUG:teuthology.orchestra.run.smithi186:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-26T04:18:27.667 INFO:teuthology.orchestra.run.smithi063.stdout:[osd.0] 2024-04-26T04:18:27.667 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQD+Kitm+gWoGhAAKjJ4IevlmbaB7MiuaZ7rtw== 2024-04-26T04:18:27.667 INFO:teuthology.orchestra.run.smithi063.stdout:[osd.1] 2024-04-26T04:18:27.667 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQABKytmlERgIBAAayqoGfdJmnTi7CfdYA8qpQ== 2024-04-26T04:18:27.667 INFO:teuthology.orchestra.run.smithi063.stdout:[osd.2] 2024-04-26T04:18:27.667 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQAEKytmEa6zJxAAhcma80Szw1dfbH2/4RyRBQ== 2024-04-26T04:18:27.667 INFO:teuthology.orchestra.run.smithi063.stdout:[mgr.x] 2024-04-26T04:18:27.667 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQD8KitmmngADRAAg5bXnQr2uF2fvtSp7x80Bw== 2024-04-26T04:18:27.667 INFO:teuthology.orchestra.run.smithi063.stdout:[osd.3] 2024-04-26T04:18:27.668 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQAJKytm/YLoOhAA669LAAh2+SG6mqfbN0EeNA== 2024-04-26T04:18:27.668 INFO:teuthology.orchestra.run.smithi063.stdout:[osd.4] 2024-04-26T04:18:27.668 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQANKytmmYV8AxAAN51LnWM1XmZZ4UFbNm9GUA== 2024-04-26T04:18:27.668 INFO:teuthology.orchestra.run.smithi063.stdout:[osd.5] 2024-04-26T04:18:27.668 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQAQKytmNkshDBAA/5ZHOP7tUPS0okmcjYZ+3g== 2024-04-26T04:18:27.668 INFO:teuthology.orchestra.run.smithi063.stdout:[client.0] 2024-04-26T04:18:27.668 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQD8KitmbVN5EhAA1LnYC/tGo2qa5inlqJ8hNw== 2024-04-26T04:18:27.675 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.0] 2024-04-26T04:18:27.676 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQD+Kitm+gWoGhAAKjJ4IevlmbaB7MiuaZ7rtw== 2024-04-26T04:18:27.676 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.1] 2024-04-26T04:18:27.676 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQABKytmlERgIBAAayqoGfdJmnTi7CfdYA8qpQ== 2024-04-26T04:18:27.676 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.2] 2024-04-26T04:18:27.676 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQAEKytmEa6zJxAAhcma80Szw1dfbH2/4RyRBQ== 2024-04-26T04:18:27.676 INFO:teuthology.orchestra.run.smithi186.stdout:[mgr.x] 2024-04-26T04:18:27.676 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQD8KitmmngADRAAg5bXnQr2uF2fvtSp7x80Bw== 2024-04-26T04:18:27.676 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.3] 2024-04-26T04:18:27.676 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQAJKytm/YLoOhAA669LAAh2+SG6mqfbN0EeNA== 2024-04-26T04:18:27.676 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.4] 2024-04-26T04:18:27.676 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQANKytmmYV8AxAAN51LnWM1XmZZ4UFbNm9GUA== 2024-04-26T04:18:27.676 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.5] 2024-04-26T04:18:27.677 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQAQKytmNkshDBAA/5ZHOP7tUPS0okmcjYZ+3g== 2024-04-26T04:18:27.677 INFO:teuthology.orchestra.run.smithi186.stdout:[client.0] 2024-04-26T04:18:27.677 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQD8KitmbVN5EhAA1LnYC/tGo2qa5inlqJ8hNw== 2024-04-26T04:18:27.678 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T04:18:27.715 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T04:18:27.763 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T04:18:27.765 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T04:18:27.852 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T04:18:27.854 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T04:18:27.941 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-26T04:18:27.943 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-26T04:18:28.025 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T04:18:28.035 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T04:18:28.107 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T04:18:28.119 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T04:18:28.191 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T04:18:28.203 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-26T04:18:28.280 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-26T04:18:28.287 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-26T04:18:28.334 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-26T04:18:28.335 DEBUG:teuthology.orchestra.run.smithi063:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-26T04:18:28.347 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-26T04:18:28.439 DEBUG:teuthology.orchestra.run.smithi063:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-26T04:18:28.451 DEBUG:teuthology.orchestra.run.smithi063:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-04-26T04:18:28.503 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-26T04:18:28.591 DEBUG:teuthology.orchestra.run.smithi063:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-04-26T04:18:28.602 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-04-26T04:18:28.616 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-26T04:18:28.708 DEBUG:teuthology.orchestra.run.smithi186:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-04-26T04:18:28.720 DEBUG:teuthology.orchestra.run.smithi063:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-26T04:18:28.722 DEBUG:teuthology.orchestra.run.smithi186:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-26T04:18:28.768 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-26T04:18:28.768 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-26T04:18:28.768 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-26T04:18:28.771 INFO:tasks.ceph.mon.a:Started 2024-04-26T04:18:28.771 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-26T04:18:28.771 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-04-26T04:18:28.774 INFO:tasks.ceph.mon.c:Started 2024-04-26T04:18:28.774 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-26T04:18:28.774 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-04-26T04:18:28.813 INFO:tasks.ceph.mon.b:Started 2024-04-26T04:18:28.813 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-26T04:18:28.813 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-26T04:18:28.814 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-26T04:18:28.816 INFO:tasks.ceph.mgr.x:Started 2024-04-26T04:18:28.816 INFO:tasks.ceph:Setting crush tunables to default 2024-04-26T04:18:28.816 DEBUG:teuthology.orchestra.run.smithi063:> sudo ceph --cluster ceph osd crush tunables default 2024-04-26T04:18:28.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:28.926+0000 7fcbc6dd7640 10 monclient: get_monmap_and_config 2024-04-26T04:18:28.931 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:28.926+0000 7fcbc6dd7640 10 monclient: build_initial_monmap 2024-04-26T04:18:28.931 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:28.926+0000 7fcbc6dd7640 10 monclient: monmap: 2024-04-26T04:18:28.931 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:28.931 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:28.931 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:28.930818+0000 2024-04-26T04:18:28.931 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:28.930818+0000 2024-04-26T04:18:28.931 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:28.931 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:28.931 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:18:28.931 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:18:28.931 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:18:28.931 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:18:28.931 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:18:28.932 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:28.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:28.926+0000 7fcbc6dd7640 1 Processor -- start 2024-04-26T04:18:28.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:28.926+0000 7fcbc6dd7640 1 -- start start 2024-04-26T04:18:28.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:28.926+0000 7fcbc6dd7640 10 monclient: init 2024-04-26T04:18:28.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:28.926+0000 7fcbc6dd7640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:28.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:28.926+0000 7fcbc6dd7640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-04-26T04:18:28.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:28.926+0000 7fcbc6dd7640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcbc00a2a90 0x7fcbc00a2e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:28.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:28.926+0000 7fcbc6dd7640 10 monclient(hunting): picked mon.noname-b con 0x7fcbc00a2a90 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:28.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:28.926+0000 7fcbc6dd7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcbc00a1ec0 addr v1:172.21.15.186:6789/0 2024-04-26T04:18:28.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:28.926+0000 7fcbc6dd7640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fcbc00a22a0 0x7fcbc00a45c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:28.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:28.926+0000 7fcbc6dd7640 10 monclient(hunting): picked mon.noname-c con 0x7fcbc00a22a0 addr v2:172.21.15.186:3300/0 2024-04-26T04:18:28.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:28.926+0000 7fcbc6dd7640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbc00a0480 con 0x7fcbc00a1ec0 2024-04-26T04:18:28.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:28.926+0000 7fcbc6dd7640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:28.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:28.926+0000 7fcbc6dd7640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fcbc00a0f40 con 0x7fcbc00a22a0 2024-04-26T04:18:28.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:28.926+0000 7fcbc6dd7640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:28.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:28.926+0000 7fcbc6dd7640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7fcbc00a0c60 con 0x7fcbc00a2a90 2024-04-26T04:18:28.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:28.926+0000 7fcbc6dd7640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:28.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:28.926+0000 7fcbc6dd7640 10 monclient(hunting): authenticate will time out at 940.638145s 2024-04-26T04:18:28.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:28.926+0000 7fcbc4b73640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcbc00a2a90 0x7fcbc00a2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:28.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:28.926+0000 7fcbc4b73640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcbc00a2a90 0x7fcbc00a2e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:34726/0 (socket says 172.21.15.63:34726) 2024-04-26T04:18:28.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:28.926+0000 7fcbc4b73640 1 -- 172.21.15.63:0/251606978 learned_addr learned my addr 172.21.15.63:0/251606978 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:18:28.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:28.926+0000 7fcbc4b73640 10 monclient(hunting): get_auth_request con 0x7fcbc00a2a90 auth_method 0 2024-04-26T04:18:28.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:28.926+0000 7fcbc4b73640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:28.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:28.926+0000 7fcbc4b73640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:28.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:28.926+0000 7fcbc4b73640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:28.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:28.926+0000 7fcbc5374640 1 --2- 172.21.15.63:0/251606978 >> v2:172.21.15.186:3300/0 conn(0x7fcbc00a22a0 0x7fcbc00a45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:28.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:28.926+0000 7fcbc4b73640 1 -- 172.21.15.63:0/251606978 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcbc00a2a90 msgr2=0x7fcbc00a2e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T04:18:28.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:28.926+0000 7fcbc4b73640 1 -- 172.21.15.63:0/251606978 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcbc00a2a90 msgr2=0x7fcbc00a2e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:18:28.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:28.926+0000 7fcbc4b73640 1 --2- 172.21.15.63:0/251606978 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcbc00a2a90 0x7fcbc00a2e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:18:28.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:28.926+0000 7fcbc5374640 10 monclient(hunting): get_auth_request con 0x7fcbc00a22a0 auth_method 0 2024-04-26T04:18:28.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:28.926+0000 7fcbc5374640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:28.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:28.926+0000 7fcbc5374640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:28.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:28.926+0000 7fcbc5374640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:28.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:28.926+0000 7fcbc4b73640 1 --2- 172.21.15.63:0/251606978 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcbc00a2a90 0x7fcbc00a2e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:18:28.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:28.926+0000 7fcbc5374640 1 -- 172.21.15.63:0/251606978 >> v2:172.21.15.186:3300/0 conn(0x7fcbc00a22a0 msgr2=0x7fcbc00a45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T04:18:28.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:28.926+0000 7fcbc5374640 1 -- 172.21.15.63:0/251606978 >> v2:172.21.15.186:3300/0 conn(0x7fcbc00a22a0 msgr2=0x7fcbc00a45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:18:28.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:28.926+0000 7fcbc5374640 1 --2- 172.21.15.63:0/251606978 >> v2:172.21.15.186:3300/0 conn(0x7fcbc00a22a0 0x7fcbc00a45c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:18:28.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:28.926+0000 7fcbc5374640 1 --2- 172.21.15.63:0/251606978 >> v2:172.21.15.186:3300/0 conn(0x7fcbc00a22a0 0x7fcbc00a45c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:18:28.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:28.930+0000 7fcbbffff640 1 -- 172.21.15.63:0/251606978 >> v1:172.21.15.186:6789/0 conn(0x7fcbc00a1ec0 legacy=0x7fcbc00a3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:29.134 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:29.130+0000 7fcbc4b73640 1 --2- 172.21.15.63:0/251606978 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcbc00a2a90 0x7fcbc00a2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:29.135 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:29.130+0000 7fcbc4b73640 10 monclient(hunting): get_auth_request con 0x7fcbc00a2a90 auth_method 0 2024-04-26T04:18:29.135 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:29.130+0000 7fcbc4b73640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:29.135 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:29.130+0000 7fcbc4b73640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:29.135 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:29.130+0000 7fcbc4b73640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:18:29.135 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:29.130+0000 7fcbc4b73640 1 -- 172.21.15.63:0/251606978 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcbc00a2a90 msgr2=0x7fcbc00a2e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T04:18:29.135 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:29.130+0000 7fcbc5374640 1 --2- 172.21.15.63:0/251606978 >> v2:172.21.15.186:3300/0 conn(0x7fcbc00a22a0 0x7fcbc00a45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:29.135 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:29.130+0000 7fcbc4b73640 1 -- 172.21.15.63:0/251606978 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcbc00a2a90 msgr2=0x7fcbc00a2e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:18:29.135 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:29.130+0000 7fcbc4b73640 1 --2- 172.21.15.63:0/251606978 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcbc00a2a90 0x7fcbc00a2e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:18:29.135 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:29.130+0000 7fcbc5374640 10 monclient(hunting): get_auth_request con 0x7fcbc00a22a0 auth_method 0 2024-04-26T04:18:29.135 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:29.130+0000 7fcbc5374640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:29.135 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:29.130+0000 7fcbc5374640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:29.135 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:29.130+0000 7fcbc5374640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:18:29.135 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:29.130+0000 7fcbc4b73640 1 --2- 172.21.15.63:0/251606978 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcbc00a2a90 0x7fcbc00a2e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T04:18:29.136 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:29.130+0000 7fcbc5374640 1 -- 172.21.15.63:0/251606978 >> v2:172.21.15.186:3300/0 conn(0x7fcbc00a22a0 msgr2=0x7fcbc00a45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T04:18:29.136 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:29.130+0000 7fcbc5374640 1 -- 172.21.15.63:0/251606978 >> v2:172.21.15.186:3300/0 conn(0x7fcbc00a22a0 msgr2=0x7fcbc00a45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:18:29.136 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:29.130+0000 7fcbc5374640 1 --2- 172.21.15.63:0/251606978 >> v2:172.21.15.186:3300/0 conn(0x7fcbc00a22a0 0x7fcbc00a45c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:18:29.136 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:29.130+0000 7fcbc5374640 1 --2- 172.21.15.63:0/251606978 >> v2:172.21.15.186:3300/0 conn(0x7fcbc00a22a0 0x7fcbc00a45c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T04:18:29.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:29.534+0000 7fcbc4b73640 1 --2- 172.21.15.63:0/251606978 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcbc00a2a90 0x7fcbc00a2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:29.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:29.534+0000 7fcbc4b73640 10 monclient(hunting): get_auth_request con 0x7fcbc00a2a90 auth_method 0 2024-04-26T04:18:29.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:29.534+0000 7fcbc4b73640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:29.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:29.534+0000 7fcbc4b73640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:29.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:29.534+0000 7fcbc4b73640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:18:29.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:29.534+0000 7fcbc5374640 1 --2- 172.21.15.63:0/251606978 >> v2:172.21.15.186:3300/0 conn(0x7fcbc00a22a0 0x7fcbc00a45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:29.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:29.534+0000 7fcbc5374640 10 monclient(hunting): get_auth_request con 0x7fcbc00a22a0 auth_method 0 2024-04-26T04:18:29.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:29.534+0000 7fcbc5374640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:29.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:29.534+0000 7fcbc5374640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:29.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:29.534+0000 7fcbc5374640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:18:29.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:29.534+0000 7fcbc4b73640 1 -- 172.21.15.63:0/251606978 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcbc00a2a90 msgr2=0x7fcbc00a2e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T04:18:29.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:29.534+0000 7fcbc4b73640 1 -- 172.21.15.63:0/251606978 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcbc00a2a90 msgr2=0x7fcbc00a2e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:18:29.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:29.534+0000 7fcbc4b73640 1 --2- 172.21.15.63:0/251606978 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcbc00a2a90 0x7fcbc00a2e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:18:29.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:29.534+0000 7fcbc5374640 1 -- 172.21.15.63:0/251606978 >> v2:172.21.15.186:3300/0 conn(0x7fcbc00a22a0 msgr2=0x7fcbc00a45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T04:18:29.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:29.534+0000 7fcbc5374640 1 -- 172.21.15.63:0/251606978 >> v2:172.21.15.186:3300/0 conn(0x7fcbc00a22a0 msgr2=0x7fcbc00a45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:18:29.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:29.534+0000 7fcbc5374640 1 --2- 172.21.15.63:0/251606978 >> v2:172.21.15.186:3300/0 conn(0x7fcbc00a22a0 0x7fcbc00a45c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:18:29.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:29.534+0000 7fcbc5374640 1 --2- 172.21.15.63:0/251606978 >> v2:172.21.15.186:3300/0 conn(0x7fcbc00a22a0 0x7fcbc00a45c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T04:18:29.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:29.534+0000 7fcbc4b73640 1 --2- 172.21.15.63:0/251606978 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcbc00a2a90 0x7fcbc00a2e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T04:18:30.341 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:30.338+0000 7fcbc4b73640 1 --2- 172.21.15.63:0/251606978 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcbc00a2a90 0x7fcbc00a2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:30.341 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:30.338+0000 7fcbc4b73640 10 monclient(hunting): get_auth_request con 0x7fcbc00a2a90 auth_method 0 2024-04-26T04:18:30.341 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:30.338+0000 7fcbc4b73640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:30.341 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:30.338+0000 7fcbc4b73640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:30.341 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:30.338+0000 7fcbc4b73640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:18:30.341 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:30.338+0000 7fcbc5374640 1 --2- 172.21.15.63:0/251606978 >> v2:172.21.15.186:3300/0 conn(0x7fcbc00a22a0 0x7fcbc00a45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:30.341 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:30.338+0000 7fcbc5374640 10 monclient(hunting): get_auth_request con 0x7fcbc00a22a0 auth_method 0 2024-04-26T04:18:30.341 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:30.338+0000 7fcbc5374640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:30.342 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:30.338+0000 7fcbc5374640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:30.342 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:30.338+0000 7fcbc5374640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:18:30.342 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:30.338+0000 7fcbc4b73640 1 -- 172.21.15.63:0/251606978 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcbc00a2a90 msgr2=0x7fcbc00a2e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T04:18:30.342 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:30.338+0000 7fcbc4b73640 1 -- 172.21.15.63:0/251606978 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcbc00a2a90 msgr2=0x7fcbc00a2e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:18:30.342 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:30.338+0000 7fcbc4b73640 1 --2- 172.21.15.63:0/251606978 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcbc00a2a90 0x7fcbc00a2e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:18:30.342 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:30.338+0000 7fcbc5374640 1 -- 172.21.15.63:0/251606978 >> v2:172.21.15.186:3300/0 conn(0x7fcbc00a22a0 msgr2=0x7fcbc00a45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T04:18:30.342 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:30.338+0000 7fcbc5374640 1 -- 172.21.15.63:0/251606978 >> v2:172.21.15.186:3300/0 conn(0x7fcbc00a22a0 msgr2=0x7fcbc00a45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:18:30.342 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:30.338+0000 7fcbc5374640 1 --2- 172.21.15.63:0/251606978 >> v2:172.21.15.186:3300/0 conn(0x7fcbc00a22a0 0x7fcbc00a45c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:18:30.342 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:30.338+0000 7fcbc5374640 1 --2- 172.21.15.63:0/251606978 >> v2:172.21.15.186:3300/0 conn(0x7fcbc00a22a0 0x7fcbc00a45c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T04:18:30.342 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:30.338+0000 7fcbc4b73640 1 --2- 172.21.15.63:0/251606978 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcbc00a2a90 0x7fcbc00a2e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T04:18:31.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:31.930+0000 7fcbbe7fc640 10 monclient(hunting): tick 2024-04-26T04:18:31.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:31.930+0000 7fcbbe7fc640 10 monclient(hunting): _check_auth_tickets 2024-04-26T04:18:31.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:31.930+0000 7fcbbe7fc640 1 monclient(hunting): continuing hunt 2024-04-26T04:18:31.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:31.930+0000 7fcbbe7fc640 10 monclient(hunting): _reopen_session rank -1 2024-04-26T04:18:31.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:31.930+0000 7fcbbe7fc640 1 -- 172.21.15.63:0/251606978 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcbc00a2a90 msgr2=0x7fcbc00a2e70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T04:18:31.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:31.930+0000 7fcbbe7fc640 1 --2- 172.21.15.63:0/251606978 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcbc00a2a90 0x7fcbc00a2e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:31.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:31.930+0000 7fcbbe7fc640 1 -- 172.21.15.63:0/251606978 >> v2:172.21.15.186:3300/0 conn(0x7fcbc00a22a0 msgr2=0x7fcbc00a45c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T04:18:31.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:31.930+0000 7fcbbe7fc640 1 --2- 172.21.15.63:0/251606978 >> v2:172.21.15.186:3300/0 conn(0x7fcbc00a22a0 0x7fcbc00a45c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:31.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:31.930+0000 7fcbbe7fc640 1 -- 172.21.15.63:0/251606978 >> v1:172.21.15.186:6789/0 conn(0x7fcbc00a1ec0 legacy=0x7fcbc00a3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:31.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:31.930+0000 7fcbbe7fc640 10 monclient: _add_conns ranks=[3,0] 2024-04-26T04:18:31.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:31.930+0000 7fcbbe7fc640 1 --2- 172.21.15.63:0/251606978 >> v2:172.21.15.63:3300/0 conn(0x7fcbac003750 0x7fcbac005bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:31.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:31.930+0000 7fcbbe7fc640 10 monclient(hunting): picked mon.noname-a con 0x7fcbac003750 addr v2:172.21.15.63:3300/0 2024-04-26T04:18:31.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:31.930+0000 7fcbbe7fc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcbac0062a0 addr v1:172.21.15.63:6789/0 2024-04-26T04:18:31.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:31.930+0000 7fcbbe7fc640 1 -- 172.21.15.63:0/251606978 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbc00a1220 con 0x7fcbac0062a0 2024-04-26T04:18:31.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:31.930+0000 7fcbbe7fc640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:31.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:31.930+0000 7fcbbe7fc640 1 -- 172.21.15.63:0/251606978 --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7fcbc00a0f40 con 0x7fcbac003750 2024-04-26T04:18:31.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:31.930+0000 7fcbbe7fc640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:31.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:31.930+0000 7fcbc4b73640 1 --2- 172.21.15.63:0/251606978 >> v2:172.21.15.63:3300/0 conn(0x7fcbac003750 0x7fcbac005bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:31.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:31.930+0000 7fcbc4b73640 10 monclient(hunting): get_auth_request con 0x7fcbac003750 auth_method 0 2024-04-26T04:18:31.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:31.930+0000 7fcbc4b73640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:31.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:31.930+0000 7fcbc4b73640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:31.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:31.930+0000 7fcbc4b73640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:31.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:31.930+0000 7fcbc4b73640 1 -- 172.21.15.63:0/251606978 >> v2:172.21.15.63:3300/0 conn(0x7fcbac003750 msgr2=0x7fcbac005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T04:18:31.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:31.930+0000 7fcbc4b73640 1 -- 172.21.15.63:0/251606978 >> v2:172.21.15.63:3300/0 conn(0x7fcbac003750 msgr2=0x7fcbac005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:18:31.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:31.930+0000 7fcbc4b73640 1 --2- 172.21.15.63:0/251606978 >> v2:172.21.15.63:3300/0 conn(0x7fcbac003750 0x7fcbac005bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:18:31.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:31.930+0000 7fcbc4b73640 1 --2- 172.21.15.63:0/251606978 >> v2:172.21.15.63:3300/0 conn(0x7fcbac003750 0x7fcbac005bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:18:31.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:31.930+0000 7fcbbffff640 1 -- 172.21.15.63:0/251606978 >> v1:172.21.15.63:6789/0 conn(0x7fcbac0062a0 legacy=0x7fcbac008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:32.138 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:32.134+0000 7fcbc4b73640 1 --2- 172.21.15.63:0/251606978 >> v2:172.21.15.63:3300/0 conn(0x7fcbac003750 0x7fcbac005bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:32.138 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:32.134+0000 7fcbc4b73640 10 monclient(hunting): get_auth_request con 0x7fcbac003750 auth_method 0 2024-04-26T04:18:32.138 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:32.134+0000 7fcbc4b73640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:32.138 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:32.134+0000 7fcbc4b73640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:32.139 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:32.134+0000 7fcbc4b73640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:18:32.139 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:32.134+0000 7fcbc4b73640 1 -- 172.21.15.63:0/251606978 >> v2:172.21.15.63:3300/0 conn(0x7fcbac003750 msgr2=0x7fcbac005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T04:18:32.139 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:32.134+0000 7fcbc4b73640 1 -- 172.21.15.63:0/251606978 >> v2:172.21.15.63:3300/0 conn(0x7fcbac003750 msgr2=0x7fcbac005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:18:32.139 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:32.134+0000 7fcbc4b73640 1 --2- 172.21.15.63:0/251606978 >> v2:172.21.15.63:3300/0 conn(0x7fcbac003750 0x7fcbac005bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:18:32.139 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:32.134+0000 7fcbc4b73640 1 --2- 172.21.15.63:0/251606978 >> v2:172.21.15.63:3300/0 conn(0x7fcbac003750 0x7fcbac005bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T04:18:32.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:32.538+0000 7fcbc4b73640 1 --2- 172.21.15.63:0/251606978 >> v2:172.21.15.63:3300/0 conn(0x7fcbac003750 0x7fcbac005bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:32.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:32.538+0000 7fcbc4b73640 10 monclient(hunting): get_auth_request con 0x7fcbac003750 auth_method 0 2024-04-26T04:18:32.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:32.538+0000 7fcbc4b73640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:32.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:32.538+0000 7fcbc4b73640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:32.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:32.538+0000 7fcbc4b73640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:18:32.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:32.538+0000 7fcbc4b73640 1 -- 172.21.15.63:0/251606978 >> v2:172.21.15.63:3300/0 conn(0x7fcbac003750 msgr2=0x7fcbac005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T04:18:32.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:32.538+0000 7fcbc4b73640 1 -- 172.21.15.63:0/251606978 >> v2:172.21.15.63:3300/0 conn(0x7fcbac003750 msgr2=0x7fcbac005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:18:32.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:32.538+0000 7fcbc4b73640 1 --2- 172.21.15.63:0/251606978 >> v2:172.21.15.63:3300/0 conn(0x7fcbac003750 0x7fcbac005bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:18:32.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:32.538+0000 7fcbc4b73640 1 --2- 172.21.15.63:0/251606978 >> v2:172.21.15.63:3300/0 conn(0x7fcbac003750 0x7fcbac005bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T04:18:33.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:33.342+0000 7fcbc4b73640 1 --2- 172.21.15.63:0/251606978 >> v2:172.21.15.63:3300/0 conn(0x7fcbac003750 0x7fcbac005bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:33.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:33.342+0000 7fcbc4b73640 10 monclient(hunting): get_auth_request con 0x7fcbac003750 auth_method 0 2024-04-26T04:18:33.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:33.342+0000 7fcbc4b73640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:33.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:33.342+0000 7fcbc4b73640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:33.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:33.342+0000 7fcbc4b73640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:18:33.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:33.342+0000 7fcbc4b73640 1 -- 172.21.15.63:0/251606978 >> v2:172.21.15.63:3300/0 conn(0x7fcbac003750 msgr2=0x7fcbac005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T04:18:33.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:33.342+0000 7fcbc4b73640 1 -- 172.21.15.63:0/251606978 >> v2:172.21.15.63:3300/0 conn(0x7fcbac003750 msgr2=0x7fcbac005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:18:33.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:33.342+0000 7fcbc4b73640 1 --2- 172.21.15.63:0/251606978 >> v2:172.21.15.63:3300/0 conn(0x7fcbac003750 0x7fcbac005bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:18:33.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:33.342+0000 7fcbc4b73640 1 --2- 172.21.15.63:0/251606978 >> v2:172.21.15.63:3300/0 conn(0x7fcbac003750 0x7fcbac005bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T04:18:34.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbbf7fe640 1 -- 172.21.15.63:0/251606978 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1543384039 0 0) 0x7fcbc00a1220 con 0x7fcbac0062a0 2024-04-26T04:18:34.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbbf7fe640 10 monclient(hunting): _negotiate 2024-04-26T04:18:34.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbbf7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:34.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbbf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:34.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbbf7fe640 10 monclient(hunting): my global_id is 4098 2024-04-26T04:18:34.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbbf7fe640 1 -- 172.21.15.63:0/251606978 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcba8003530 con 0x7fcbac0062a0 2024-04-26T04:18:34.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbbf7fe640 1 -- 172.21.15.63:0/251606978 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2637363133 0 0) 0x7fcba8003530 con 0x7fcbac0062a0 2024-04-26T04:18:34.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbbf7fe640 1 -- 172.21.15.63:0/251606978 >> v2:172.21.15.63:3300/0 conn(0x7fcbac003750 msgr2=0x7fcbac005bb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T04:18:34.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbbf7fe640 1 --2- 172.21.15.63:0/251606978 >> v2:172.21.15.63:3300/0 conn(0x7fcbac003750 0x7fcbac005bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:34.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbbf7fe640 10 monclient: _finish_hunting 0 2024-04-26T04:18:34.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbbf7fe640 1 monclient: found mon.noname-a-legacy 2024-04-26T04:18:34.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbbf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:34.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbbf7fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.63:6789/0 2024-04-26T04:18:34.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbbf7fe640 1 -- 172.21.15.63:0/251606978 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbac00b090 con 0x7fcbac0062a0 2024-04-26T04:18:34.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbbf7fe640 10 monclient: _finish_auth 0 2024-04-26T04:18:34.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbbf7fe640 10 monclient: _check_auth_tickets 2024-04-26T04:18:34.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbbf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:34.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbbf7fe640 1 -- 172.21.15.63:0/251606978 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2612289768 0 0) 0x7fcbb4001980 con 0x7fcbac0062a0 2024-04-26T04:18:34.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbbf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:34.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbbf7fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-26T04:18:34.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbbf7fe640 10 monclient: dump: 2024-04-26T04:18:34.283 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:34.283 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:34.283 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:34.283 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:34.283 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:34.283 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:34.283 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:34.283 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:34.284 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:34.284 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:34.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbbf7fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] but i'm connected to v1:172.21.15.63:6789/0, reconnecting 2024-04-26T04:18:34.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbbf7fe640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:34.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbbf7fe640 1 -- 172.21.15.63:0/251606978 >> v1:172.21.15.63:6789/0 conn(0x7fcbac0062a0 legacy=0x7fcbac008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:34.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbc5374640 1 -- 172.21.15.63:0/251606978 reap_dead start 2024-04-26T04:18:34.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbbf7fe640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T04:18:34.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbbf7fe640 1 --2- 172.21.15.63:0/251606978 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcba8003af0 0x7fcba8003ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:34.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbbf7fe640 10 monclient(hunting): picked mon.a con 0x7fcba8003af0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:18:34.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbbf7fe640 1 --2- 172.21.15.63:0/251606978 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcba8004810 0x7fcba8004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:34.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbbf7fe640 10 monclient(hunting): picked mon.b con 0x7fcba8004810 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:18:34.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbbf7fe640 1 --2- 172.21.15.63:0/251606978 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcba8005420 0x7fcba800ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:34.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbbf7fe640 10 monclient(hunting): picked mon.c con 0x7fcba8005420 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:34.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbbf7fe640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:34.285 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbbf7fe640 1 -- 172.21.15.63:0/251606978 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fcbc00a0f40 con 0x7fcba8003af0 2024-04-26T04:18:34.285 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbbf7fe640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:34.285 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbbf7fe640 1 -- 172.21.15.63:0/251606978 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcba800c360 con 0x7fcba8004810 2024-04-26T04:18:34.285 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbbf7fe640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:34.285 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbbf7fe640 1 -- 172.21.15.63:0/251606978 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7fcba800c560 con 0x7fcba8005420 2024-04-26T04:18:34.285 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbbf7fe640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:34.285 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbbf7fe640 10 monclient(hunting): _finish_auth 0 2024-04-26T04:18:34.285 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbc4b73640 1 --2- 172.21.15.63:0/251606978 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcba8003af0 0x7fcba8003ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:34.285 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbbffff640 1 --2- 172.21.15.63:0/251606978 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcba8004810 0x7fcba8004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:34.285 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbc5374640 1 --2- 172.21.15.63:0/251606978 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcba8005420 0x7fcba800ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:34.285 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbc4b73640 10 monclient(hunting): get_auth_request con 0x7fcba8003af0 auth_method 0 2024-04-26T04:18:34.285 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbc4b73640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:34.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbc4b73640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:34.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbc4b73640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:18:34.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbc5374640 10 monclient(hunting): get_auth_request con 0x7fcba8005420 auth_method 0 2024-04-26T04:18:34.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbc5374640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:34.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbc5374640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:34.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbc5374640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:18:34.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbbffff640 10 monclient(hunting): get_auth_request con 0x7fcba8004810 auth_method 0 2024-04-26T04:18:34.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbbffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:34.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbbffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:34.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbbffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:18:34.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbc4b73640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:34.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbc4b73640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:34.287 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbc4b73640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T04:18:34.287 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbc5374640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:34.287 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbc5374640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:34.287 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbc5374640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T04:18:34.287 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbbffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:34.287 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbbffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:34.287 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbbffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T04:18:34.287 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbc4b73640 10 monclient(hunting): handle_auth_done global_id 4098 payload 293 2024-04-26T04:18:34.287 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbc4b73640 1 -- 172.21.15.63:0/251606978 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcba8005420 msgr2=0x7fcba800ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:34.287 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbc4b73640 1 --2- 172.21.15.63:0/251606978 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcba8005420 0x7fcba800ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:34.287 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbc4b73640 1 -- 172.21.15.63:0/251606978 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcba8004810 msgr2=0x7fcba8004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:34.287 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbc4b73640 1 --2- 172.21.15.63:0/251606978 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcba8004810 0x7fcba8004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:34.287 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbc4b73640 10 monclient: _finish_hunting 0 2024-04-26T04:18:34.287 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbc4b73640 1 monclient: found mon.a 2024-04-26T04:18:34.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbc4b73640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:34.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbc4b73640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:34.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbc4b73640 1 -- 172.21.15.63:0/251606978 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcba8003530 con 0x7fcba8003af0 2024-04-26T04:18:34.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbc4b73640 10 monclient: _finish_auth 0 2024-04-26T04:18:34.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbc4b73640 10 monclient: _check_auth_tickets 2024-04-26T04:18:34.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbc4b73640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:34.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbc6dd7640 5 monclient: authenticate success, global_id 4098 2024-04-26T04:18:34.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbc6dd7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T04:18:34.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.274+0000 7fcbc5374640 1 --2- 172.21.15.63:0/251606978 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcba8005420 0x7fcba800ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T04:18:34.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.282+0000 7fcbc4b73640 1 -- 172.21.15.63:0/251606978 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcba8003af0 msgr2=0x7fcba8003ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:34.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.282+0000 7fcbc4b73640 1 --2- 172.21.15.63:0/251606978 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcba8003af0 0x7fcba8003ef0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fcbb0019af0 tx=0x7fcbb001de30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=290ec2ee6c0511db server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:34.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.282+0000 7fcbbf7fe640 1 -- 172.21.15.63:0/251606978 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb000f040 con 0x7fcba8003af0 2024-04-26T04:18:34.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.282+0000 7fcbbf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:34.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.282+0000 7fcbbf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:18:34.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.282+0000 7fcbbf7fe640 10 monclient: dump: 2024-04-26T04:18:34.289 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:34.290 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:34.290 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:34.290 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:34.290 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:34.290 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:34.290 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:34.290 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:34.290 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:34.290 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:34.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.282+0000 7fcbbf7fe640 1 -- 172.21.15.63:0/251606978 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbb00a6d30 con 0x7fcba8003af0 2024-04-26T04:18:34.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.282+0000 7fcbc6dd7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T04:18:34.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.282+0000 7fcbbf7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:34.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.282+0000 7fcbc6dd7640 10 monclient: get_monmap_and_config success 2024-04-26T04:18:34.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.282+0000 7fcbc6dd7640 4 set_mon_vals no callback set 2024-04-26T04:18:34.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.282+0000 7fcbbf7fe640 1 -- 172.21.15.63:0/251606978 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb0004270 con 0x7fcba8003af0 2024-04-26T04:18:34.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.282+0000 7fcbbf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:34.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.282+0000 7fcbbf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:18:34.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.282+0000 7fcbbf7fe640 10 monclient: dump: 2024-04-26T04:18:34.291 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:34.291 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:34.291 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:34.291 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:34.291 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:34.291 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:34.291 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:34.291 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:34.291 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:34.291 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:34.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.286+0000 7fcbc6dd7640 10 monclient: shutdown 2024-04-26T04:18:34.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.286+0000 7fcbc6dd7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:34.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.286+0000 7fcbc6dd7640 1 -- 172.21.15.63:0/251606978 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcba8003af0 msgr2=0x7fcba8003ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:34.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.286+0000 7fcbc6dd7640 1 --2- 172.21.15.63:0/251606978 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcba8003af0 0x7fcba8003ef0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fcbb0019af0 tx=0x7fcbb001de30 comp rx=0 tx=0).stop 2024-04-26T04:18:34.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.286+0000 7fcbc6dd7640 1 -- 172.21.15.63:0/251606978 shutdown_connections 2024-04-26T04:18:34.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.286+0000 7fcbc6dd7640 1 --2- 172.21.15.63:0/251606978 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcba8005420 0x7fcba800ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:34.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.286+0000 7fcbc6dd7640 1 --2- 172.21.15.63:0/251606978 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcba8004810 0x7fcba8004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:34.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.286+0000 7fcbc6dd7640 1 --2- 172.21.15.63:0/251606978 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcba8003af0 0x7fcba8003ef0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:34.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.286+0000 7fcbc6dd7640 1 -- 172.21.15.63:0/251606978 >> 172.21.15.63:0/251606978 conn(0x7fcbc00a5fb0 msgr2=0x7fcbc00a63c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:34.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.286+0000 7fcbc6dd7640 1 -- 172.21.15.63:0/251606978 shutdown_connections 2024-04-26T04:18:34.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.286+0000 7fcbc6dd7640 1 -- 172.21.15.63:0/251606978 wait complete. 2024-04-26T04:18:34.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.286+0000 7fcbc6dd7640 10 monclient: build_initial_monmap 2024-04-26T04:18:34.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.286+0000 7fcbc6dd7640 10 monclient: monmap: 2024-04-26T04:18:34.294 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:34.294 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:18:34.294 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:18:34.294 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:18:34.294 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:34.294 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:34.294 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:18:34.294 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:18:34.295 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:18:34.295 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:34.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.286+0000 7fcbc6dd7640 1 Processor -- start 2024-04-26T04:18:34.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.286+0000 7fcbc6dd7640 1 -- start start 2024-04-26T04:18:34.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.286+0000 7fcbc6dd7640 10 monclient: init 2024-04-26T04:18:34.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.290+0000 7fcbc6dd7640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:34.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.290+0000 7fcbc6dd7640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T04:18:34.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.290+0000 7fcbc6dd7640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcba8003af0 0x7fcbc01cf2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:34.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.290+0000 7fcbc6dd7640 10 monclient(hunting): picked mon.noname-c con 0x7fcba8003af0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:34.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.290+0000 7fcbc6dd7640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcba8004810 0x7fcbc01cf820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:34.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.290+0000 7fcbc6dd7640 10 monclient(hunting): picked mon.noname-a con 0x7fcba8004810 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:18:34.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.290+0000 7fcbc6dd7640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcba8005420 0x7fcbc0121f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:34.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.290+0000 7fcbc6dd7640 10 monclient(hunting): picked mon.noname-b con 0x7fcba8005420 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:18:34.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.290+0000 7fcbc6dd7640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:34.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.290+0000 7fcbc6dd7640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fcbc00a4a50 con 0x7fcba8004810 2024-04-26T04:18:34.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.290+0000 7fcbc6dd7640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:34.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.290+0000 7fcbc6dd7640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcbc00a48d0 con 0x7fcba8005420 2024-04-26T04:18:34.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.290+0000 7fcbc6dd7640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:34.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.290+0000 7fcbc6dd7640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7fcbc00a4bd0 con 0x7fcba8003af0 2024-04-26T04:18:34.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.290+0000 7fcbc6dd7640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:34.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.290+0000 7fcbc6dd7640 10 monclient(hunting): authenticate will time out at 946.001086s 2024-04-26T04:18:34.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.290+0000 7fcbc4b73640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcba8003af0 0x7fcbc01cf2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:34.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.290+0000 7fcbbffff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcba8004810 0x7fcbc01cf820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:34.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.290+0000 7fcbc4b73640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcba8003af0 0x7fcbc01cf2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:41568/0 (socket says 172.21.15.63:41568) 2024-04-26T04:18:34.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.290+0000 7fcbc4b73640 1 -- 172.21.15.63:0/2361661055 learned_addr learned my addr 172.21.15.63:0/2361661055 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:18:34.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.290+0000 7fcbc4b73640 10 monclient(hunting): get_auth_request con 0x7fcba8003af0 auth_method 0 2024-04-26T04:18:34.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.290+0000 7fcbc4b73640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:34.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.290+0000 7fcbc4b73640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:34.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.290+0000 7fcbc4b73640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:34.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.290+0000 7fcbbffff640 10 monclient(hunting): get_auth_request con 0x7fcba8004810 auth_method 0 2024-04-26T04:18:34.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.290+0000 7fcbbffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:34.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.290+0000 7fcbbffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:34.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.290+0000 7fcbbffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:34.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.290+0000 7fcbc4b73640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:34.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.290+0000 7fcbc4b73640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:34.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.290+0000 7fcbc4b73640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:34.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.290+0000 7fcbbffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:34.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.290+0000 7fcbbffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:34.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.290+0000 7fcbbffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:34.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.290+0000 7fcbc5374640 1 --2- 172.21.15.63:0/2361661055 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcba8005420 0x7fcbc0121f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:34.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.290+0000 7fcbc5374640 10 monclient(hunting): get_auth_request con 0x7fcba8005420 auth_method 0 2024-04-26T04:18:34.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.290+0000 7fcbc5374640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:34.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.290+0000 7fcbc5374640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:34.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.290+0000 7fcbc5374640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:34.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.290+0000 7fcbc4b73640 10 monclient(hunting): handle_auth_done global_id 4100 payload 824 2024-04-26T04:18:34.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.290+0000 7fcbc4b73640 1 -- 172.21.15.63:0/2361661055 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcba8005420 msgr2=0x7fcbc0121f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:34.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.290+0000 7fcbc4b73640 1 --2- 172.21.15.63:0/2361661055 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcba8005420 0x7fcbc0121f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:34.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.290+0000 7fcbc4b73640 1 -- 172.21.15.63:0/2361661055 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcba8004810 msgr2=0x7fcbc01cf820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:34.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.290+0000 7fcbc4b73640 1 --2- 172.21.15.63:0/2361661055 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcba8004810 0x7fcbc01cf820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:34.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.290+0000 7fcbc4b73640 10 monclient: _finish_hunting 0 2024-04-26T04:18:34.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.290+0000 7fcbc4b73640 1 monclient: found mon.noname-c 2024-04-26T04:18:34.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.290+0000 7fcbc4b73640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:34.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.290+0000 7fcbc4b73640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.63:3301/0 2024-04-26T04:18:34.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.290+0000 7fcbc4b73640 1 -- 172.21.15.63:0/2361661055 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbc00a3720 con 0x7fcba8003af0 2024-04-26T04:18:34.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.290+0000 7fcbc5374640 1 --2- 172.21.15.63:0/2361661055 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcba8005420 0x7fcbc0121f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T04:18:34.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.294+0000 7fcbc4b73640 1 -- 172.21.15.63:0/2361661055 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcba8003af0 msgr2=0x7fcbc01cf2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:34.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.294+0000 7fcbc4b73640 1 --2- 172.21.15.63:0/2361661055 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcba8003af0 0x7fcbc01cf2e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fcbb001dc60 tx=0x7fcba8005840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:34.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.294+0000 7fcbbd7fa640 1 -- 172.21.15.63:0/2361661055 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb00bc020 con 0x7fcba8003af0 2024-04-26T04:18:34.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.294+0000 7fcbbd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:34.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.294+0000 7fcbbd7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:18:34.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.294+0000 7fcbbd7fa640 10 monclient: dump: 2024-04-26T04:18:34.302 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:34.302 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:34.302 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:34.302 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:34.303 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:34.303 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:34.303 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:34.303 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:34.303 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:34.303 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:34.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.294+0000 7fcbbd7fa640 10 monclient: _finish_auth 0 2024-04-26T04:18:34.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.294+0000 7fcbbd7fa640 10 monclient: _check_auth_tickets 2024-04-26T04:18:34.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.294+0000 7fcbbd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:34.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.294+0000 7fcbbd7fa640 1 -- 172.21.15.63:0/2361661055 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbb0003830 con 0x7fcba8003af0 2024-04-26T04:18:34.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.294+0000 7fcbbd7fa640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:34.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.294+0000 7fcbbd7fa640 1 -- 172.21.15.63:0/2361661055 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb000f040 con 0x7fcba8003af0 2024-04-26T04:18:34.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.294+0000 7fcbc6dd7640 5 monclient: authenticate success, global_id 4100 2024-04-26T04:18:34.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.294+0000 7fcbc6dd7640 10 monclient: _renew_subs 2024-04-26T04:18:34.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.294+0000 7fcbbe7fc640 4 set_mon_vals no callback set 2024-04-26T04:18:34.304 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.294+0000 7fcbc6dd7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:34.304 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.294+0000 7fcbc6dd7640 1 -- 172.21.15.63:0/2361661055 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbc0123b80 con 0x7fcba8003af0 2024-04-26T04:18:34.304 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.294+0000 7fcbbd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:34.304 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.294+0000 7fcbbd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:18:34.304 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.294+0000 7fcbbd7fa640 10 monclient: dump: 2024-04-26T04:18:34.304 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:34.304 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:34.304 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:34.304 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:34.304 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:34.304 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:34.304 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:34.304 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:34.304 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:34.304 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:34.305 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.294+0000 7fcbc6dd7640 10 monclient: _renew_subs 2024-04-26T04:18:34.305 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.294+0000 7fcbc6dd7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:34.305 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.294+0000 7fcbc6dd7640 1 -- 172.21.15.63:0/2361661055 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbc0124460 con 0x7fcba8003af0 2024-04-26T04:18:34.305 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.294+0000 7fcbbd7fa640 1 -- 172.21.15.63:0/2361661055 <== mon.2 v2:172.21.15.63:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fcbb00bf020 con 0x7fcba8003af0 2024-04-26T04:18:34.305 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.294+0000 7fcbbd7fa640 1 -- 172.21.15.63:0/2361661055 <== mon.2 v2:172.21.15.63:3301/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fcbb00c1400 con 0x7fcba8003af0 2024-04-26T04:18:34.305 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.294+0000 7fcbc6dd7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:34.305 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.294+0000 7fcbc6dd7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:34.305 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.294+0000 7fcbc6dd7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:34.305 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.294+0000 7fcbc6dd7640 1 -- 172.21.15.63:0/2361661055 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbc0062e10 con 0x7fcba8003af0 2024-04-26T04:18:34.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.306+0000 7fcbbd7fa640 1 -- 172.21.15.63:0/2361661055 <== mon.2 v2:172.21.15.63:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fcbb00c6030 con 0x7fcba8003af0 2024-04-26T04:18:34.311 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.306+0000 7fcbbd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:34.311 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.306+0000 7fcbbd7fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:18:34.376 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.370+0000 7fcbc6dd7640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-04-26T04:18:34.376 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.370+0000 7fcbc6dd7640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-04-26T04:18:34.376 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.370+0000 7fcbc6dd7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:34.376 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.370+0000 7fcbc6dd7640 1 -- 172.21.15.63:0/2361661055 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7fcbc00683c0 con 0x7fcba8003af0 2024-04-26T04:18:34.393 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.386+0000 7fcbbd7fa640 1 -- 172.21.15.63:0/2361661055 <== mon.2 v2:172.21.15.63:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7fcbb000b980 con 0x7fcba8003af0 2024-04-26T04:18:34.393 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.386+0000 7fcbbd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-04-26T04:18:34.393 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.386+0000 7fcbbd7fa640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-04-26T04:18:34.393 INFO:teuthology.orchestra.run.smithi063.stderr:adjusted tunables profile to default 2024-04-26T04:18:34.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.390+0000 7fcbc6dd7640 10 monclient: shutdown 2024-04-26T04:18:34.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.390+0000 7fcbc6dd7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:34.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.390+0000 7fcbc6dd7640 1 -- 172.21.15.63:0/2361661055 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcba8003af0 msgr2=0x7fcbc01cf2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:34.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.390+0000 7fcbc6dd7640 1 --2- 172.21.15.63:0/2361661055 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcba8003af0 0x7fcbc01cf2e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fcbb001dc60 tx=0x7fcba8005840 comp rx=0 tx=0).stop 2024-04-26T04:18:34.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.390+0000 7fcbc6dd7640 1 -- 172.21.15.63:0/2361661055 shutdown_connections 2024-04-26T04:18:34.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.390+0000 7fcbc6dd7640 1 --2- 172.21.15.63:0/2361661055 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcba8005420 0x7fcbc0121f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:34.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.390+0000 7fcbc6dd7640 1 --2- 172.21.15.63:0/2361661055 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcba8004810 0x7fcbc01cf820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:34.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.390+0000 7fcbc6dd7640 1 --2- 172.21.15.63:0/2361661055 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcba8003af0 0x7fcbc01cf2e0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:34.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.390+0000 7fcbc6dd7640 1 -- 172.21.15.63:0/2361661055 >> 172.21.15.63:0/2361661055 conn(0x7fcbc00a5fb0 msgr2=0x7fcbc0071a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:34.395 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.390+0000 7fcbc6dd7640 1 -- 172.21.15.63:0/2361661055 shutdown_connections 2024-04-26T04:18:34.395 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:34.390+0000 7fcbc6dd7640 1 -- 172.21.15.63:0/2361661055 wait complete. 2024-04-26T04:18:34.412 INFO:tasks.ceph:check_enable_crimson: False 2024-04-26T04:18:34.412 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-26T04:18:34.413 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-26T04:18:34.413 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-26T04:18:34.423 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-26T04:18:34.423 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-26T04:18:34.476 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-26T04:18:34.476 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-26T04:18:34.527 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-26T04:18:34.527 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-04-26T04:18:34.543 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-26T04:18:34.543 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-04-26T04:18:34.596 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-26T04:18:34.597 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-04-26T04:18:34.650 DEBUG:teuthology.orchestra.run.smithi186:> sudo ceph --cluster ceph osd new 2d37141b-8282-4c62-864a-84a75ebf8f6f 0 2024-04-26T04:18:34.817 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.816+0000 7f9264db0640 10 monclient: get_monmap_and_config 2024-04-26T04:18:34.817 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.816+0000 7f9264db0640 10 monclient: build_initial_monmap 2024-04-26T04:18:34.817 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.816+0000 7f9264db0640 10 monclient: monmap: 2024-04-26T04:18:34.817 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 0 2024-04-26T04:18:34.818 INFO:teuthology.orchestra.run.smithi186.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:34.818 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-04-26T04:18:34.818479+0000 2024-04-26T04:18:34.818 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-04-26T04:18:34.818479+0000 2024-04-26T04:18:34.818 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:34.818 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-04-26T04:18:34.818 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:18:34.818 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:18:34.818 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:18:34.818 INFO:teuthology.orchestra.run.smithi186.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:18:34.818 INFO:teuthology.orchestra.run.smithi186.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:18:34.818 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-04-26T04:18:34.821 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.816+0000 7f9264db0640 1 Processor -- start 2024-04-26T04:18:34.821 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.816+0000 7f9264db0640 1 -- start start 2024-04-26T04:18:34.821 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.816+0000 7f9264db0640 10 monclient: init 2024-04-26T04:18:34.821 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.816+0000 7f9264db0640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:34.821 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.816+0000 7f9264db0640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-04-26T04:18:34.821 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.816+0000 7f9264db0640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f92600a2a90 0x7f92600a2e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:34.821 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.816+0000 7f9264db0640 10 monclient(hunting): picked mon.noname-b con 0x7f92600a2a90 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:34.821 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.816+0000 7f9264db0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f92600a1ec0 addr v1:172.21.15.186:6789/0 2024-04-26T04:18:34.821 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.816+0000 7f9264db0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f92600a22a0 addr v1:172.21.15.63:6789/0 2024-04-26T04:18:34.821 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.816+0000 7f9264db0640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92600a0480 con 0x7f92600a22a0 2024-04-26T04:18:34.822 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.816+0000 7f9264db0640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92600a4cd0 con 0x7f92600a1ec0 2024-04-26T04:18:34.822 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.816+0000 7f9264db0640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:34.822 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.816+0000 7f9264db0640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f92600a0f40 con 0x7f92600a2a90 2024-04-26T04:18:34.822 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.816+0000 7f9264db0640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:34.822 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.816+0000 7f9264db0640 10 monclient(hunting): authenticate will time out at 944.595189s 2024-04-26T04:18:34.822 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.816+0000 7f925dd9b640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f92600a1ec0 0x7f92600a3eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:46920/0 (socket says 172.21.15.186:46920) 2024-04-26T04:18:34.822 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.816+0000 7f925dd9b640 1 -- 172.21.15.186:0/1433772013 learned_addr learned my addr 172.21.15.186:0/1433772013 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-04-26T04:18:34.822 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.816+0000 7f925e59c640 1 --2- 172.21.15.186:0/1433772013 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f92600a2a90 0x7f92600a2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:34.822 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.816+0000 7f925e59c640 10 monclient(hunting): get_auth_request con 0x7f92600a2a90 auth_method 0 2024-04-26T04:18:34.822 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.816+0000 7f925e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:34.822 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.816+0000 7f925e59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:34.823 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.816+0000 7f925e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:34.823 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.816+0000 7f925e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:34.823 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.816+0000 7f925e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:34.823 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.816+0000 7f925e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:34.823 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.820+0000 7f925e59c640 10 monclient(hunting): handle_auth_done global_id 4103 payload 274 2024-04-26T04:18:34.823 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.820+0000 7f925e59c640 1 -- 172.21.15.186:0/1433772013 >> v1:172.21.15.186:6789/0 conn(0x7f92600a1ec0 legacy=0x7f92600a3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:34.823 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.820+0000 7f925e59c640 1 -- 172.21.15.186:0/1433772013 >> v1:172.21.15.63:6789/0 conn(0x7f92600a22a0 legacy=0x7f92600a45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:34.823 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.820+0000 7f925e59c640 10 monclient: _finish_hunting 0 2024-04-26T04:18:34.823 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.820+0000 7f925e59c640 1 monclient: found mon.noname-b 2024-04-26T04:18:34.823 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.820+0000 7f925e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:34.824 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.820+0000 7f925e59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.63:3301/0 2024-04-26T04:18:34.824 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.820+0000 7f925e59c640 1 -- 172.21.15.186:0/1433772013 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92600a0c60 con 0x7f92600a2a90 2024-04-26T04:18:34.824 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.824+0000 7f925e59c640 1 -- 172.21.15.186:0/1433772013 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f92600a2a90 msgr2=0x7f92600a2e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:34.824 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.824+0000 7f925e59c640 1 --2- 172.21.15.186:0/1433772013 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f92600a2a90 0x7f92600a2e70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f92580097f0 tx=0x7f925801c2f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ced7a36d22f9692a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:34.825 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.824+0000 7f925d59a640 1 -- 172.21.15.186:0/1433772013 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92580b0070 con 0x7f92600a2a90 2024-04-26T04:18:34.825 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.824+0000 7f925d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:34.825 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.824+0000 7f925d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T04:18:34.825 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.824+0000 7f925d59a640 10 monclient: dump: 2024-04-26T04:18:34.825 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-04-26T04:18:34.825 INFO:teuthology.orchestra.run.smithi186.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:34.825 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:34.825 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:34.825 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:34.826 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-04-26T04:18:34.826 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:34.826 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:34.826 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:34.826 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-04-26T04:18:34.826 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.824+0000 7f925d59a640 10 monclient: _finish_auth 0 2024-04-26T04:18:34.826 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.824+0000 7f925d59a640 10 monclient: _check_auth_tickets 2024-04-26T04:18:34.826 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.824+0000 7f925d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:34.826 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.824+0000 7f9264db0640 5 monclient: authenticate success, global_id 4103 2024-04-26T04:18:34.826 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.824+0000 7f9264db0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T04:18:34.826 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.824+0000 7f925d59a640 1 -- 172.21.15.186:0/1433772013 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92580ab600 con 0x7f92600a2a90 2024-04-26T04:18:34.826 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.824+0000 7f925d59a640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:34.827 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.824+0000 7f9264db0640 10 monclient: get_monmap_and_config success 2024-04-26T04:18:34.827 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.824+0000 7f9264db0640 4 set_mon_vals no callback set 2024-04-26T04:18:34.827 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.824+0000 7f925d59a640 1 -- 172.21.15.186:0/1433772013 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92580ab9d0 con 0x7f92600a2a90 2024-04-26T04:18:34.827 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.824+0000 7f925d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:34.827 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.824+0000 7f925d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:18:34.827 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.824+0000 7f925d59a640 10 monclient: dump: 2024-04-26T04:18:34.827 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-04-26T04:18:34.827 INFO:teuthology.orchestra.run.smithi186.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:34.827 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:34.827 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:34.827 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:34.827 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-04-26T04:18:34.827 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:34.827 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:34.828 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:34.828 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-04-26T04:18:34.839 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.824+0000 7f9264db0640 10 monclient: shutdown 2024-04-26T04:18:34.839 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.824+0000 7f9264db0640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:34.840 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.824+0000 7f9264db0640 1 -- 172.21.15.186:0/1433772013 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f92600a2a90 msgr2=0x7f92600a2e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:34.840 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.824+0000 7f9264db0640 1 --2- 172.21.15.186:0/1433772013 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f92600a2a90 0x7f92600a2e70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f92580097f0 tx=0x7f925801c2f0 comp rx=0 tx=0).stop 2024-04-26T04:18:34.840 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.824+0000 7f9264db0640 1 -- 172.21.15.186:0/1433772013 shutdown_connections 2024-04-26T04:18:34.840 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.824+0000 7f9264db0640 1 --2- 172.21.15.186:0/1433772013 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f92600a2a90 0x7f92600a2e70 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:34.840 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.824+0000 7f9264db0640 1 -- 172.21.15.186:0/1433772013 >> 172.21.15.186:0/1433772013 conn(0x7f92600a5fb0 msgr2=0x7f92600a63c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:34.840 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.824+0000 7f9264db0640 1 -- 172.21.15.186:0/1433772013 shutdown_connections 2024-04-26T04:18:34.840 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.824+0000 7f9264db0640 1 -- 172.21.15.186:0/1433772013 wait complete. 2024-04-26T04:18:34.840 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.824+0000 7f9264db0640 10 monclient: build_initial_monmap 2024-04-26T04:18:34.840 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.824+0000 7f9264db0640 10 monclient: monmap: 2024-04-26T04:18:34.840 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 0 2024-04-26T04:18:34.840 INFO:teuthology.orchestra.run.smithi186.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:18:34.840 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 0.000000 2024-04-26T04:18:34.840 INFO:teuthology.orchestra.run.smithi186.stderr:created 0.000000 2024-04-26T04:18:34.841 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:34.841 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-04-26T04:18:34.841 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:18:34.841 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:18:34.841 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:18:34.841 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-04-26T04:18:34.841 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.824+0000 7f9264db0640 1 Processor -- start 2024-04-26T04:18:34.841 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.824+0000 7f9264db0640 1 -- start start 2024-04-26T04:18:34.841 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.824+0000 7f9264db0640 10 monclient: init 2024-04-26T04:18:34.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.824+0000 7f9264db0640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:34.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.824+0000 7f9264db0640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T04:18:34.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.824+0000 7f9264db0640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f92600a1ec0 0x7f92601cf0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:34.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.824+0000 7f9264db0640 10 monclient(hunting): picked mon.noname-a con 0x7f92600a1ec0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:18:34.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.824+0000 7f9264db0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f92600a22a0 0x7f926013b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:34.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.824+0000 7f9264db0640 10 monclient(hunting): picked mon.noname-b con 0x7f92600a22a0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:18:34.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.824+0000 7f9264db0640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9260120a60 0x7f9260120ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:34.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.824+0000 7f9264db0640 10 monclient(hunting): picked mon.noname-c con 0x7f9260120a60 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:34.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.824+0000 7f9264db0640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:34.843 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.824+0000 7f9264db0640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f92601428c0 con 0x7f92600a1ec0 2024-04-26T04:18:34.843 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.824+0000 7f9264db0640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:34.843 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.824+0000 7f9264db0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9260142740 con 0x7f92600a22a0 2024-04-26T04:18:34.843 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.824+0000 7f9264db0640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:34.843 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.824+0000 7f9264db0640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f9260142a40 con 0x7f9260120a60 2024-04-26T04:18:34.843 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.824+0000 7f9264db0640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:34.843 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.824+0000 7f9264db0640 10 monclient(hunting): authenticate will time out at 944.603719s 2024-04-26T04:18:34.843 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.824+0000 7f925dd9b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f92600a22a0 0x7f926013b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:34.843 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.824+0000 7f925dd9b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f92600a22a0 0x7f926013b010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:40916/0 (socket says 172.21.15.186:40916) 2024-04-26T04:18:34.844 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.824+0000 7f925dd9b640 1 -- 172.21.15.186:0/2799147091 learned_addr learned my addr 172.21.15.186:0/2799147091 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-04-26T04:18:34.844 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.824+0000 7f925dd9b640 10 monclient(hunting): get_auth_request con 0x7f92600a22a0 auth_method 0 2024-04-26T04:18:34.844 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.824+0000 7f925dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:34.844 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.824+0000 7f925dd9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:34.844 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.824+0000 7f925dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:34.844 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f925dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:34.844 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f925dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:34.844 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f925dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:34.844 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f925e59c640 1 --2- 172.21.15.186:0/2799147091 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f92600a1ec0 0x7f92601cf0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:34.845 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f925ed9d640 1 --2- 172.21.15.186:0/2799147091 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9260120a60 0x7f9260120ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:34.845 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f925ed9d640 1 -- 172.21.15.186:0/2799147091 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9260120a60 msgr2=0x7f9260120ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-26T04:18:34.845 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f925ed9d640 1 -- 172.21.15.186:0/2799147091 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9260120a60 msgr2=0x7f9260120ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T04:18:34.845 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f925ed9d640 1 --2- 172.21.15.186:0/2799147091 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9260120a60 0x7f9260120ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:18:34.845 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f925e59c640 10 monclient(hunting): get_auth_request con 0x7f92600a1ec0 auth_method 0 2024-04-26T04:18:34.845 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f925e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:34.845 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f925e59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:34.845 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f925e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:34.845 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f925dd9b640 10 monclient(hunting): handle_auth_done global_id 4102 payload 824 2024-04-26T04:18:34.845 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f925ed9d640 1 --2- 172.21.15.186:0/2799147091 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9260120a60 0x7f9260120ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:18:34.845 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f925dd9b640 1 -- 172.21.15.186:0/2799147091 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9260120a60 msgr2=0x7f9260120ef0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T04:18:34.845 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f925dd9b640 1 --2- 172.21.15.186:0/2799147091 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9260120a60 0x7f9260120ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:34.846 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f925dd9b640 1 -- 172.21.15.186:0/2799147091 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f92600a1ec0 msgr2=0x7f92601cf0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:34.846 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f925dd9b640 1 --2- 172.21.15.186:0/2799147091 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f92600a1ec0 0x7f92601cf0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:34.846 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f925dd9b640 10 monclient: _finish_hunting 0 2024-04-26T04:18:34.846 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f925dd9b640 1 monclient: found mon.noname-b 2024-04-26T04:18:34.846 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f925dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:34.846 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f925dd9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.186:3300/0 2024-04-26T04:18:34.846 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f925dd9b640 1 -- 172.21.15.186:0/2799147091 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92601249b0 con 0x7f92600a22a0 2024-04-26T04:18:34.847 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f925dd9b640 1 -- 172.21.15.186:0/2799147091 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f92600a22a0 msgr2=0x7f926013b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:34.847 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f925dd9b640 1 --2- 172.21.15.186:0/2799147091 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f92600a22a0 0x7f926013b010 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f924800cc10 tx=0x7f924800a430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:34.847 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f9256ffd640 1 -- 172.21.15.186:0/2799147091 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9248012400 con 0x7f92600a22a0 2024-04-26T04:18:34.847 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f9256ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:34.847 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f9256ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T04:18:34.847 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f9256ffd640 10 monclient: dump: 2024-04-26T04:18:34.847 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-04-26T04:18:34.847 INFO:teuthology.orchestra.run.smithi186.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:34.847 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:34.847 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:34.847 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:34.847 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-04-26T04:18:34.847 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:34.847 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:34.847 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:34.848 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-04-26T04:18:34.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f9256ffd640 10 monclient: _finish_auth 0 2024-04-26T04:18:34.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f9256ffd640 10 monclient: _check_auth_tickets 2024-04-26T04:18:34.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f9256ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:34.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f9256ffd640 1 -- 172.21.15.186:0/2799147091 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92480125e0 con 0x7f92600a22a0 2024-04-26T04:18:34.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f9256ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:34.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f9256ffd640 1 -- 172.21.15.186:0/2799147091 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92480129d0 con 0x7f92600a22a0 2024-04-26T04:18:34.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f9264db0640 5 monclient: authenticate success, global_id 4102 2024-04-26T04:18:34.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f9264db0640 10 monclient: _renew_subs 2024-04-26T04:18:34.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f9264db0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-04-26T04:18:34.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f9264db0640 1 -- 172.21.15.186:0/2799147091 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92600a3720 con 0x7f92600a22a0 2024-04-26T04:18:34.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f9257fff640 4 set_mon_vals no callback set 2024-04-26T04:18:34.848 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f9256ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:34.849 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f9256ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T04:18:34.849 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f9256ffd640 10 monclient: dump: 2024-04-26T04:18:34.849 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-04-26T04:18:34.849 INFO:teuthology.orchestra.run.smithi186.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:34.849 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:34.849 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:34.849 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:34.849 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-04-26T04:18:34.849 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:34.849 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:34.849 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:34.849 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-04-26T04:18:34.849 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f9264db0640 10 monclient: _renew_subs 2024-04-26T04:18:34.849 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f9264db0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-04-26T04:18:34.850 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f9264db0640 1 -- 172.21.15.186:0/2799147091 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92601246d0 con 0x7f92600a22a0 2024-04-26T04:18:34.850 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f925dd9b640 1 -- 172.21.15.186:0/2799147091 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f92600a22a0 msgr2=0x7f926013b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T04:18:34.850 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f925dd9b640 1 -- 172.21.15.186:0/2799147091 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f92600a22a0 msgr2=0x7f926013b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T04:18:34.850 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f925dd9b640 1 --2- 172.21.15.186:0/2799147091 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f92600a22a0 0x7f926013b010 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f924800cc10 tx=0x7f924800a430 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:18:34.850 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f925dd9b640 1 --2- 172.21.15.186:0/2799147091 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f92600a22a0 0x7f926013b010 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f924800cc10 tx=0x7f924800a430 comp rx=0 tx=0).stop 2024-04-26T04:18:34.850 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f9256ffd640 10 monclient: ms_handle_reset current mon [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:18:34.850 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f9256ffd640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:34.850 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f9256ffd640 1 -- 172.21.15.186:0/2799147091 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f92600a22a0 msgr2=0x7f926013b010 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:18:34.850 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f9256ffd640 1 --2- 172.21.15.186:0/2799147091 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f92600a22a0 0x7f926013b010 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:34.850 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f9256ffd640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T04:18:34.850 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f9256ffd640 1 --2- 172.21.15.186:0/2799147091 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f92600a1ec0 0x7f92400058d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:34.850 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f9256ffd640 10 monclient(hunting): picked mon.a con 0x7f92600a1ec0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:18:34.850 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f9256ffd640 1 --2- 172.21.15.186:0/2799147091 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9260120a60 0x7f9240008310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:34.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f9256ffd640 10 monclient(hunting): picked mon.c con 0x7f9260120a60 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:34.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f9256ffd640 1 --2- 172.21.15.186:0/2799147091 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9240008d20 0x7f924000b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:34.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f9256ffd640 10 monclient(hunting): picked mon.b con 0x7f9240008d20 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:18:34.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f9256ffd640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:34.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f9256ffd640 1 -- 172.21.15.186:0/2799147091 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f924000bc60 con 0x7f92600a1ec0 2024-04-26T04:18:34.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f9256ffd640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:34.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f9256ffd640 1 -- 172.21.15.186:0/2799147091 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f924000bde0 con 0x7f9240008d20 2024-04-26T04:18:34.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f9256ffd640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:34.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f9256ffd640 1 -- 172.21.15.186:0/2799147091 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f924000bf60 con 0x7f9260120a60 2024-04-26T04:18:34.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f9256ffd640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:34.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f925ed9d640 1 --2- 172.21.15.186:0/2799147091 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9240008d20 0x7f924000b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:34.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f925ed9d640 10 monclient(hunting): get_auth_request con 0x7f9240008d20 auth_method 0 2024-04-26T04:18:34.851 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f925ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:34.852 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f925ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:34.852 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f925ed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:18:34.852 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f9264db0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:34.852 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f9264db0640 10 monclient(hunting): _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:34.852 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f925ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:34.852 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f925ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:34.852 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f925ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T04:18:34.852 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f925ed9d640 10 monclient(hunting): handle_auth_done global_id 4102 payload 293 2024-04-26T04:18:34.852 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f925ed9d640 1 -- 172.21.15.186:0/2799147091 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9260120a60 msgr2=0x7f9240008310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:34.852 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f925dd9b640 1 --2- 172.21.15.186:0/2799147091 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9260120a60 0x7f9240008310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:34.852 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f925ed9d640 1 --2- 172.21.15.186:0/2799147091 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9260120a60 0x7f9240008310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:34.852 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f925ed9d640 1 -- 172.21.15.186:0/2799147091 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f92600a1ec0 msgr2=0x7f92400058d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:34.852 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f925ed9d640 1 --2- 172.21.15.186:0/2799147091 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f92600a1ec0 0x7f92400058d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:34.853 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f925ed9d640 10 monclient: _finish_hunting 0 2024-04-26T04:18:34.853 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f925ed9d640 1 monclient: found mon.b 2024-04-26T04:18:34.853 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f925ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:34.853 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f925ed9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-04-26T04:18:34.853 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f925ed9d640 1 -- 172.21.15.186:0/2799147091 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) -- 0x7f924000c280 con 0x7f9240008d20 2024-04-26T04:18:34.853 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f925ed9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-04-26T04:18:34.853 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f925ed9d640 1 -- 172.21.15.186:0/2799147091 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9260062e10 con 0x7f9240008d20 2024-04-26T04:18:34.853 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f925ed9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:34.853 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f925ed9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-04-26T04:18:34.853 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f925ed9d640 1 -- 172.21.15.186:0/2799147091 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9250004250 con 0x7f9240008d20 2024-04-26T04:18:34.853 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f925ed9d640 10 monclient: _finish_auth 0 2024-04-26T04:18:34.853 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f925ed9d640 10 monclient: _check_auth_tickets 2024-04-26T04:18:34.853 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.828+0000 7f925ed9d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:34.853 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.832+0000 7f925ed9d640 1 -- 172.21.15.186:0/2799147091 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9240008d20 msgr2=0x7f924000b1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:34.854 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.832+0000 7f925ed9d640 1 --2- 172.21.15.186:0/2799147091 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9240008d20 0x7f924000b1b0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9250004480 tx=0x7f9248005490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:34.854 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.832+0000 7f9256ffd640 1 -- 172.21.15.186:0/2799147091 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9250003970 con 0x7f9240008d20 2024-04-26T04:18:34.854 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.832+0000 7f9256ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:34.854 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.832+0000 7f9256ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T04:18:34.854 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.832+0000 7f9256ffd640 10 monclient: dump: 2024-04-26T04:18:34.854 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-04-26T04:18:34.854 INFO:teuthology.orchestra.run.smithi186.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:34.854 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:34.854 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:34.854 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:34.854 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-04-26T04:18:34.854 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:34.854 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:34.854 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:34.855 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-04-26T04:18:34.855 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.832+0000 7f9256ffd640 1 -- 172.21.15.186:0/2799147091 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9250017070 con 0x7f9240008d20 2024-04-26T04:18:34.855 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.832+0000 7f9256ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:34.855 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.832+0000 7f9256ffd640 1 -- 172.21.15.186:0/2799147091 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f925001b800 con 0x7f9240008d20 2024-04-26T04:18:34.855 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.832+0000 7f925cd99640 4 set_mon_vals no callback set 2024-04-26T04:18:34.855 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.832+0000 7f9256ffd640 1 -- 172.21.15.186:0/2799147091 <== mon.1 v2:172.21.15.186:3300/0 4 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7f9250012d90 con 0x7f9240008d20 2024-04-26T04:18:34.855 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.840+0000 7f9256ffd640 1 -- 172.21.15.186:0/2799147091 <== mon.1 v2:172.21.15.186:3300/0 5 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f925000dd00 con 0x7f9240008d20 2024-04-26T04:18:34.856 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.840+0000 7f9256ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:34.856 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.840+0000 7f9256ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:18:34.856 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.844+0000 7f9256ffd640 1 -- 172.21.15.186:0/2799147091 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f92500123f0 con 0x7f9240008d20 2024-04-26T04:18:34.856 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.844+0000 7f9256ffd640 10 monclient: handle_mon_command_ack 1 not found 2024-04-26T04:18:34.913 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.912+0000 7f9264db0640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "2d37141b-8282-4c62-864a-84a75ebf8f6f", "id": 0}] 2024-04-26T04:18:34.913 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.912+0000 7f9264db0640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "2d37141b-8282-4c62-864a-84a75ebf8f6f", "id": 0}] 2024-04-26T04:18:34.913 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.912+0000 7f9264db0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-04-26T04:18:34.913 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.912+0000 7f9264db0640 1 -- 172.21.15.186:0/2799147091 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd new", "uuid": "2d37141b-8282-4c62-864a-84a75ebf8f6f", "id": 0} v 0) -- 0x7f9260067710 con 0x7f9240008d20 2024-04-26T04:18:34.918 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.916+0000 7f9256ffd640 1 -- 172.21.15.186:0/2799147091 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2d37141b-8282-4c62-864a-84a75ebf8f6f", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7f925001baf0 con 0x7f9240008d20 2024-04-26T04:18:34.918 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.916+0000 7f9256ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "2d37141b-8282-4c62-864a-84a75ebf8f6f", "id": 0}] 2024-04-26T04:18:34.918 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.916+0000 7f9256ffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:18:34.918 INFO:teuthology.orchestra.run.smithi186.stdout:0 2024-04-26T04:18:34.920 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.916+0000 7f9264db0640 10 monclient: shutdown 2024-04-26T04:18:34.920 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.916+0000 7f9264db0640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:34.920 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.916+0000 7f9264db0640 1 -- 172.21.15.186:0/2799147091 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9240008d20 msgr2=0x7f924000b1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:34.920 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.916+0000 7f9264db0640 1 --2- 172.21.15.186:0/2799147091 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9240008d20 0x7f924000b1b0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9250004480 tx=0x7f9248005490 comp rx=0 tx=0).stop 2024-04-26T04:18:34.920 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.920+0000 7f9264db0640 1 -- 172.21.15.186:0/2799147091 shutdown_connections 2024-04-26T04:18:34.920 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.920+0000 7f9264db0640 1 --2- 172.21.15.186:0/2799147091 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9240008d20 0x7f924000b1b0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:34.920 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.920+0000 7f9264db0640 1 --2- 172.21.15.186:0/2799147091 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9260120a60 0x7f9240008310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:34.920 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.920+0000 7f9264db0640 1 --2- 172.21.15.186:0/2799147091 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f92600a1ec0 0x7f92400058d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:34.920 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.920+0000 7f9264db0640 1 -- 172.21.15.186:0/2799147091 >> 172.21.15.186:0/2799147091 conn(0x7f92600a5fb0 msgr2=0x7f9260070d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:34.920 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.920+0000 7f9264db0640 1 -- 172.21.15.186:0/2799147091 shutdown_connections 2024-04-26T04:18:34.920 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:34.920+0000 7f9264db0640 1 -- 172.21.15.186:0/2799147091 wait complete. 2024-04-26T04:18:34.936 DEBUG:teuthology.orchestra.run.smithi186:> sudo ceph --cluster ceph osd new 7220e45a-72bb-4d8e-93c2-32005fa2c85e 1 2024-04-26T04:18:35.048 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.048+0000 7f75c34ed640 10 monclient: get_monmap_and_config 2024-04-26T04:18:35.049 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.048+0000 7f75c34ed640 10 monclient: build_initial_monmap 2024-04-26T04:18:35.049 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.048+0000 7f75c34ed640 10 monclient: monmap: 2024-04-26T04:18:35.049 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 0 2024-04-26T04:18:35.049 INFO:teuthology.orchestra.run.smithi186.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:35.049 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-04-26T04:18:35.049402+0000 2024-04-26T04:18:35.049 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-04-26T04:18:35.049402+0000 2024-04-26T04:18:35.050 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:35.050 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-04-26T04:18:35.050 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:18:35.050 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:18:35.050 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:18:35.050 INFO:teuthology.orchestra.run.smithi186.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:18:35.050 INFO:teuthology.orchestra.run.smithi186.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:18:35.050 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-04-26T04:18:35.050 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.048+0000 7f75c34ed640 1 Processor -- start 2024-04-26T04:18:35.050 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.048+0000 7f75c34ed640 1 -- start start 2024-04-26T04:18:35.050 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.048+0000 7f75c34ed640 10 monclient: init 2024-04-26T04:18:35.053 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.048+0000 7f75c34ed640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:35.053 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.048+0000 7f75c34ed640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-04-26T04:18:35.053 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.048+0000 7f75c34ed640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f75bc0a2ae0 0x7f75bc0a1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:35.053 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.048+0000 7f75c34ed640 10 monclient(hunting): picked mon.noname-a con 0x7f75bc0a2ae0 addr v2:172.21.15.63:3300/0 2024-04-26T04:18:35.053 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.048+0000 7f75c34ed640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f75bc0a2ec0 addr v1:172.21.15.63:6789/0 2024-04-26T04:18:35.053 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.048+0000 7f75c34ed640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f75bc135310 addr v1:172.21.15.186:6789/0 2024-04-26T04:18:35.053 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.048+0000 7f75c34ed640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75bc0a1550 con 0x7f75bc0a2ec0 2024-04-26T04:18:35.053 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.048+0000 7f75c34ed640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75bc09fa70 con 0x7f75bc135310 2024-04-26T04:18:35.053 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.048+0000 7f75c34ed640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:35.053 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.048+0000 7f75c34ed640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7f75bc09fd50 con 0x7f75bc0a2ae0 2024-04-26T04:18:35.054 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.048+0000 7f75c34ed640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:35.054 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.048+0000 7f75c34ed640 10 monclient(hunting): authenticate will time out at 944.826203s 2024-04-26T04:18:35.054 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.048+0000 7f75c1a8a640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f75bc135310 0x7f75bc13cf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:46932/0 (socket says 172.21.15.186:46932) 2024-04-26T04:18:35.054 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.048+0000 7f75c1a8a640 1 -- 172.21.15.186:0/3372724552 learned_addr learned my addr 172.21.15.186:0/3372724552 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-04-26T04:18:35.054 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.048+0000 7f75c1289640 1 --2- 172.21.15.186:0/3372724552 >> v2:172.21.15.63:3300/0 conn(0x7f75bc0a2ae0 0x7f75bc0a1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:35.054 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.048+0000 7f75c1289640 10 monclient(hunting): get_auth_request con 0x7f75bc0a2ae0 auth_method 0 2024-04-26T04:18:35.054 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.048+0000 7f75c1289640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:35.054 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.048+0000 7f75c1289640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:35.054 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.048+0000 7f75c1289640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:35.055 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.048+0000 7f75c1289640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:35.055 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.048+0000 7f75c1289640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:35.055 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.048+0000 7f75c1289640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:35.055 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.052+0000 7f75c1a8a640 1 -- 172.21.15.186:0/3372724552 >> v1:172.21.15.186:6789/0 conn(0x7f75bc135310 legacy=0x7f75bc13cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:35.055 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.052+0000 7f75c1a8a640 1 -- 172.21.15.186:0/3372724552 >> v1:172.21.15.186:6789/0 conn(0x7f75bc135310 legacy=0x7f75bc13cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-04-26T04:18:35.055 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.052+0000 7f75c1a8a640 1 -- 172.21.15.186:0/3372724552 >> v1:172.21.15.186:6789/0 conn(0x7f75bc135310 legacy=0x7f75bc13cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T04:18:35.055 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.052+0000 7f75c1a8a640 1 --1- 172.21.15.186:0/3372724552 >> v1:172.21.15.186:6789/0 conn(0x7f75bc135310 0x7f75bc13cf40 :-1 s=OPENED pgs=13 cs=1 l=1).handle_message read tag failed 2024-04-26T04:18:35.055 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.052+0000 7f75c1a8a640 1 --1- 172.21.15.186:0/3372724552 >> v1:172.21.15.186:6789/0 conn(0x7f75bc135310 0x7f75bc13cf40 :-1 s=OPENED pgs=13 cs=1 l=1).fault on lossy channel, failing 2024-04-26T04:18:35.055 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.052+0000 7f75b3fff640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.186:6789/0 2024-04-26T04:18:35.055 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.052+0000 7f75c0a88640 1 -- 172.21.15.186:0/3372724552 >> v1:172.21.15.63:6789/0 conn(0x7f75bc0a2ec0 legacy=0x7f75bc134c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:35.055 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.052+0000 7f75c1289640 10 monclient(hunting): handle_auth_done global_id 4113 payload 274 2024-04-26T04:18:35.055 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.052+0000 7f75c1289640 1 -- 172.21.15.186:0/3372724552 >> v1:172.21.15.186:6789/0 conn(0x7f75bc135310 legacy=0x7f75bc13cf40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:18:35.055 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.052+0000 7f75c1289640 1 -- 172.21.15.186:0/3372724552 >> v1:172.21.15.63:6789/0 conn(0x7f75bc0a2ec0 legacy=0x7f75bc134c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:35.055 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.052+0000 7f75c1289640 10 monclient: _finish_hunting 0 2024-04-26T04:18:35.056 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.052+0000 7f75c1289640 1 monclient: found mon.noname-a 2024-04-26T04:18:35.056 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.052+0000 7f75c1289640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:35.056 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.052+0000 7f75c1289640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:18:35.056 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.052+0000 7f75c1289640 1 -- 172.21.15.186:0/3372724552 --> v2:172.21.15.63:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75bc0a0030 con 0x7f75bc0a2ae0 2024-04-26T04:18:35.057 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.056+0000 7f75c1289640 1 -- 172.21.15.186:0/3372724552 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f75bc0a2ae0 msgr2=0x7f75bc0a1cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:35.057 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.056+0000 7f75c1289640 1 --2- 172.21.15.186:0/3372724552 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f75bc0a2ae0 0x7f75bc0a1cd0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f75bc0a33a0 tx=0x7f75ac01c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=285d0ffe4a16461f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:35.058 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.056+0000 7f75b3fff640 1 -- 172.21.15.186:0/3372724552 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75ac0b0070 con 0x7f75bc0a2ae0 2024-04-26T04:18:35.058 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.056+0000 7f75b3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:35.058 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.056+0000 7f75b3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:18:35.058 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.056+0000 7f75b3fff640 10 monclient: dump: 2024-04-26T04:18:35.058 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-04-26T04:18:35.058 INFO:teuthology.orchestra.run.smithi186.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:35.058 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:35.058 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:35.058 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:35.058 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-04-26T04:18:35.058 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:35.058 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:35.059 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:35.059 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-04-26T04:18:35.059 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.056+0000 7f75b3fff640 10 monclient: _finish_auth 0 2024-04-26T04:18:35.059 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.056+0000 7f75b3fff640 10 monclient: _check_auth_tickets 2024-04-26T04:18:35.059 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.056+0000 7f75b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:35.059 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.056+0000 7f75b3fff640 1 -- 172.21.15.186:0/3372724552 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75ac0ab600 con 0x7f75bc0a2ae0 2024-04-26T04:18:35.059 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.056+0000 7f75b3fff640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:35.059 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.056+0000 7f75b3fff640 1 -- 172.21.15.186:0/3372724552 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75ac0ab9d0 con 0x7f75bc0a2ae0 2024-04-26T04:18:35.059 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.056+0000 7f75b3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:35.059 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.056+0000 7f75b3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:18:35.059 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.056+0000 7f75b3fff640 10 monclient: dump: 2024-04-26T04:18:35.059 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-04-26T04:18:35.059 INFO:teuthology.orchestra.run.smithi186.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:35.059 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:35.059 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:35.060 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:35.060 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-04-26T04:18:35.060 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:35.060 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:35.060 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:35.060 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-04-26T04:18:35.060 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.056+0000 7f75c34ed640 5 monclient: authenticate success, global_id 4113 2024-04-26T04:18:35.060 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.056+0000 7f75c34ed640 10 monclient: get_monmap_and_config success 2024-04-26T04:18:35.060 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.056+0000 7f75c34ed640 4 set_mon_vals no callback set 2024-04-26T04:18:35.065 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.056+0000 7f75c34ed640 10 monclient: shutdown 2024-04-26T04:18:35.065 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.056+0000 7f75c34ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:35.065 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.056+0000 7f75c34ed640 1 -- 172.21.15.186:0/3372724552 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f75bc0a2ae0 msgr2=0x7f75bc0a1cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:35.065 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.056+0000 7f75c34ed640 1 --2- 172.21.15.186:0/3372724552 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f75bc0a2ae0 0x7f75bc0a1cd0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f75bc0a33a0 tx=0x7f75ac01c340 comp rx=0 tx=0).stop 2024-04-26T04:18:35.065 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.056+0000 7f75c34ed640 1 -- 172.21.15.186:0/3372724552 shutdown_connections 2024-04-26T04:18:35.065 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.056+0000 7f75c34ed640 1 --2- 172.21.15.186:0/3372724552 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f75bc0a2ae0 0x7f75bc0a1cd0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:35.065 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.056+0000 7f75c34ed640 1 -- 172.21.15.186:0/3372724552 >> 172.21.15.186:0/3372724552 conn(0x7f75bc0a73d0 msgr2=0x7f75bc0a77b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:35.066 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.056+0000 7f75c34ed640 1 -- 172.21.15.186:0/3372724552 shutdown_connections 2024-04-26T04:18:35.066 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.056+0000 7f75c34ed640 1 -- 172.21.15.186:0/3372724552 wait complete. 2024-04-26T04:18:35.066 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.056+0000 7f75c34ed640 10 monclient: build_initial_monmap 2024-04-26T04:18:35.066 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.056+0000 7f75c34ed640 10 monclient: monmap: 2024-04-26T04:18:35.066 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 0 2024-04-26T04:18:35.066 INFO:teuthology.orchestra.run.smithi186.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:18:35.066 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 0.000000 2024-04-26T04:18:35.066 INFO:teuthology.orchestra.run.smithi186.stderr:created 0.000000 2024-04-26T04:18:35.066 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:35.066 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-04-26T04:18:35.066 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:18:35.066 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:18:35.066 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:18:35.066 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-04-26T04:18:35.067 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.060+0000 7f75c34ed640 1 Processor -- start 2024-04-26T04:18:35.067 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.060+0000 7f75c34ed640 1 -- start start 2024-04-26T04:18:35.067 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.060+0000 7f75c34ed640 10 monclient: init 2024-04-26T04:18:35.067 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.060+0000 7f75c34ed640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:35.067 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.060+0000 7f75c34ed640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T04:18:35.067 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.060+0000 7f75c34ed640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75bc0a2ae0 0x7f75bc139fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:35.067 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.060+0000 7f75c34ed640 10 monclient(hunting): picked mon.noname-b con 0x7f75bc0a2ae0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:18:35.067 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.060+0000 7f75c34ed640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f75bc0a2ec0 0x7f75bc13a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:35.067 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.060+0000 7f75c34ed640 10 monclient(hunting): picked mon.noname-c con 0x7f75bc0a2ec0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:35.067 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.060+0000 7f75c34ed640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f75bc135310 0x7f75bc13aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:35.067 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.060+0000 7f75c34ed640 10 monclient(hunting): picked mon.noname-a con 0x7f75bc135310 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:18:35.067 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.060+0000 7f75c34ed640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:35.067 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.060+0000 7f75c34ed640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f75bc141090 con 0x7f75bc135310 2024-04-26T04:18:35.067 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.060+0000 7f75c34ed640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:35.067 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.060+0000 7f75c34ed640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f75bc141210 con 0x7f75bc0a2ae0 2024-04-26T04:18:35.068 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.060+0000 7f75c34ed640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:35.068 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.060+0000 7f75c34ed640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f75bc141390 con 0x7f75bc0a2ec0 2024-04-26T04:18:35.068 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.060+0000 7f75c34ed640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:35.068 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.060+0000 7f75c34ed640 10 monclient(hunting): authenticate will time out at 944.836738s 2024-04-26T04:18:35.068 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.060+0000 7f75c1289640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75bc0a2ae0 0x7f75bc139fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:35.068 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.060+0000 7f75c1289640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75bc0a2ae0 0x7f75bc139fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:40948/0 (socket says 172.21.15.186:40948) 2024-04-26T04:18:35.068 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.060+0000 7f75c1289640 1 -- 172.21.15.186:0/841524574 learned_addr learned my addr 172.21.15.186:0/841524574 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-04-26T04:18:35.068 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.060+0000 7f75c1289640 10 monclient(hunting): get_auth_request con 0x7f75bc0a2ae0 auth_method 0 2024-04-26T04:18:35.068 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.060+0000 7f75c1289640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:35.068 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.060+0000 7f75c1289640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:35.068 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.060+0000 7f75c1289640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:35.068 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.060+0000 7f75c1289640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:35.068 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.060+0000 7f75c1289640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:35.068 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.060+0000 7f75c1289640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:35.069 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.060+0000 7f75c0a88640 1 --2- 172.21.15.186:0/841524574 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f75bc0a2ec0 0x7f75bc13a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:35.069 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.060+0000 7f75c1a8a640 1 --2- 172.21.15.186:0/841524574 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f75bc135310 0x7f75bc13aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:35.069 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.060+0000 7f75c0a88640 10 monclient(hunting): get_auth_request con 0x7f75bc0a2ec0 auth_method 0 2024-04-26T04:18:35.069 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.060+0000 7f75c0a88640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:35.069 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.060+0000 7f75c0a88640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:35.069 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.060+0000 7f75c0a88640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:35.069 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.060+0000 7f75c1289640 10 monclient(hunting): handle_auth_done global_id 4108 payload 824 2024-04-26T04:18:35.069 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.060+0000 7f75c1289640 1 -- 172.21.15.186:0/841524574 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f75bc0a2ec0 msgr2=0x7f75bc13a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:35.069 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.060+0000 7f75c1289640 1 --2- 172.21.15.186:0/841524574 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f75bc0a2ec0 0x7f75bc13a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:35.069 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.060+0000 7f75c1289640 1 -- 172.21.15.186:0/841524574 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f75bc135310 msgr2=0x7f75bc13aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:35.069 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.060+0000 7f75c1289640 1 --2- 172.21.15.186:0/841524574 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f75bc135310 0x7f75bc13aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:35.069 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.060+0000 7f75c1289640 10 monclient: _finish_hunting 0 2024-04-26T04:18:35.069 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.060+0000 7f75c1289640 1 monclient: found mon.noname-b 2024-04-26T04:18:35.069 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.060+0000 7f75c1289640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:35.069 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.060+0000 7f75c1289640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.186:3300/0 2024-04-26T04:18:35.070 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.060+0000 7f75c1289640 1 -- 172.21.15.186:0/841524574 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75bc13d640 con 0x7f75bc0a2ae0 2024-04-26T04:18:35.070 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.060+0000 7f75c1a8a640 10 monclient: get_auth_request con 0x7f75bc135310 auth_method 0 2024-04-26T04:18:35.070 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.060+0000 7f75c1a8a640 1 --2- 172.21.15.186:0/841524574 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f75bc135310 0x7f75bc13aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T04:18:35.070 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.060+0000 7f75c1289640 1 -- 172.21.15.186:0/841524574 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75bc0a2ae0 msgr2=0x7f75bc139fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:35.070 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.060+0000 7f75c1289640 1 --2- 172.21.15.186:0/841524574 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75bc0a2ae0 0x7f75bc139fd0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f75ac018f50 tx=0x7f75ac0190e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:35.070 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.060+0000 7f75b1ffb640 1 -- 172.21.15.186:0/841524574 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75ac0b8020 con 0x7f75bc0a2ae0 2024-04-26T04:18:35.070 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.060+0000 7f75b1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:35.070 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.060+0000 7f75b1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T04:18:35.070 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.060+0000 7f75b1ffb640 10 monclient: dump: 2024-04-26T04:18:35.070 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-04-26T04:18:35.070 INFO:teuthology.orchestra.run.smithi186.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:35.070 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:35.071 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:35.071 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:35.071 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-04-26T04:18:35.071 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:35.071 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:35.071 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:35.071 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-04-26T04:18:35.071 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.060+0000 7f75b1ffb640 10 monclient: _finish_auth 0 2024-04-26T04:18:35.071 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.060+0000 7f75b1ffb640 10 monclient: _check_auth_tickets 2024-04-26T04:18:35.071 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.060+0000 7f75b1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:35.072 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.060+0000 7f75b1ffb640 1 -- 172.21.15.186:0/841524574 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75ac0b4810 con 0x7f75bc0a2ae0 2024-04-26T04:18:35.072 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.060+0000 7f75c34ed640 5 monclient: authenticate success, global_id 4108 2024-04-26T04:18:35.072 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.060+0000 7f75c34ed640 10 monclient: _renew_subs 2024-04-26T04:18:35.072 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.060+0000 7f75c34ed640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-04-26T04:18:35.072 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.060+0000 7f75c34ed640 1 -- 172.21.15.186:0/841524574 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75bc09dd30 con 0x7f75bc0a2ae0 2024-04-26T04:18:35.072 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.060+0000 7f75b1ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:35.072 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.060+0000 7f75b1ffb640 1 -- 172.21.15.186:0/841524574 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75ac0b0040 con 0x7f75bc0a2ae0 2024-04-26T04:18:35.072 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.060+0000 7f75b1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:35.072 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.060+0000 7f75b1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T04:18:35.072 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.060+0000 7f75b1ffb640 10 monclient: dump: 2024-04-26T04:18:35.072 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-04-26T04:18:35.072 INFO:teuthology.orchestra.run.smithi186.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:35.072 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:35.073 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:35.073 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:35.073 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-04-26T04:18:35.073 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:35.073 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:35.073 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:35.073 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-04-26T04:18:35.073 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.060+0000 7f75b37fe640 4 set_mon_vals no callback set 2024-04-26T04:18:35.073 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.060+0000 7f75c34ed640 10 monclient: _renew_subs 2024-04-26T04:18:35.073 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.060+0000 7f75c34ed640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-04-26T04:18:35.073 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.060+0000 7f75c34ed640 1 -- 172.21.15.186:0/841524574 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75bc13c670 con 0x7f75bc0a2ae0 2024-04-26T04:18:35.073 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.064+0000 7f75b1ffb640 1 -- 172.21.15.186:0/841524574 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f75ac0bb020 con 0x7f75bc0a2ae0 2024-04-26T04:18:35.074 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.064+0000 7f75b1ffb640 1 -- 172.21.15.186:0/841524574 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7f75ac0bdbc0 con 0x7f75bc0a2ae0 2024-04-26T04:18:35.074 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.064+0000 7f75c34ed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:35.074 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.064+0000 7f75c34ed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:35.074 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.064+0000 7f75c34ed640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-04-26T04:18:35.074 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.064+0000 7f75c34ed640 1 -- 172.21.15.186:0/841524574 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75bc062e10 con 0x7f75bc0a2ae0 2024-04-26T04:18:35.075 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.064+0000 7f75b1ffb640 1 -- 172.21.15.186:0/841524574 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f75ac0c2030 con 0x7f75bc0a2ae0 2024-04-26T04:18:35.075 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.064+0000 7f75b1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:35.075 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.064+0000 7f75b1ffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:18:35.134 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.132+0000 7f75c34ed640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "7220e45a-72bb-4d8e-93c2-32005fa2c85e", "id": 1}] 2024-04-26T04:18:35.134 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.132+0000 7f75c34ed640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "7220e45a-72bb-4d8e-93c2-32005fa2c85e", "id": 1}] 2024-04-26T04:18:35.134 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.132+0000 7f75c34ed640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-04-26T04:18:35.135 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.132+0000 7f75c34ed640 1 -- 172.21.15.186:0/841524574 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd new", "uuid": "7220e45a-72bb-4d8e-93c2-32005fa2c85e", "id": 1} v 0) -- 0x7f75bc09e9f0 con 0x7f75bc0a2ae0 2024-04-26T04:18:35.136 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-26T04:18:35.136+0000 7f2753209180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-26T04:18:35.141 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.140+0000 7f75b1ffb640 1 -- 172.21.15.186:0/841524574 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7220e45a-72bb-4d8e-93c2-32005fa2c85e", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7f75ac016050 con 0x7f75bc0a2ae0 2024-04-26T04:18:35.141 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.140+0000 7f75b1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "7220e45a-72bb-4d8e-93c2-32005fa2c85e", "id": 1}] 2024-04-26T04:18:35.141 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.140+0000 7f75b1ffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:18:35.141 INFO:teuthology.orchestra.run.smithi186.stdout:1 2024-04-26T04:18:35.142 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.140+0000 7f75c34ed640 10 monclient: shutdown 2024-04-26T04:18:35.142 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.140+0000 7f75c34ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:35.143 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.140+0000 7f75c34ed640 1 -- 172.21.15.186:0/841524574 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75bc0a2ae0 msgr2=0x7f75bc139fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:35.143 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.140+0000 7f75c34ed640 1 --2- 172.21.15.186:0/841524574 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75bc0a2ae0 0x7f75bc139fd0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f75ac018f50 tx=0x7f75ac0190e0 comp rx=0 tx=0).stop 2024-04-26T04:18:35.143 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.140+0000 7f75c34ed640 1 -- 172.21.15.186:0/841524574 shutdown_connections 2024-04-26T04:18:35.143 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.140+0000 7f75c34ed640 1 --2- 172.21.15.186:0/841524574 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f75bc135310 0x7f75bc13aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:35.143 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.140+0000 7f75c34ed640 1 --2- 172.21.15.186:0/841524574 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f75bc0a2ec0 0x7f75bc13a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:35.143 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.140+0000 7f75c34ed640 1 --2- 172.21.15.186:0/841524574 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75bc0a2ae0 0x7f75bc139fd0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:35.144 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.140+0000 7f75c34ed640 1 -- 172.21.15.186:0/841524574 >> 172.21.15.186:0/841524574 conn(0x7f75bc0a73d0 msgr2=0x7f75bc0a69f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:35.144 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.140+0000 7f75c34ed640 1 -- 172.21.15.186:0/841524574 shutdown_connections 2024-04-26T04:18:35.144 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.140+0000 7f75c34ed640 1 -- 172.21.15.186:0/841524574 wait complete. 2024-04-26T04:18:35.160 DEBUG:teuthology.orchestra.run.smithi186:> sudo ceph --cluster ceph osd new 913de4d8-31af-4394-a0ee-194cc31f822f 2 2024-04-26T04:18:35.267 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.264+0000 7fddaf1cd640 10 monclient: get_monmap_and_config 2024-04-26T04:18:35.267 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.264+0000 7fddaf1cd640 10 monclient: build_initial_monmap 2024-04-26T04:18:35.267 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.264+0000 7fddaf1cd640 10 monclient: monmap: 2024-04-26T04:18:35.267 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 0 2024-04-26T04:18:35.267 INFO:teuthology.orchestra.run.smithi186.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:35.267 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-04-26T04:18:35.268486+0000 2024-04-26T04:18:35.267 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-04-26T04:18:35.268486+0000 2024-04-26T04:18:35.268 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:35.268 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-04-26T04:18:35.268 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:18:35.268 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:18:35.268 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:18:35.268 INFO:teuthology.orchestra.run.smithi186.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:18:35.268 INFO:teuthology.orchestra.run.smithi186.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:18:35.268 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-04-26T04:18:35.271 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.268+0000 7fddaf1cd640 1 Processor -- start 2024-04-26T04:18:35.271 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.268+0000 7fddaf1cd640 1 -- start start 2024-04-26T04:18:35.271 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.268+0000 7fddaf1cd640 10 monclient: init 2024-04-26T04:18:35.271 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.268+0000 7fddaf1cd640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:35.271 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.268+0000 7fddaf1cd640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-04-26T04:18:35.271 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.268+0000 7fddaf1cd640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7fdda806bde0 0x7fdda81350a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:35.271 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.268+0000 7fddaf1cd640 10 monclient(hunting): picked mon.noname-a con 0x7fdda806bde0 addr v2:172.21.15.63:3300/0 2024-04-26T04:18:35.271 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.268+0000 7fddaf1cd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdda81355e0 addr v1:172.21.15.186:6789/0 2024-04-26T04:18:35.271 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.268+0000 7fddaf1cd640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fdda80a1bb0 0x7fdda813ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:35.271 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.268+0000 7fddaf1cd640 10 monclient(hunting): picked mon.noname-b con 0x7fdda80a1bb0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:35.272 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.268+0000 7fddaf1cd640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdda81349b0 con 0x7fdda81355e0 2024-04-26T04:18:35.272 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.268+0000 7fddaf1cd640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:35.272 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.268+0000 7fddaf1cd640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7fdda80a31a0 con 0x7fdda806bde0 2024-04-26T04:18:35.272 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.268+0000 7fddaf1cd640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:35.272 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.268+0000 7fddaf1cd640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7fdda806c1c0 con 0x7fdda80a1bb0 2024-04-26T04:18:35.272 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.268+0000 7fddaf1cd640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:35.272 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.268+0000 7fddaf1cd640 10 monclient(hunting): authenticate will time out at 945.044996s 2024-04-26T04:18:35.272 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.268+0000 7fdda7fff640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fdda81355e0 0x7fdda80a14a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:46942/0 (socket says 172.21.15.186:46942) 2024-04-26T04:18:35.272 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.268+0000 7fdda7fff640 1 -- 172.21.15.186:0/139223772 learned_addr learned my addr 172.21.15.186:0/139223772 (peer_addr_for_me v1:172.21.15.186:0/0) 2024-04-26T04:18:35.272 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.268+0000 7fddacf69640 1 --2- 172.21.15.186:0/139223772 >> v2:172.21.15.63:3300/0 conn(0x7fdda806bde0 0x7fdda81350a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:35.272 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.268+0000 7fddacf69640 10 monclient(hunting): get_auth_request con 0x7fdda806bde0 auth_method 0 2024-04-26T04:18:35.272 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.268+0000 7fddacf69640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:35.273 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.268+0000 7fddacf69640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:35.273 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.268+0000 7fddacf69640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:35.273 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.268+0000 7fddad76a640 1 --2- 172.21.15.186:0/139223772 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fdda80a1bb0 0x7fdda813ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:35.273 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.268+0000 7fddad76a640 10 monclient(hunting): get_auth_request con 0x7fdda80a1bb0 auth_method 0 2024-04-26T04:18:35.273 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.268+0000 7fddad76a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:35.273 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.268+0000 7fddad76a640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:35.273 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.268+0000 7fddad76a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:35.273 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.268+0000 7fddacf69640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:35.273 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.268+0000 7fddacf69640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:35.273 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.268+0000 7fddacf69640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:35.273 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.268+0000 7fddad76a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:35.274 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.268+0000 7fddad76a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:35.274 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.268+0000 7fddad76a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:35.274 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.268+0000 7fddacf69640 10 monclient(hunting): handle_auth_done global_id 4116 payload 274 2024-04-26T04:18:35.274 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.268+0000 7fddacf69640 1 -- 172.21.15.186:0/139223772 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fdda80a1bb0 msgr2=0x7fdda813ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:35.274 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.268+0000 7fddacf69640 1 --2- 172.21.15.186:0/139223772 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fdda80a1bb0 0x7fdda813ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:35.274 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.268+0000 7fddacf69640 1 -- 172.21.15.186:0/139223772 >> v1:172.21.15.186:6789/0 conn(0x7fdda81355e0 legacy=0x7fdda80a14a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:35.274 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.268+0000 7fddacf69640 10 monclient: _finish_hunting 0 2024-04-26T04:18:35.274 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.268+0000 7fddacf69640 1 monclient: found mon.noname-a 2024-04-26T04:18:35.274 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.268+0000 7fddacf69640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:35.274 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.268+0000 7fddacf69640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:18:35.275 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.268+0000 7fddacf69640 1 -- 172.21.15.186:0/139223772 --> v2:172.21.15.63:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdda8134c50 con 0x7fdda806bde0 2024-04-26T04:18:35.279 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.272+0000 7fddacf69640 1 -- 172.21.15.186:0/139223772 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdda806bde0 msgr2=0x7fdda81350a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:35.279 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.272+0000 7fddacf69640 1 --2- 172.21.15.186:0/139223772 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdda806bde0 0x7fdda81350a0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fdda806c3f0 tx=0x7fdd9c01c2f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a9597f15c0035e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:35.279 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.272+0000 7fdda77fe640 1 -- 172.21.15.186:0/139223772 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd9c0b0070 con 0x7fdda806bde0 2024-04-26T04:18:35.279 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.272+0000 7fdda77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:35.279 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.272+0000 7fdda77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:18:35.279 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.272+0000 7fdda77fe640 10 monclient: dump: 2024-04-26T04:18:35.279 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-04-26T04:18:35.279 INFO:teuthology.orchestra.run.smithi186.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:35.279 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:35.280 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:35.280 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:35.280 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-04-26T04:18:35.280 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:35.280 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:35.280 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:35.280 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-04-26T04:18:35.280 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.272+0000 7fdda77fe640 10 monclient: _finish_auth 0 2024-04-26T04:18:35.280 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.272+0000 7fdda77fe640 10 monclient: _check_auth_tickets 2024-04-26T04:18:35.280 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.272+0000 7fdda77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:35.280 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.272+0000 7fdda77fe640 1 -- 172.21.15.186:0/139223772 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd9c0a4e50 con 0x7fdda806bde0 2024-04-26T04:18:35.280 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.272+0000 7fdda77fe640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:35.281 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.272+0000 7fdda77fe640 1 -- 172.21.15.186:0/139223772 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd9c0ab820 con 0x7fdda806bde0 2024-04-26T04:18:35.281 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.272+0000 7fddaf1cd640 5 monclient: authenticate success, global_id 4116 2024-04-26T04:18:35.281 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.272+0000 7fddaf1cd640 10 monclient: get_monmap_and_config success 2024-04-26T04:18:35.281 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.272+0000 7fddaf1cd640 4 set_mon_vals no callback set 2024-04-26T04:18:35.281 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.272+0000 7fdda77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:35.281 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.272+0000 7fdda77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:18:35.281 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.272+0000 7fdda77fe640 10 monclient: dump: 2024-04-26T04:18:35.281 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-04-26T04:18:35.281 INFO:teuthology.orchestra.run.smithi186.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:35.281 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:35.281 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:35.281 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:35.282 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-04-26T04:18:35.282 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:35.282 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:35.282 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:35.282 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-04-26T04:18:35.282 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.272+0000 7fddaf1cd640 10 monclient: shutdown 2024-04-26T04:18:35.282 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.272+0000 7fddaf1cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:35.282 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.272+0000 7fddaf1cd640 1 -- 172.21.15.186:0/139223772 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdda806bde0 msgr2=0x7fdda81350a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:35.282 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.272+0000 7fddaf1cd640 1 --2- 172.21.15.186:0/139223772 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdda806bde0 0x7fdda81350a0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fdda806c3f0 tx=0x7fdd9c01c2f0 comp rx=0 tx=0).stop 2024-04-26T04:18:35.282 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.272+0000 7fddaf1cd640 1 -- 172.21.15.186:0/139223772 shutdown_connections 2024-04-26T04:18:35.283 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.272+0000 7fddaf1cd640 1 --2- 172.21.15.186:0/139223772 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fdda80a1bb0 0x7fdda813ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:35.283 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.272+0000 7fddaf1cd640 1 --2- 172.21.15.186:0/139223772 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdda806bde0 0x7fdda81350a0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:35.283 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.272+0000 7fddaf1cd640 1 -- 172.21.15.186:0/139223772 >> 172.21.15.186:0/139223772 conn(0x7fdda80a6090 msgr2=0x7fdda80a2590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:35.283 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.272+0000 7fddaf1cd640 1 -- 172.21.15.186:0/139223772 shutdown_connections 2024-04-26T04:18:35.283 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.272+0000 7fddaf1cd640 1 -- 172.21.15.186:0/139223772 wait complete. 2024-04-26T04:18:35.283 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.276+0000 7fddaf1cd640 10 monclient: build_initial_monmap 2024-04-26T04:18:35.283 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.276+0000 7fddaf1cd640 10 monclient: monmap: 2024-04-26T04:18:35.283 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 0 2024-04-26T04:18:35.283 INFO:teuthology.orchestra.run.smithi186.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:18:35.283 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 0.000000 2024-04-26T04:18:35.283 INFO:teuthology.orchestra.run.smithi186.stderr:created 0.000000 2024-04-26T04:18:35.283 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:35.283 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-04-26T04:18:35.284 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:18:35.284 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:18:35.284 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:18:35.284 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-04-26T04:18:35.284 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.276+0000 7fddaf1cd640 1 Processor -- start 2024-04-26T04:18:35.284 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.276+0000 7fddaf1cd640 1 -- start start 2024-04-26T04:18:35.284 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.276+0000 7fddaf1cd640 10 monclient: init 2024-04-26T04:18:35.284 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.276+0000 7fddaf1cd640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:35.284 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.276+0000 7fddaf1cd640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T04:18:35.284 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.276+0000 7fddaf1cd640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fdda806bde0 0x7fdda81cf390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:35.284 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.276+0000 7fddaf1cd640 10 monclient(hunting): picked mon.noname-c con 0x7fdda806bde0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:35.284 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.276+0000 7fddaf1cd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdda80a1bb0 0x7fdda8133860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:35.285 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.276+0000 7fddaf1cd640 10 monclient(hunting): picked mon.noname-b con 0x7fdda80a1bb0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:18:35.285 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.276+0000 7fddaf1cd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdda81355e0 0x7fdda8133da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:35.285 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.276+0000 7fddaf1cd640 10 monclient(hunting): picked mon.noname-a con 0x7fdda81355e0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:18:35.285 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.276+0000 7fddaf1cd640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:35.285 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.276+0000 7fddaf1cd640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fdda8072c40 con 0x7fdda81355e0 2024-04-26T04:18:35.285 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.276+0000 7fddaf1cd640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:35.285 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.276+0000 7fddaf1cd640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fdda8072ac0 con 0x7fdda80a1bb0 2024-04-26T04:18:35.285 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.276+0000 7fddaf1cd640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:35.285 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.276+0000 7fddaf1cd640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7fdda813fa50 con 0x7fdda806bde0 2024-04-26T04:18:35.285 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.276+0000 7fddaf1cd640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:35.286 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.276+0000 7fddaf1cd640 10 monclient(hunting): authenticate will time out at 945.053131s 2024-04-26T04:18:35.286 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.276+0000 7fdda7fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdda80a1bb0 0x7fdda8133860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:35.286 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.276+0000 7fdda7fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdda80a1bb0 0x7fdda8133860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:40962/0 (socket says 172.21.15.186:40962) 2024-04-26T04:18:35.286 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.276+0000 7fdda7fff640 1 -- 172.21.15.186:0/1262843237 learned_addr learned my addr 172.21.15.186:0/1262843237 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-04-26T04:18:35.286 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.276+0000 7fdda7fff640 10 monclient(hunting): get_auth_request con 0x7fdda80a1bb0 auth_method 0 2024-04-26T04:18:35.286 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.276+0000 7fdda7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:35.286 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.276+0000 7fdda7fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:35.286 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.276+0000 7fdda7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:35.286 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.276+0000 7fdda7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:35.286 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.276+0000 7fdda7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:35.286 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.276+0000 7fdda7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:35.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.276+0000 7fddacf69640 1 --2- 172.21.15.186:0/1262843237 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fdda806bde0 0x7fdda81cf390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:35.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.276+0000 7fddacf69640 10 monclient(hunting): get_auth_request con 0x7fdda806bde0 auth_method 0 2024-04-26T04:18:35.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.276+0000 7fddad76a640 1 --2- 172.21.15.186:0/1262843237 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdda81355e0 0x7fdda8133da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:35.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.276+0000 7fddacf69640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:35.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.276+0000 7fddacf69640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:35.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.276+0000 7fddacf69640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:35.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.276+0000 7fddad76a640 10 monclient(hunting): get_auth_request con 0x7fdda81355e0 auth_method 0 2024-04-26T04:18:35.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.276+0000 7fddad76a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:35.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.276+0000 7fddad76a640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:35.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.276+0000 7fddad76a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:35.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.276+0000 7fdda7fff640 10 monclient(hunting): handle_auth_done global_id 4111 payload 824 2024-04-26T04:18:35.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.276+0000 7fdda7fff640 1 -- 172.21.15.186:0/1262843237 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fdda806bde0 msgr2=0x7fdda81cf390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:35.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.276+0000 7fdda7fff640 1 --2- 172.21.15.186:0/1262843237 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fdda806bde0 0x7fdda81cf390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:35.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.276+0000 7fdda7fff640 1 -- 172.21.15.186:0/1262843237 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdda81355e0 msgr2=0x7fdda8133da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:35.288 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.276+0000 7fdda7fff640 1 --2- 172.21.15.186:0/1262843237 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdda81355e0 0x7fdda8133da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:35.288 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.276+0000 7fdda7fff640 10 monclient: _finish_hunting 0 2024-04-26T04:18:35.288 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.276+0000 7fdda7fff640 1 monclient: found mon.noname-b 2024-04-26T04:18:35.288 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.276+0000 7fdda7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:35.288 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.276+0000 7fdda7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.186:3300/0 2024-04-26T04:18:35.288 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.276+0000 7fdda7fff640 1 -- 172.21.15.186:0/1262843237 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdda813c6c0 con 0x7fdda80a1bb0 2024-04-26T04:18:35.288 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.276+0000 7fdda7fff640 1 -- 172.21.15.186:0/1262843237 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdda80a1bb0 msgr2=0x7fdda8133860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:35.289 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.276+0000 7fdda7fff640 1 --2- 172.21.15.186:0/1262843237 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdda80a1bb0 0x7fdda8133860 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fdd9400c4b0 tx=0x7fdd9400c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:35.290 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.280+0000 7fdda57fa640 1 -- 172.21.15.186:0/1262843237 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd94007070 con 0x7fdda80a1bb0 2024-04-26T04:18:35.290 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.280+0000 7fdda57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:35.291 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.280+0000 7fdda57fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T04:18:35.291 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.280+0000 7fdda57fa640 10 monclient: dump: 2024-04-26T04:18:35.291 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-04-26T04:18:35.291 INFO:teuthology.orchestra.run.smithi186.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:35.291 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:35.291 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:35.291 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:35.291 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-04-26T04:18:35.291 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:35.291 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:35.291 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:35.291 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-04-26T04:18:35.292 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.280+0000 7fdda57fa640 10 monclient: _finish_auth 0 2024-04-26T04:18:35.292 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.280+0000 7fdda57fa640 10 monclient: _check_auth_tickets 2024-04-26T04:18:35.292 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.280+0000 7fdda57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:35.292 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.280+0000 7fdda57fa640 1 -- 172.21.15.186:0/1262843237 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd94007210 con 0x7fdda80a1bb0 2024-04-26T04:18:35.292 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.280+0000 7fdda57fa640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:35.292 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.280+0000 7fdda57fa640 1 -- 172.21.15.186:0/1262843237 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd94007600 con 0x7fdda80a1bb0 2024-04-26T04:18:35.292 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.280+0000 7fdda57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:35.292 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.280+0000 7fdda57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T04:18:35.292 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.280+0000 7fdda57fa640 10 monclient: dump: 2024-04-26T04:18:35.292 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-04-26T04:18:35.292 INFO:teuthology.orchestra.run.smithi186.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:35.293 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:35.293 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:35.293 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:35.293 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-04-26T04:18:35.293 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:35.293 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:35.293 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:35.293 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-04-26T04:18:35.293 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.280+0000 7fddaf1cd640 5 monclient: authenticate success, global_id 4111 2024-04-26T04:18:35.294 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.280+0000 7fddaf1cd640 10 monclient: _renew_subs 2024-04-26T04:18:35.294 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.280+0000 7fdda6ffd640 4 set_mon_vals no callback set 2024-04-26T04:18:35.294 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.280+0000 7fddaf1cd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-04-26T04:18:35.294 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.280+0000 7fddaf1cd640 1 -- 172.21.15.186:0/1262843237 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdda8070340 con 0x7fdda80a1bb0 2024-04-26T04:18:35.294 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.280+0000 7fddaf1cd640 10 monclient: _renew_subs 2024-04-26T04:18:35.294 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.280+0000 7fddaf1cd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-04-26T04:18:35.294 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.280+0000 7fddaf1cd640 1 -- 172.21.15.186:0/1262843237 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdda806fc50 con 0x7fdda80a1bb0 2024-04-26T04:18:35.294 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.280+0000 7fdda57fa640 1 -- 172.21.15.186:0/1262843237 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fdd9401b400 con 0x7fdda80a1bb0 2024-04-26T04:18:35.295 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.280+0000 7fdda57fa640 1 -- 172.21.15.186:0/1262843237 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7fdd94012b70 con 0x7fdda80a1bb0 2024-04-26T04:18:35.295 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.280+0000 7fddaf1cd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:35.295 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.280+0000 7fddaf1cd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:35.295 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.280+0000 7fddaf1cd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-04-26T04:18:35.295 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.280+0000 7fddaf1cd640 1 -- 172.21.15.186:0/1262843237 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdda8062e10 con 0x7fdda80a1bb0 2024-04-26T04:18:35.295 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.280+0000 7fdda57fa640 1 -- 172.21.15.186:0/1262843237 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fdd940126d0 con 0x7fdda80a1bb0 2024-04-26T04:18:35.295 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.280+0000 7fdda57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:35.295 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.280+0000 7fdda57fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:18:35.349 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.348+0000 7fddaf1cd640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "913de4d8-31af-4394-a0ee-194cc31f822f", "id": 2}] 2024-04-26T04:18:35.350 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.348+0000 7fddaf1cd640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "913de4d8-31af-4394-a0ee-194cc31f822f", "id": 2}] 2024-04-26T04:18:35.350 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.348+0000 7fddaf1cd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-04-26T04:18:35.350 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.348+0000 7fddaf1cd640 1 -- 172.21.15.186:0/1262843237 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd new", "uuid": "913de4d8-31af-4394-a0ee-194cc31f822f", "id": 2} v 0) -- 0x7fdda806e5d0 con 0x7fdda80a1bb0 2024-04-26T04:18:35.356 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.356+0000 7fdda57fa640 1 -- 172.21.15.186:0/1262843237 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "913de4d8-31af-4394-a0ee-194cc31f822f", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7fdd94012990 con 0x7fdda80a1bb0 2024-04-26T04:18:35.356 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.356+0000 7fdda57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "913de4d8-31af-4394-a0ee-194cc31f822f", "id": 2}] 2024-04-26T04:18:35.356 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.356+0000 7fdda57fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:18:35.357 INFO:teuthology.orchestra.run.smithi186.stdout:2 2024-04-26T04:18:35.358 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.356+0000 7fddaf1cd640 10 monclient: shutdown 2024-04-26T04:18:35.358 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.356+0000 7fddaf1cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:35.358 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.356+0000 7fddaf1cd640 1 -- 172.21.15.186:0/1262843237 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdda80a1bb0 msgr2=0x7fdda8133860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:35.358 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.356+0000 7fddaf1cd640 1 --2- 172.21.15.186:0/1262843237 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdda80a1bb0 0x7fdda8133860 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fdd9400c4b0 tx=0x7fdd9400c970 comp rx=0 tx=0).stop 2024-04-26T04:18:35.359 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.356+0000 7fddaf1cd640 1 -- 172.21.15.186:0/1262843237 shutdown_connections 2024-04-26T04:18:35.359 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.356+0000 7fddaf1cd640 1 --2- 172.21.15.186:0/1262843237 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdda81355e0 0x7fdda8133da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:35.359 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.356+0000 7fddaf1cd640 1 --2- 172.21.15.186:0/1262843237 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdda80a1bb0 0x7fdda8133860 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:35.359 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.356+0000 7fddaf1cd640 1 --2- 172.21.15.186:0/1262843237 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fdda806bde0 0x7fdda81cf390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:35.359 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.356+0000 7fddaf1cd640 1 -- 172.21.15.186:0/1262843237 >> 172.21.15.186:0/1262843237 conn(0x7fdda80a6090 msgr2=0x7fdda809c260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:35.359 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.356+0000 7fddaf1cd640 1 -- 172.21.15.186:0/1262843237 shutdown_connections 2024-04-26T04:18:35.359 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.356+0000 7fddaf1cd640 1 -- 172.21.15.186:0/1262843237 wait complete. 2024-04-26T04:18:35.372 DEBUG:teuthology.orchestra.run.smithi186:> sudo ceph --cluster ceph osd new c492772c-ccf5-4b85-9056-6a5ee9686c0a 3 2024-04-26T04:18:35.483 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.480+0000 7f67fe8eb640 10 monclient: get_monmap_and_config 2024-04-26T04:18:35.483 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.480+0000 7f67fe8eb640 10 monclient: build_initial_monmap 2024-04-26T04:18:35.483 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.480+0000 7f67fe8eb640 10 monclient: monmap: 2024-04-26T04:18:35.483 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 0 2024-04-26T04:18:35.483 INFO:teuthology.orchestra.run.smithi186.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:35.483 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-04-26T04:18:35.484460+0000 2024-04-26T04:18:35.483 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-04-26T04:18:35.484460+0000 2024-04-26T04:18:35.484 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:35.484 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-04-26T04:18:35.484 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:18:35.484 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:18:35.484 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:18:35.484 INFO:teuthology.orchestra.run.smithi186.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:18:35.484 INFO:teuthology.orchestra.run.smithi186.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:18:35.484 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-04-26T04:18:35.487 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.484+0000 7f67fe8eb640 1 Processor -- start 2024-04-26T04:18:35.487 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.484+0000 7f67fe8eb640 1 -- start start 2024-04-26T04:18:35.487 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.484+0000 7f67fe8eb640 10 monclient: init 2024-04-26T04:18:35.487 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.484+0000 7f67fe8eb640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:35.487 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.484+0000 7f67fe8eb640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-04-26T04:18:35.487 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.484+0000 7f67fe8eb640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f67f80a1c60 0x7f67f809bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:35.487 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.484+0000 7f67fe8eb640 10 monclient(hunting): picked mon.noname-c con 0x7f67f80a1c60 addr v2:172.21.15.186:3300/0 2024-04-26T04:18:35.487 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.484+0000 7f67fe8eb640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f67f80a2040 0x7f67f80a2470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:35.487 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.484+0000 7f67fe8eb640 10 monclient(hunting): picked mon.noname-a con 0x7f67f80a2040 addr v2:172.21.15.63:3300/0 2024-04-26T04:18:35.488 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.484+0000 7f67fe8eb640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f67f8133a10 0x7f67f8133e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:35.488 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.484+0000 7f67fe8eb640 10 monclient(hunting): picked mon.noname-b con 0x7f67f8133a10 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:35.488 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.484+0000 7f67fe8eb640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:35.488 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.484+0000 7f67fe8eb640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7f67f806d360 con 0x7f67f80a2040 2024-04-26T04:18:35.488 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.484+0000 7f67fe8eb640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:35.488 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.484+0000 7f67fe8eb640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f67f809b310 con 0x7f67f80a1c60 2024-04-26T04:18:35.488 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.484+0000 7f67fe8eb640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:35.489 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.484+0000 7f67fe8eb640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f67f809b5f0 con 0x7f67f8133a10 2024-04-26T04:18:35.489 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.484+0000 7f67fe8eb640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:35.489 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.484+0000 7f67fe8eb640 10 monclient(hunting): authenticate will time out at 945.261173s 2024-04-26T04:18:35.489 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.484+0000 7f67f7fff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f67f80a1c60 0x7f67f809bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:35.489 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.484+0000 7f67f7fff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f67f80a1c60 0x7f67f809bfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:40978/0 (socket says 172.21.15.186:40978) 2024-04-26T04:18:35.489 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.484+0000 7f67f7fff640 1 -- 172.21.15.186:0/1374763938 learned_addr learned my addr 172.21.15.186:0/1374763938 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-04-26T04:18:35.489 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.484+0000 7f67f7fff640 10 monclient(hunting): get_auth_request con 0x7f67f80a1c60 auth_method 0 2024-04-26T04:18:35.489 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.484+0000 7f67f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:35.489 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.484+0000 7f67f7fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:35.489 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.484+0000 7f67f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:35.489 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.484+0000 7f67f7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:35.489 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.484+0000 7f67f7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:35.489 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.484+0000 7f67f7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:35.489 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.484+0000 7f67fce88640 1 --2- 172.21.15.186:0/1374763938 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f67f8133a10 0x7f67f8133e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:35.490 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.484+0000 7f67f77fe640 1 --2- 172.21.15.186:0/1374763938 >> v2:172.21.15.63:3300/0 conn(0x7f67f80a2040 0x7f67f80a2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:35.490 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.484+0000 7f67f7fff640 10 monclient(hunting): handle_auth_done global_id 4114 payload 274 2024-04-26T04:18:35.490 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.484+0000 7f67f7fff640 1 -- 172.21.15.186:0/1374763938 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f67f8133a10 msgr2=0x7f67f8133e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:35.490 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.484+0000 7f67f7fff640 1 --2- 172.21.15.186:0/1374763938 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f67f8133a10 0x7f67f8133e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:35.490 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.484+0000 7f67f7fff640 1 -- 172.21.15.186:0/1374763938 >> v2:172.21.15.63:3300/0 conn(0x7f67f80a2040 msgr2=0x7f67f80a2470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:35.490 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.484+0000 7f67f7fff640 1 --2- 172.21.15.186:0/1374763938 >> v2:172.21.15.63:3300/0 conn(0x7f67f80a2040 0x7f67f80a2470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:35.490 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.484+0000 7f67f7fff640 10 monclient: _finish_hunting 0 2024-04-26T04:18:35.490 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.484+0000 7f67f7fff640 1 monclient: found mon.noname-c 2024-04-26T04:18:35.490 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.484+0000 7f67f7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:35.490 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.484+0000 7f67f7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.186:3300/0 2024-04-26T04:18:35.490 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.484+0000 7f67f7fff640 1 -- 172.21.15.186:0/1374763938 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67f80a29b0 con 0x7f67f80a1c60 2024-04-26T04:18:35.490 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.484+0000 7f67f77fe640 10 monclient: get_auth_request con 0x7f67f80a2040 auth_method 0 2024-04-26T04:18:35.491 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.484+0000 7f67fce88640 10 monclient: get_auth_request con 0x7f67f8133a10 auth_method 0 2024-04-26T04:18:35.491 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.484+0000 7f67f77fe640 1 --2- 172.21.15.186:0/1374763938 >> v2:172.21.15.63:3300/0 conn(0x7f67f80a2040 0x7f67f80a2470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T04:18:35.491 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.484+0000 7f67fce88640 1 --2- 172.21.15.186:0/1374763938 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f67f8133a10 0x7f67f8133e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T04:18:35.496 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.488+0000 7f67f7fff640 1 -- 172.21.15.186:0/1374763938 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f67f80a1c60 msgr2=0x7f67f809bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:35.496 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.488+0000 7f67f7fff640 1 --2- 172.21.15.186:0/1374763938 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f67f80a1c60 0x7f67f809bfd0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f67f806e8b0 tx=0x7f67e801c3e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a1f57c5def9c7ed8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:35.496 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.488+0000 7f67f6ffd640 1 -- 172.21.15.186:0/1374763938 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67e80b0070 con 0x7f67f80a1c60 2024-04-26T04:18:35.496 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.488+0000 7f67f6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:35.496 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.488+0000 7f67f6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:18:35.496 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.488+0000 7f67f6ffd640 10 monclient: dump: 2024-04-26T04:18:35.496 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-04-26T04:18:35.496 INFO:teuthology.orchestra.run.smithi186.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:35.496 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:35.496 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:35.497 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:35.497 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-04-26T04:18:35.497 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:35.497 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:35.497 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:35.497 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-04-26T04:18:35.497 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.488+0000 7f67f6ffd640 10 monclient: _finish_auth 0 2024-04-26T04:18:35.497 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.488+0000 7f67f6ffd640 10 monclient: _check_auth_tickets 2024-04-26T04:18:35.497 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.488+0000 7f67f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:35.497 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.488+0000 7f67f6ffd640 1 -- 172.21.15.186:0/1374763938 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67e80ab600 con 0x7f67f80a1c60 2024-04-26T04:18:35.497 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.488+0000 7f67f6ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:35.497 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.488+0000 7f67f6ffd640 1 -- 172.21.15.186:0/1374763938 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67e80ab9d0 con 0x7f67f80a1c60 2024-04-26T04:18:35.497 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.488+0000 7f67fe8eb640 5 monclient: authenticate success, global_id 4114 2024-04-26T04:18:35.497 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.488+0000 7f67fe8eb640 10 monclient: get_monmap_and_config success 2024-04-26T04:18:35.497 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.488+0000 7f67fe8eb640 4 set_mon_vals no callback set 2024-04-26T04:18:35.498 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.488+0000 7f67f6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:35.498 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.488+0000 7f67f6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T04:18:35.498 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.488+0000 7f67f6ffd640 10 monclient: dump: 2024-04-26T04:18:35.498 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-04-26T04:18:35.498 INFO:teuthology.orchestra.run.smithi186.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:35.498 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:35.498 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:35.498 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:35.498 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-04-26T04:18:35.498 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:35.498 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:35.498 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:35.498 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-04-26T04:18:35.498 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.488+0000 7f67fe8eb640 10 monclient: shutdown 2024-04-26T04:18:35.499 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.488+0000 7f67fe8eb640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:35.499 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.488+0000 7f67fe8eb640 1 -- 172.21.15.186:0/1374763938 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f67f80a1c60 msgr2=0x7f67f809bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:35.499 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.488+0000 7f67fe8eb640 1 --2- 172.21.15.186:0/1374763938 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f67f80a1c60 0x7f67f809bfd0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f67f806e8b0 tx=0x7f67e801c3e0 comp rx=0 tx=0).stop 2024-04-26T04:18:35.499 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.488+0000 7f67fe8eb640 1 -- 172.21.15.186:0/1374763938 shutdown_connections 2024-04-26T04:18:35.499 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.488+0000 7f67fe8eb640 1 --2- 172.21.15.186:0/1374763938 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f67f8133a10 0x7f67f8133e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:35.499 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.488+0000 7f67fe8eb640 1 --2- 172.21.15.186:0/1374763938 >> v2:172.21.15.63:3300/0 conn(0x7f67f80a2040 0x7f67f80a2470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:35.499 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.488+0000 7f67fe8eb640 1 --2- 172.21.15.186:0/1374763938 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f67f80a1c60 0x7f67f809bfd0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:35.499 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.488+0000 7f67fe8eb640 1 -- 172.21.15.186:0/1374763938 >> 172.21.15.186:0/1374763938 conn(0x7f67f80a2f20 msgr2=0x7f67f80a3330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:35.499 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.488+0000 7f67fe8eb640 1 -- 172.21.15.186:0/1374763938 shutdown_connections 2024-04-26T04:18:35.499 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.488+0000 7f67fe8eb640 1 -- 172.21.15.186:0/1374763938 wait complete. 2024-04-26T04:18:35.499 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.488+0000 7f67fe8eb640 10 monclient: build_initial_monmap 2024-04-26T04:18:35.499 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.488+0000 7f67fe8eb640 10 monclient: monmap: 2024-04-26T04:18:35.499 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 0 2024-04-26T04:18:35.500 INFO:teuthology.orchestra.run.smithi186.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:18:35.500 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 0.000000 2024-04-26T04:18:35.500 INFO:teuthology.orchestra.run.smithi186.stderr:created 0.000000 2024-04-26T04:18:35.500 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:35.500 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-04-26T04:18:35.500 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:18:35.500 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:18:35.500 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:18:35.500 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-04-26T04:18:35.500 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.488+0000 7f67fe8eb640 1 Processor -- start 2024-04-26T04:18:35.500 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.488+0000 7f67fe8eb640 1 -- start start 2024-04-26T04:18:35.500 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.488+0000 7f67fe8eb640 10 monclient: init 2024-04-26T04:18:35.500 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.492+0000 7f67fe8eb640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:35.500 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.492+0000 7f67fe8eb640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T04:18:35.500 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.492+0000 7f67fe8eb640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f67f80a1c60 0x7f67f81cefc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:35.501 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.492+0000 7f67fe8eb640 10 monclient(hunting): picked mon.noname-c con 0x7f67f80a1c60 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:35.501 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.492+0000 7f67fe8eb640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f67f80a2040 0x7f67f8132d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:35.501 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.492+0000 7f67fe8eb640 10 monclient(hunting): picked mon.noname-b con 0x7f67f80a2040 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:18:35.501 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.492+0000 7f67fe8eb640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f67f8133a10 0x7f67f8133290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:35.501 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.492+0000 7f67fe8eb640 10 monclient(hunting): picked mon.noname-a con 0x7f67f8133a10 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:18:35.501 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.492+0000 7f67fe8eb640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:35.501 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.492+0000 7f67fe8eb640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f67f806d860 con 0x7f67f8133a10 2024-04-26T04:18:35.501 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.492+0000 7f67fe8eb640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:35.501 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.492+0000 7f67fe8eb640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f67f806d6e0 con 0x7f67f80a2040 2024-04-26T04:18:35.501 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.492+0000 7f67fe8eb640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:35.501 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.492+0000 7f67fe8eb640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f67f8140a90 con 0x7f67f80a1c60 2024-04-26T04:18:35.501 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.492+0000 7f67fe8eb640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:35.501 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.492+0000 7f67fe8eb640 10 monclient(hunting): authenticate will time out at 945.268109s 2024-04-26T04:18:35.501 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.492+0000 7f67f77fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f67f80a2040 0x7f67f8132d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:35.501 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.492+0000 7f67f77fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f67f80a2040 0x7f67f8132d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:40982/0 (socket says 172.21.15.186:40982) 2024-04-26T04:18:35.502 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.492+0000 7f67f77fe640 1 -- 172.21.15.186:0/3776456222 learned_addr learned my addr 172.21.15.186:0/3776456222 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-04-26T04:18:35.502 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.492+0000 7f67f77fe640 10 monclient(hunting): get_auth_request con 0x7f67f80a2040 auth_method 0 2024-04-26T04:18:35.502 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.492+0000 7f67f77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:35.502 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.492+0000 7f67f77fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:35.502 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.492+0000 7f67f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:35.502 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.492+0000 7f67f77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:35.502 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.492+0000 7f67f77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:35.502 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.492+0000 7f67f77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:35.502 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.492+0000 7f67f7fff640 1 --2- 172.21.15.186:0/3776456222 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f67f80a1c60 0x7f67f81cefc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:35.503 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.492+0000 7f67f77fe640 10 monclient(hunting): handle_auth_done global_id 4117 payload 824 2024-04-26T04:18:35.503 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.492+0000 7f67fce88640 1 --2- 172.21.15.186:0/3776456222 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f67f8133a10 0x7f67f8133290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:35.503 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.492+0000 7f67f77fe640 1 -- 172.21.15.186:0/3776456222 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f67f80a1c60 msgr2=0x7f67f81cefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:35.503 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.492+0000 7f67f77fe640 1 --2- 172.21.15.186:0/3776456222 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f67f80a1c60 0x7f67f81cefc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:35.503 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.492+0000 7f67f77fe640 1 -- 172.21.15.186:0/3776456222 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f67f8133a10 msgr2=0x7f67f8133290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:35.503 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.492+0000 7f67f77fe640 1 --2- 172.21.15.186:0/3776456222 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f67f8133a10 0x7f67f8133290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:35.503 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.492+0000 7f67f77fe640 10 monclient: _finish_hunting 0 2024-04-26T04:18:35.503 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.492+0000 7f67f77fe640 1 monclient: found mon.noname-b 2024-04-26T04:18:35.503 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.492+0000 7f67f77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:35.503 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.492+0000 7f67f77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.186:3300/0 2024-04-26T04:18:35.503 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.492+0000 7f67f77fe640 1 -- 172.21.15.186:0/3776456222 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67f811e710 con 0x7f67f80a2040 2024-04-26T04:18:35.503 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.492+0000 7f67f7fff640 10 monclient: get_auth_request con 0x7f67f80a1c60 auth_method 0 2024-04-26T04:18:35.503 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.492+0000 7f67f7fff640 1 --2- 172.21.15.186:0/3776456222 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f67f80a1c60 0x7f67f81cefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T04:18:35.503 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.492+0000 7f67f77fe640 1 -- 172.21.15.186:0/3776456222 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f67f80a2040 msgr2=0x7f67f8132d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:35.504 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.492+0000 7f67f77fe640 1 --2- 172.21.15.186:0/3776456222 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f67f80a2040 0x7f67f8132d50 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f67ec00c7c0 tx=0x7f67ec00cc80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:35.504 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.492+0000 7f67f4ff9640 1 -- 172.21.15.186:0/3776456222 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67ec007c40 con 0x7f67f80a2040 2024-04-26T04:18:35.504 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.492+0000 7f67f4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:35.504 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.492+0000 7f67f4ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T04:18:35.504 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.492+0000 7f67f4ff9640 10 monclient: dump: 2024-04-26T04:18:35.504 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-04-26T04:18:35.504 INFO:teuthology.orchestra.run.smithi186.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:35.504 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:35.504 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:35.504 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:35.504 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-04-26T04:18:35.504 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:35.504 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:35.504 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:35.504 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-04-26T04:18:35.505 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.492+0000 7f67f4ff9640 10 monclient: _finish_auth 0 2024-04-26T04:18:35.505 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.492+0000 7f67f4ff9640 10 monclient: _check_auth_tickets 2024-04-26T04:18:35.505 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.492+0000 7f67f4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:35.505 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.492+0000 7f67f4ff9640 1 -- 172.21.15.186:0/3776456222 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67ec007de0 con 0x7f67f80a2040 2024-04-26T04:18:35.505 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.492+0000 7f67fe8eb640 5 monclient: authenticate success, global_id 4117 2024-04-26T04:18:35.505 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.492+0000 7f67fe8eb640 10 monclient: _renew_subs 2024-04-26T04:18:35.505 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.492+0000 7f67fe8eb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-04-26T04:18:35.505 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.492+0000 7f67fe8eb640 1 -- 172.21.15.186:0/3776456222 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67f8137560 con 0x7f67f80a2040 2024-04-26T04:18:35.505 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.492+0000 7f67f4ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:35.505 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.492+0000 7f67f4ff9640 1 -- 172.21.15.186:0/3776456222 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67ec011630 con 0x7f67f80a2040 2024-04-26T04:18:35.505 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.492+0000 7f67f4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:35.505 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.492+0000 7f67f4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T04:18:35.505 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.492+0000 7f67f4ff9640 10 monclient: dump: 2024-04-26T04:18:35.505 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-04-26T04:18:35.506 INFO:teuthology.orchestra.run.smithi186.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:35.506 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:35.506 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:35.506 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:35.506 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-04-26T04:18:35.506 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:35.506 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:35.506 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:35.506 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-04-26T04:18:35.506 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.492+0000 7f67fe8eb640 10 monclient: _renew_subs 2024-04-26T04:18:35.506 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.492+0000 7f67f67fc640 4 set_mon_vals no callback set 2024-04-26T04:18:35.506 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.492+0000 7f67fe8eb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-04-26T04:18:35.506 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.492+0000 7f67fe8eb640 1 -- 172.21.15.186:0/3776456222 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67f811de70 con 0x7f67f80a2040 2024-04-26T04:18:35.506 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.492+0000 7f67f4ff9640 1 -- 172.21.15.186:0/3776456222 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f67ec011b60 con 0x7f67f80a2040 2024-04-26T04:18:35.506 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.492+0000 7f67f4ff9640 1 -- 172.21.15.186:0/3776456222 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7f67ec0217e0 con 0x7f67f80a2040 2024-04-26T04:18:35.507 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.492+0000 7f67fe8eb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:35.507 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.492+0000 7f67fe8eb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:35.507 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.492+0000 7f67fe8eb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-04-26T04:18:35.507 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.492+0000 7f67fe8eb640 1 -- 172.21.15.186:0/3776456222 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67f8062e10 con 0x7f67f80a2040 2024-04-26T04:18:35.507 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.496+0000 7f67f4ff9640 1 -- 172.21.15.186:0/3776456222 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f67ec015020 con 0x7f67f80a2040 2024-04-26T04:18:35.507 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.496+0000 7f67f4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:35.507 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.496+0000 7f67f4ff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:18:35.561 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.560+0000 7f67fe8eb640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "c492772c-ccf5-4b85-9056-6a5ee9686c0a", "id": 3}] 2024-04-26T04:18:35.561 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.560+0000 7f67fe8eb640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "c492772c-ccf5-4b85-9056-6a5ee9686c0a", "id": 3}] 2024-04-26T04:18:35.561 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.560+0000 7f67fe8eb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-04-26T04:18:35.561 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.560+0000 7f67fe8eb640 1 -- 172.21.15.186:0/3776456222 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd new", "uuid": "c492772c-ccf5-4b85-9056-6a5ee9686c0a", "id": 3} v 0) -- 0x7f67f8134d30 con 0x7f67f80a2040 2024-04-26T04:18:35.567 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.564+0000 7f67f4ff9640 1 -- 172.21.15.186:0/3776456222 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c492772c-ccf5-4b85-9056-6a5ee9686c0a", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7f67ec021350 con 0x7f67f80a2040 2024-04-26T04:18:35.567 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.564+0000 7f67f4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "c492772c-ccf5-4b85-9056-6a5ee9686c0a", "id": 3}] 2024-04-26T04:18:35.567 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.564+0000 7f67f4ff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:18:35.567 INFO:teuthology.orchestra.run.smithi186.stdout:3 2024-04-26T04:18:35.568 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.568+0000 7f67fe8eb640 10 monclient: shutdown 2024-04-26T04:18:35.569 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.568+0000 7f67fe8eb640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:35.569 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.568+0000 7f67fe8eb640 1 -- 172.21.15.186:0/3776456222 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f67f80a2040 msgr2=0x7f67f8132d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:35.569 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.568+0000 7f67fe8eb640 1 --2- 172.21.15.186:0/3776456222 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f67f80a2040 0x7f67f8132d50 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f67ec00c7c0 tx=0x7f67ec00cc80 comp rx=0 tx=0).stop 2024-04-26T04:18:35.569 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.568+0000 7f67fe8eb640 1 -- 172.21.15.186:0/3776456222 shutdown_connections 2024-04-26T04:18:35.569 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.568+0000 7f67fe8eb640 1 --2- 172.21.15.186:0/3776456222 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f67f8133a10 0x7f67f8133290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:35.569 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.568+0000 7f67fe8eb640 1 --2- 172.21.15.186:0/3776456222 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f67f80a2040 0x7f67f8132d50 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:35.569 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.568+0000 7f67fe8eb640 1 --2- 172.21.15.186:0/3776456222 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f67f80a1c60 0x7f67f81cefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:35.569 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.568+0000 7f67fe8eb640 1 -- 172.21.15.186:0/3776456222 >> 172.21.15.186:0/3776456222 conn(0x7f67f80a2f20 msgr2=0x7f67f8135cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:35.569 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.568+0000 7f67fe8eb640 1 -- 172.21.15.186:0/3776456222 shutdown_connections 2024-04-26T04:18:35.569 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.568+0000 7f67fe8eb640 1 -- 172.21.15.186:0/3776456222 wait complete. 2024-04-26T04:18:35.584 DEBUG:teuthology.orchestra.run.smithi186:> sudo ceph --cluster ceph osd new 24f067f0-554a-4dd2-a26c-7bda3d0978f7 4 2024-04-26T04:18:35.609 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-26T04:18:35.608+0000 7f2753209180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-26T04:18:35.697 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.696+0000 7f5312f21640 10 monclient: get_monmap_and_config 2024-04-26T04:18:35.697 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.696+0000 7f5312f21640 10 monclient: build_initial_monmap 2024-04-26T04:18:35.697 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.696+0000 7f5312f21640 10 monclient: monmap: 2024-04-26T04:18:35.697 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 0 2024-04-26T04:18:35.697 INFO:teuthology.orchestra.run.smithi186.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:35.697 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-04-26T04:18:35.698336+0000 2024-04-26T04:18:35.697 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-04-26T04:18:35.698336+0000 2024-04-26T04:18:35.697 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:35.697 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-04-26T04:18:35.698 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:18:35.698 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:18:35.698 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:18:35.698 INFO:teuthology.orchestra.run.smithi186.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:18:35.698 INFO:teuthology.orchestra.run.smithi186.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:18:35.698 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-04-26T04:18:35.700 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.696+0000 7f5312f21640 1 Processor -- start 2024-04-26T04:18:35.700 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.696+0000 7f5312f21640 1 -- start start 2024-04-26T04:18:35.700 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.696+0000 7f5312f21640 10 monclient: init 2024-04-26T04:18:35.700 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.696+0000 7f5312f21640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:35.700 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.696+0000 7f5312f21640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-04-26T04:18:35.700 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.696+0000 7f5312f21640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f530c0a2ae0 0x7f530c0a1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:35.700 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.696+0000 7f5312f21640 10 monclient(hunting): picked mon.noname-a con 0x7f530c0a2ae0 addr v2:172.21.15.63:3300/0 2024-04-26T04:18:35.700 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.696+0000 7f5312f21640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f530c0a2ec0 addr v1:172.21.15.63:6789/0 2024-04-26T04:18:35.701 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.696+0000 7f5312f21640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f530c135310 0x7f530c13cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:35.701 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.696+0000 7f5312f21640 10 monclient(hunting): picked mon.noname-c con 0x7f530c135310 addr v2:172.21.15.186:3300/0 2024-04-26T04:18:35.701 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.696+0000 7f5312f21640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f530c0a1550 con 0x7f530c0a2ec0 2024-04-26T04:18:35.701 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.696+0000 7f5312f21640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:35.701 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.696+0000 7f5312f21640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7f530c09fa70 con 0x7f530c0a2ae0 2024-04-26T04:18:35.701 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.696+0000 7f5312f21640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:35.701 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.696+0000 7f5312f21640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f530c09fd50 con 0x7f530c135310 2024-04-26T04:18:35.701 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.696+0000 7f5312f21640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:35.701 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.696+0000 7f5312f21640 10 monclient(hunting): authenticate will time out at 945.474887s 2024-04-26T04:18:35.701 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.696+0000 7f53114be640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f530c135310 0x7f530c13cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:35.701 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.696+0000 7f53114be640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f530c135310 0x7f530c13cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:40996/0 (socket says 172.21.15.186:40996) 2024-04-26T04:18:35.701 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.696+0000 7f53114be640 1 -- 172.21.15.186:0/4048319954 learned_addr learned my addr 172.21.15.186:0/4048319954 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-04-26T04:18:35.701 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.696+0000 7f53114be640 10 monclient(hunting): get_auth_request con 0x7f530c135310 auth_method 0 2024-04-26T04:18:35.701 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.696+0000 7f53114be640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:35.701 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.696+0000 7f53114be640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:35.702 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.696+0000 7f53114be640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:35.702 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.696+0000 7f53114be640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:35.702 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.696+0000 7f53114be640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:35.702 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.696+0000 7f53114be640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:35.702 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.696+0000 7f5310cbd640 1 --2- 172.21.15.186:0/4048319954 >> v2:172.21.15.63:3300/0 conn(0x7f530c0a2ae0 0x7f530c0a1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:35.702 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.696+0000 7f5310cbd640 10 monclient(hunting): get_auth_request con 0x7f530c0a2ae0 auth_method 0 2024-04-26T04:18:35.702 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.696+0000 7f5310cbd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:35.702 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.696+0000 7f5310cbd640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:35.702 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.696+0000 7f5310cbd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:35.702 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.696+0000 7f53114be640 10 monclient(hunting): handle_auth_done global_id 4120 payload 274 2024-04-26T04:18:35.702 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.696+0000 7f53114be640 1 -- 172.21.15.186:0/4048319954 >> v2:172.21.15.63:3300/0 conn(0x7f530c0a2ae0 msgr2=0x7f530c0a1cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:35.702 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.696+0000 7f53114be640 1 --2- 172.21.15.186:0/4048319954 >> v2:172.21.15.63:3300/0 conn(0x7f530c0a2ae0 0x7f530c0a1cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:35.702 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.696+0000 7f53114be640 1 -- 172.21.15.186:0/4048319954 >> v1:172.21.15.63:6789/0 conn(0x7f530c0a2ec0 legacy=0x7f530c134c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:35.703 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.696+0000 7f53114be640 10 monclient: _finish_hunting 0 2024-04-26T04:18:35.703 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.696+0000 7f53114be640 1 monclient: found mon.noname-c 2024-04-26T04:18:35.703 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.696+0000 7f53114be640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:35.703 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.696+0000 7f53114be640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.186:3300/0 2024-04-26T04:18:35.703 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.696+0000 7f53114be640 1 -- 172.21.15.186:0/4048319954 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f530c0a0030 con 0x7f530c135310 2024-04-26T04:18:35.707 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.700+0000 7f53114be640 1 -- 172.21.15.186:0/4048319954 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f530c135310 msgr2=0x7f530c13cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:35.708 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.700+0000 7f53114be640 1 --2- 172.21.15.186:0/4048319954 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f530c135310 0x7f530c13cf40 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f530c06db80 tx=0x7f530401c870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a82d0c29aa47e37 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:35.708 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.700+0000 7f530b7fe640 1 -- 172.21.15.186:0/4048319954 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53040a5530 con 0x7f530c135310 2024-04-26T04:18:35.708 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.700+0000 7f530b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:35.708 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.700+0000 7f530b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:18:35.708 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.700+0000 7f530b7fe640 10 monclient: dump: 2024-04-26T04:18:35.708 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-04-26T04:18:35.708 INFO:teuthology.orchestra.run.smithi186.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:35.708 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:35.708 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:35.708 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:35.708 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-04-26T04:18:35.708 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:35.708 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:35.708 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:35.709 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-04-26T04:18:35.709 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.700+0000 7f530b7fe640 10 monclient: _finish_auth 0 2024-04-26T04:18:35.709 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.700+0000 7f530b7fe640 10 monclient: _check_auth_tickets 2024-04-26T04:18:35.709 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.700+0000 7f530b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:35.709 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.700+0000 7f530b7fe640 1 -- 172.21.15.186:0/4048319954 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53040a56d0 con 0x7f530c135310 2024-04-26T04:18:35.709 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.700+0000 7f530b7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:35.709 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.700+0000 7f530b7fe640 1 -- 172.21.15.186:0/4048319954 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53040a5aa0 con 0x7f530c135310 2024-04-26T04:18:35.709 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.700+0000 7f5312f21640 5 monclient: authenticate success, global_id 4120 2024-04-26T04:18:35.709 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.700+0000 7f5312f21640 10 monclient: get_monmap_and_config success 2024-04-26T04:18:35.709 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.700+0000 7f5312f21640 4 set_mon_vals no callback set 2024-04-26T04:18:35.709 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.700+0000 7f530b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:35.709 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.700+0000 7f530b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T04:18:35.709 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.700+0000 7f530b7fe640 10 monclient: dump: 2024-04-26T04:18:35.709 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-04-26T04:18:35.710 INFO:teuthology.orchestra.run.smithi186.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:35.710 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:35.710 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:35.710 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:35.710 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-04-26T04:18:35.710 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:35.710 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:35.710 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:35.710 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-04-26T04:18:35.710 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.704+0000 7f5312f21640 10 monclient: shutdown 2024-04-26T04:18:35.710 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.704+0000 7f5312f21640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:35.710 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.704+0000 7f5312f21640 1 -- 172.21.15.186:0/4048319954 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f530c135310 msgr2=0x7f530c13cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:35.710 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.704+0000 7f5312f21640 1 --2- 172.21.15.186:0/4048319954 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f530c135310 0x7f530c13cf40 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f530c06db80 tx=0x7f530401c870 comp rx=0 tx=0).stop 2024-04-26T04:18:35.710 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.704+0000 7f5312f21640 1 -- 172.21.15.186:0/4048319954 shutdown_connections 2024-04-26T04:18:35.711 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.704+0000 7f5312f21640 1 --2- 172.21.15.186:0/4048319954 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f530c135310 0x7f530c13cf40 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:35.711 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.704+0000 7f5312f21640 1 --2- 172.21.15.186:0/4048319954 >> v2:172.21.15.63:3300/0 conn(0x7f530c0a2ae0 0x7f530c0a1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:35.711 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.704+0000 7f5312f21640 1 -- 172.21.15.186:0/4048319954 >> 172.21.15.186:0/4048319954 conn(0x7f530c0a73d0 msgr2=0x7f530c0a77b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:35.711 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.704+0000 7f5312f21640 1 -- 172.21.15.186:0/4048319954 shutdown_connections 2024-04-26T04:18:35.711 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.704+0000 7f5312f21640 1 -- 172.21.15.186:0/4048319954 wait complete. 2024-04-26T04:18:35.711 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.704+0000 7f5312f21640 10 monclient: build_initial_monmap 2024-04-26T04:18:35.711 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.704+0000 7f5312f21640 10 monclient: monmap: 2024-04-26T04:18:35.711 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 0 2024-04-26T04:18:35.711 INFO:teuthology.orchestra.run.smithi186.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:18:35.711 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 0.000000 2024-04-26T04:18:35.711 INFO:teuthology.orchestra.run.smithi186.stderr:created 0.000000 2024-04-26T04:18:35.711 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:35.711 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-04-26T04:18:35.711 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:18:35.712 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:18:35.712 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:18:35.712 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-04-26T04:18:35.712 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.704+0000 7f5312f21640 1 Processor -- start 2024-04-26T04:18:35.712 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.704+0000 7f5312f21640 1 -- start start 2024-04-26T04:18:35.712 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.704+0000 7f5312f21640 10 monclient: init 2024-04-26T04:18:35.712 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.704+0000 7f5312f21640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:35.712 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.704+0000 7f5312f21640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T04:18:35.712 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.704+0000 7f5312f21640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f530c0a2ae0 0x7f530c1cd4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:35.712 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.704+0000 7f5312f21640 10 monclient(hunting): picked mon.noname-c con 0x7f530c0a2ae0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:35.712 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.704+0000 7f5312f21640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f530c0a2ec0 0x7f530c13bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:35.712 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.704+0000 7f5312f21640 10 monclient(hunting): picked mon.noname-a con 0x7f530c0a2ec0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:18:35.712 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.704+0000 7f5312f21640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f530c135310 0x7f530c13c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:35.712 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.704+0000 7f5312f21640 10 monclient(hunting): picked mon.noname-b con 0x7f530c135310 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:18:35.712 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.704+0000 7f5312f21640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:35.713 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.704+0000 7f5312f21640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f530c13feb0 con 0x7f530c0a2ec0 2024-04-26T04:18:35.713 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.704+0000 7f5312f21640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:35.713 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.704+0000 7f5312f21640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f530c140030 con 0x7f530c135310 2024-04-26T04:18:35.713 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.704+0000 7f5312f21640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:35.713 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.704+0000 7f5312f21640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f530c1401b0 con 0x7f530c0a2ae0 2024-04-26T04:18:35.713 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.704+0000 7f5312f21640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:35.713 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.704+0000 7f5312f21640 10 monclient(hunting): authenticate will time out at 945.482241s 2024-04-26T04:18:35.713 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.704+0000 7f53114be640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f530c135310 0x7f530c13c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:35.713 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.704+0000 7f53114be640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f530c135310 0x7f530c13c1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:40998/0 (socket says 172.21.15.186:40998) 2024-04-26T04:18:35.713 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.704+0000 7f53114be640 1 -- 172.21.15.186:0/3283210836 learned_addr learned my addr 172.21.15.186:0/3283210836 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-04-26T04:18:35.713 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.704+0000 7f53114be640 10 monclient(hunting): get_auth_request con 0x7f530c135310 auth_method 0 2024-04-26T04:18:35.713 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.704+0000 7f53114be640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:35.713 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.704+0000 7f53114be640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:35.713 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.704+0000 7f53114be640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:35.714 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.704+0000 7f53114be640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:35.714 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.704+0000 7f53114be640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:35.714 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.704+0000 7f53114be640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:35.714 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.704+0000 7f5310cbd640 1 --2- 172.21.15.186:0/3283210836 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f530c0a2ae0 0x7f530c1cd4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:35.714 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.704+0000 7f530bfff640 1 --2- 172.21.15.186:0/3283210836 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f530c0a2ec0 0x7f530c13bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:35.714 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.704+0000 7f5310cbd640 10 monclient(hunting): get_auth_request con 0x7f530c0a2ae0 auth_method 0 2024-04-26T04:18:35.714 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.704+0000 7f5310cbd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:35.714 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.704+0000 7f5310cbd640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:35.714 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.704+0000 7f5310cbd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:35.714 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.704+0000 7f530bfff640 10 monclient(hunting): get_auth_request con 0x7f530c0a2ec0 auth_method 0 2024-04-26T04:18:35.714 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.704+0000 7f530bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:35.714 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.704+0000 7f530bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:35.714 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.704+0000 7f530bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:35.714 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.704+0000 7f53114be640 10 monclient(hunting): handle_auth_done global_id 4123 payload 824 2024-04-26T04:18:35.715 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.704+0000 7f53114be640 1 -- 172.21.15.186:0/3283210836 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f530c0a2ae0 msgr2=0x7f530c1cd4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:35.715 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.704+0000 7f53114be640 1 --2- 172.21.15.186:0/3283210836 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f530c0a2ae0 0x7f530c1cd4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:35.715 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.704+0000 7f53114be640 1 -- 172.21.15.186:0/3283210836 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f530c0a2ec0 msgr2=0x7f530c13bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:35.715 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.704+0000 7f53114be640 1 --2- 172.21.15.186:0/3283210836 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f530c0a2ec0 0x7f530c13bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:35.715 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.704+0000 7f53114be640 10 monclient: _finish_hunting 0 2024-04-26T04:18:35.715 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.704+0000 7f53114be640 1 monclient: found mon.noname-b 2024-04-26T04:18:35.715 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.704+0000 7f53114be640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:35.715 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.704+0000 7f53114be640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.186:3300/0 2024-04-26T04:18:35.715 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.704+0000 7f53114be640 1 -- 172.21.15.186:0/3283210836 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f530c0a1340 con 0x7f530c135310 2024-04-26T04:18:35.717 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.708+0000 7f53114be640 1 -- 172.21.15.186:0/3283210836 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f530c135310 msgr2=0x7f530c13c1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:35.717 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.708+0000 7f53114be640 1 --2- 172.21.15.186:0/3283210836 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f530c135310 0x7f530c13c1f0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f530401af20 tx=0x7f5304019610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:35.717 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.708+0000 7f53097fa640 1 -- 172.21.15.186:0/3283210836 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53040bc020 con 0x7f530c135310 2024-04-26T04:18:35.717 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.708+0000 7f53097fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:35.717 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.708+0000 7f53097fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T04:18:35.717 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.708+0000 7f53097fa640 10 monclient: dump: 2024-04-26T04:18:35.717 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-04-26T04:18:35.717 INFO:teuthology.orchestra.run.smithi186.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:35.717 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:35.717 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:35.718 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:35.718 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-04-26T04:18:35.718 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:35.718 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:35.718 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:35.718 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-04-26T04:18:35.718 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.708+0000 7f53097fa640 10 monclient: _finish_auth 0 2024-04-26T04:18:35.718 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.708+0000 7f53097fa640 10 monclient: _check_auth_tickets 2024-04-26T04:18:35.718 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.708+0000 7f53097fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:35.718 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.708+0000 7f5312f21640 5 monclient: authenticate success, global_id 4123 2024-04-26T04:18:35.718 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.708+0000 7f53097fa640 1 -- 172.21.15.186:0/3283210836 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53040b88a0 con 0x7f530c135310 2024-04-26T04:18:35.718 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.708+0000 7f5312f21640 10 monclient: _renew_subs 2024-04-26T04:18:35.718 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.708+0000 7f5312f21640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-04-26T04:18:35.718 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.708+0000 7f5312f21640 1 -- 172.21.15.186:0/3283210836 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f530c070d50 con 0x7f530c135310 2024-04-26T04:18:35.719 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.708+0000 7f53097fa640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:35.719 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.708+0000 7f53097fa640 1 -- 172.21.15.186:0/3283210836 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53040a5530 con 0x7f530c135310 2024-04-26T04:18:35.719 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.708+0000 7f53097fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:35.719 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.708+0000 7f53097fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T04:18:35.719 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.708+0000 7f53097fa640 10 monclient: dump: 2024-04-26T04:18:35.719 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-04-26T04:18:35.719 INFO:teuthology.orchestra.run.smithi186.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:35.719 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:35.719 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:35.719 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:35.719 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-04-26T04:18:35.719 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:35.720 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:35.720 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:35.720 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-04-26T04:18:35.720 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.708+0000 7f5312f21640 10 monclient: _renew_subs 2024-04-26T04:18:35.720 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.708+0000 7f5312f21640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-04-26T04:18:35.720 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.708+0000 7f5312f21640 1 -- 172.21.15.186:0/3283210836 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f530c137f70 con 0x7f530c135310 2024-04-26T04:18:35.720 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.708+0000 7f530affd640 4 set_mon_vals no callback set 2024-04-26T04:18:35.720 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.708+0000 7f53097fa640 1 -- 172.21.15.186:0/3283210836 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f53040bf020 con 0x7f530c135310 2024-04-26T04:18:35.720 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.708+0000 7f53097fa640 1 -- 172.21.15.186:0/3283210836 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7f53040b8c90 con 0x7f530c135310 2024-04-26T04:18:35.720 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.708+0000 7f5312f21640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:35.720 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.708+0000 7f5312f21640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:35.720 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.708+0000 7f5312f21640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-04-26T04:18:35.720 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.708+0000 7f5312f21640 1 -- 172.21.15.186:0/3283210836 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f530c138f00 con 0x7f530c135310 2024-04-26T04:18:35.721 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.712+0000 7f53097fa640 1 -- 172.21.15.186:0/3283210836 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f53040c6030 con 0x7f530c135310 2024-04-26T04:18:35.721 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.712+0000 7f53097fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:35.721 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.712+0000 7f53097fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:18:35.774 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.772+0000 7f5312f21640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "24f067f0-554a-4dd2-a26c-7bda3d0978f7", "id": 4}] 2024-04-26T04:18:35.774 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.772+0000 7f5312f21640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "24f067f0-554a-4dd2-a26c-7bda3d0978f7", "id": 4}] 2024-04-26T04:18:35.774 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.772+0000 7f5312f21640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-04-26T04:18:35.774 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.772+0000 7f5312f21640 1 -- 172.21.15.186:0/3283210836 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd new", "uuid": "24f067f0-554a-4dd2-a26c-7bda3d0978f7", "id": 4} v 0) -- 0x7f530c062e10 con 0x7f530c135310 2024-04-26T04:18:35.779 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.780+0000 7f53097fa640 1 -- 172.21.15.186:0/3283210836 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "24f067f0-554a-4dd2-a26c-7bda3d0978f7", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7f5304016580 con 0x7f530c135310 2024-04-26T04:18:35.779 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.780+0000 7f53097fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "24f067f0-554a-4dd2-a26c-7bda3d0978f7", "id": 4}] 2024-04-26T04:18:35.780 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.780+0000 7f53097fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:18:35.780 INFO:teuthology.orchestra.run.smithi186.stdout:4 2024-04-26T04:18:35.781 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.780+0000 7f5312f21640 10 monclient: shutdown 2024-04-26T04:18:35.781 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.780+0000 7f5312f21640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:35.781 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.780+0000 7f5312f21640 1 -- 172.21.15.186:0/3283210836 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f530c135310 msgr2=0x7f530c13c1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:35.781 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.780+0000 7f5312f21640 1 --2- 172.21.15.186:0/3283210836 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f530c135310 0x7f530c13c1f0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f530401af20 tx=0x7f5304019610 comp rx=0 tx=0).stop 2024-04-26T04:18:35.781 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.780+0000 7f5312f21640 1 -- 172.21.15.186:0/3283210836 shutdown_connections 2024-04-26T04:18:35.781 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.780+0000 7f5312f21640 1 --2- 172.21.15.186:0/3283210836 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f530c135310 0x7f530c13c1f0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:35.781 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.780+0000 7f5312f21640 1 --2- 172.21.15.186:0/3283210836 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f530c0a2ec0 0x7f530c13bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:35.781 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.780+0000 7f5312f21640 1 --2- 172.21.15.186:0/3283210836 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f530c0a2ae0 0x7f530c1cd4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:35.781 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.780+0000 7f5312f21640 1 -- 172.21.15.186:0/3283210836 >> 172.21.15.186:0/3283210836 conn(0x7f530c0a73d0 msgr2=0x7f530c071d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:35.781 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.780+0000 7f5312f21640 1 -- 172.21.15.186:0/3283210836 shutdown_connections 2024-04-26T04:18:35.781 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.780+0000 7f5312f21640 1 -- 172.21.15.186:0/3283210836 wait complete. 2024-04-26T04:18:35.783 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-26T04:18:35.784+0000 7f2753209180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-26T04:18:35.796 DEBUG:teuthology.orchestra.run.smithi186:> sudo ceph --cluster ceph osd new 70454d09-e50c-4f65-83c5-0f9bb5c0f8f6 5 2024-04-26T04:18:35.949 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.948+0000 7fe97ddf8640 10 monclient: get_monmap_and_config 2024-04-26T04:18:35.949 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.948+0000 7fe97ddf8640 10 monclient: build_initial_monmap 2024-04-26T04:18:35.950 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.948+0000 7fe97ddf8640 10 monclient: monmap: 2024-04-26T04:18:35.950 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 0 2024-04-26T04:18:35.950 INFO:teuthology.orchestra.run.smithi186.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:35.950 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-04-26T04:18:35.950871+0000 2024-04-26T04:18:35.950 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-04-26T04:18:35.950871+0000 2024-04-26T04:18:35.950 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:35.950 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-04-26T04:18:35.950 INFO:teuthology.orchestra.run.smithi186.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:18:35.950 INFO:teuthology.orchestra.run.smithi186.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:18:35.950 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:18:35.950 INFO:teuthology.orchestra.run.smithi186.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:18:35.950 INFO:teuthology.orchestra.run.smithi186.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:18:35.950 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-04-26T04:18:35.953 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.948+0000 7fe97ddf8640 1 Processor -- start 2024-04-26T04:18:35.953 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.948+0000 7fe97ddf8640 1 -- start start 2024-04-26T04:18:35.953 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.948+0000 7fe97ddf8640 10 monclient: init 2024-04-26T04:18:35.954 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.948+0000 7fe97ddf8640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:35.954 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.948+0000 7fe97ddf8640 10 monclient: _add_conns ranks=[3,2,0,1,4] 2024-04-26T04:18:35.954 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.948+0000 7fe97ddf8640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7fe9780723a0 0x7fe978072780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:35.954 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.948+0000 7fe97ddf8640 10 monclient(hunting): picked mon.noname-a con 0x7fe9780723a0 addr v2:172.21.15.63:3300/0 2024-04-26T04:18:35.954 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.948+0000 7fe97ddf8640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fe97813b5a0 0x7fe9780a14a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:35.954 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.948+0000 7fe97ddf8640 10 monclient(hunting): picked mon.noname-b con 0x7fe97813b5a0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:35.954 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.948+0000 7fe97ddf8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe97813b9e0 addr v1:172.21.15.63:6789/0 2024-04-26T04:18:35.954 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.948+0000 7fe97ddf8640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe97806f650 con 0x7fe97813b9e0 2024-04-26T04:18:35.954 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.948+0000 7fe97ddf8640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:35.954 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.948+0000 7fe97ddf8640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7fe97806f8c0 con 0x7fe9780723a0 2024-04-26T04:18:35.954 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.948+0000 7fe97ddf8640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:35.954 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.948+0000 7fe97ddf8640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7fe97806fba0 con 0x7fe97813b5a0 2024-04-26T04:18:35.954 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.948+0000 7fe97ddf8640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:35.955 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.948+0000 7fe97ddf8640 10 monclient(hunting): authenticate will time out at 945.727748s 2024-04-26T04:18:35.955 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.952+0000 7fe9777fe640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7fe9780723a0 0x7fe978072780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:35.955 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.952+0000 7fe976ffd640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fe97813b5a0 0x7fe9780a14a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:35.955 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.952+0000 7fe9777fe640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7fe9780723a0 0x7fe978072780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.186:52108/0 (socket says 172.21.15.186:52108) 2024-04-26T04:18:35.955 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.952+0000 7fe976ffd640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fe97813b5a0 0x7fe9780a14a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.186:56174/0 (socket says 172.21.15.186:56174) 2024-04-26T04:18:35.955 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.952+0000 7fe9777fe640 1 -- 172.21.15.186:0/3144740338 learned_addr learned my addr 172.21.15.186:0/3144740338 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-04-26T04:18:35.955 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.952+0000 7fe9777fe640 10 monclient(hunting): get_auth_request con 0x7fe9780723a0 auth_method 0 2024-04-26T04:18:35.955 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.952+0000 7fe9777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:35.955 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.952+0000 7fe9777fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:35.955 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.952+0000 7fe9777fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:35.955 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.952+0000 7fe976ffd640 10 monclient(hunting): get_auth_request con 0x7fe97813b5a0 auth_method 0 2024-04-26T04:18:35.955 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.952+0000 7fe976ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:35.956 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.952+0000 7fe976ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:35.956 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.952+0000 7fe976ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:35.956 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.952+0000 7fe976ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:35.956 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.952+0000 7fe976ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:35.956 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.952+0000 7fe976ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:35.956 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.952+0000 7fe9777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:35.956 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.952+0000 7fe9777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:35.956 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.952+0000 7fe9777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:35.956 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.952+0000 7fe976ffd640 10 monclient(hunting): handle_auth_done global_id 4124 payload 274 2024-04-26T04:18:35.956 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.952+0000 7fe976ffd640 1 -- 172.21.15.186:0/3144740338 >> v2:172.21.15.63:3300/0 conn(0x7fe9780723a0 msgr2=0x7fe978072780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:35.956 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.952+0000 7fe976ffd640 1 --2- 172.21.15.186:0/3144740338 >> v2:172.21.15.63:3300/0 conn(0x7fe9780723a0 0x7fe978072780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:35.957 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.952+0000 7fe976ffd640 1 -- 172.21.15.186:0/3144740338 >> v1:172.21.15.63:6789/0 conn(0x7fe97813b9e0 legacy=0x7fe9780a19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:35.957 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.952+0000 7fe976ffd640 10 monclient: _finish_hunting 0 2024-04-26T04:18:35.957 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.952+0000 7fe976ffd640 1 monclient: found mon.noname-b 2024-04-26T04:18:35.957 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.952+0000 7fe976ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:35.957 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.952+0000 7fe976ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.63:3301/0 2024-04-26T04:18:35.957 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.952+0000 7fe976ffd640 1 -- 172.21.15.186:0/3144740338 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe97806fe80 con 0x7fe97813b5a0 2024-04-26T04:18:35.959 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.956+0000 7fe976ffd640 1 -- 172.21.15.186:0/3144740338 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fe97813b5a0 msgr2=0x7fe9780a14a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:35.959 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.956+0000 7fe976ffd640 1 --2- 172.21.15.186:0/3144740338 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fe97813b5a0 0x7fe9780a14a0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe9680097c0 tx=0x7fe96801c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=233b453f83922a4b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:35.959 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.956+0000 7fe9767fc640 1 -- 172.21.15.186:0/3144740338 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9680b1070 con 0x7fe97813b5a0 2024-04-26T04:18:35.959 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.956+0000 7fe9767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:35.959 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.956+0000 7fe9767fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T04:18:35.959 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.956+0000 7fe9767fc640 10 monclient: dump: 2024-04-26T04:18:35.959 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-04-26T04:18:35.959 INFO:teuthology.orchestra.run.smithi186.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:35.959 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:35.959 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:35.959 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:35.959 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-04-26T04:18:35.959 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:35.959 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:35.960 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:35.960 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-04-26T04:18:35.960 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.956+0000 7fe9767fc640 10 monclient: _finish_auth 0 2024-04-26T04:18:35.960 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.956+0000 7fe9767fc640 10 monclient: _check_auth_tickets 2024-04-26T04:18:35.960 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.956+0000 7fe9767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:35.960 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.956+0000 7fe9767fc640 1 -- 172.21.15.186:0/3144740338 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9780a2300 con 0x7fe97813b5a0 2024-04-26T04:18:35.961 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.956+0000 7fe9767fc640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:35.961 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.956+0000 7fe9767fc640 1 -- 172.21.15.186:0/3144740338 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9680a4a20 con 0x7fe97813b5a0 2024-04-26T04:18:35.961 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.956+0000 7fe9767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:35.961 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.956+0000 7fe9767fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:18:35.961 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.956+0000 7fe9767fc640 10 monclient: dump: 2024-04-26T04:18:35.961 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-04-26T04:18:35.961 INFO:teuthology.orchestra.run.smithi186.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:35.961 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:35.961 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:35.961 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:35.962 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-04-26T04:18:35.962 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:35.962 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:35.962 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:35.962 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-04-26T04:18:35.962 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.956+0000 7fe97ddf8640 5 monclient: authenticate success, global_id 4124 2024-04-26T04:18:35.962 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.956+0000 7fe97ddf8640 10 monclient: get_monmap_and_config success 2024-04-26T04:18:35.962 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.956+0000 7fe97ddf8640 4 set_mon_vals no callback set 2024-04-26T04:18:35.962 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.956+0000 7fe97ddf8640 10 monclient: shutdown 2024-04-26T04:18:35.962 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.956+0000 7fe97ddf8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:35.962 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.956+0000 7fe97ddf8640 1 -- 172.21.15.186:0/3144740338 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fe97813b5a0 msgr2=0x7fe9780a14a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:35.962 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.956+0000 7fe97ddf8640 1 --2- 172.21.15.186:0/3144740338 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fe97813b5a0 0x7fe9780a14a0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe9680097c0 tx=0x7fe96801c000 comp rx=0 tx=0).stop 2024-04-26T04:18:35.963 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.956+0000 7fe97ddf8640 1 -- 172.21.15.186:0/3144740338 shutdown_connections 2024-04-26T04:18:35.963 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.956+0000 7fe97ddf8640 1 --2- 172.21.15.186:0/3144740338 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fe97813b5a0 0x7fe9780a14a0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:35.963 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.956+0000 7fe97ddf8640 1 --2- 172.21.15.186:0/3144740338 >> v2:172.21.15.63:3300/0 conn(0x7fe9780723a0 0x7fe978072780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:35.963 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.956+0000 7fe97ddf8640 1 -- 172.21.15.186:0/3144740338 >> 172.21.15.186:0/3144740338 conn(0x7fe9780a6090 msgr2=0x7fe9780a2590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:35.963 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.956+0000 7fe97ddf8640 1 -- 172.21.15.186:0/3144740338 shutdown_connections 2024-04-26T04:18:35.963 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.956+0000 7fe97ddf8640 1 -- 172.21.15.186:0/3144740338 wait complete. 2024-04-26T04:18:35.963 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.956+0000 7fe97ddf8640 10 monclient: build_initial_monmap 2024-04-26T04:18:35.963 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.956+0000 7fe97ddf8640 10 monclient: monmap: 2024-04-26T04:18:35.963 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 0 2024-04-26T04:18:35.963 INFO:teuthology.orchestra.run.smithi186.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:18:35.963 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 0.000000 2024-04-26T04:18:35.963 INFO:teuthology.orchestra.run.smithi186.stderr:created 0.000000 2024-04-26T04:18:35.963 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:35.964 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-04-26T04:18:35.964 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:18:35.964 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:18:35.964 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:18:35.964 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-04-26T04:18:35.964 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.956+0000 7fe97ddf8640 1 Processor -- start 2024-04-26T04:18:35.964 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.956+0000 7fe97ddf8640 1 -- start start 2024-04-26T04:18:35.964 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.956+0000 7fe97ddf8640 10 monclient: init 2024-04-26T04:18:35.967 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe97ddf8640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:35.967 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe97ddf8640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T04:18:35.967 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe97ddf8640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe9780723a0 0x7fe978137c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:35.968 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe97ddf8640 10 monclient(hunting): picked mon.noname-b con 0x7fe9780723a0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:18:35.968 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe97ddf8640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fe97813b5a0 0x7fe9781381c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:35.968 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe97ddf8640 10 monclient(hunting): picked mon.noname-c con 0x7fe97813b5a0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:35.968 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe97ddf8640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe97813b9e0 0x7fe97813ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:35.968 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe97ddf8640 10 monclient(hunting): picked mon.noname-a con 0x7fe97813b9e0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:18:35.968 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe97ddf8640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:35.968 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe97ddf8640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe978146700 con 0x7fe97813b9e0 2024-04-26T04:18:35.968 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe97ddf8640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:35.968 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe97ddf8640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe978146580 con 0x7fe9780723a0 2024-04-26T04:18:35.968 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe97ddf8640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:35.968 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe97ddf8640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7fe978146880 con 0x7fe97813b5a0 2024-04-26T04:18:35.969 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe97ddf8640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:35.969 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe97ddf8640 10 monclient(hunting): authenticate will time out at 945.736167s 2024-04-26T04:18:35.969 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe9777fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe9780723a0 0x7fe978137c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:35.969 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe9777fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe9780723a0 0x7fe978137c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:41006/0 (socket says 172.21.15.186:41006) 2024-04-26T04:18:35.969 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe9777fe640 1 -- 172.21.15.186:0/2478817900 learned_addr learned my addr 172.21.15.186:0/2478817900 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-04-26T04:18:35.969 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe9777fe640 10 monclient(hunting): get_auth_request con 0x7fe9780723a0 auth_method 0 2024-04-26T04:18:35.969 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe9777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:35.969 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe9777fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:35.969 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe9777fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:35.969 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe9777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:35.969 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe9777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:35.969 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe9777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:35.969 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe976ffd640 1 --2- 172.21.15.186:0/2478817900 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fe97813b5a0 0x7fe9781381c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:35.969 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe977fff640 1 --2- 172.21.15.186:0/2478817900 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe97813b9e0 0x7fe97813ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:35.969 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe976ffd640 10 monclient(hunting): get_auth_request con 0x7fe97813b5a0 auth_method 0 2024-04-26T04:18:35.970 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe976ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:35.970 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe976ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:35.970 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe976ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:35.970 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe977fff640 10 monclient(hunting): get_auth_request con 0x7fe97813b9e0 auth_method 0 2024-04-26T04:18:35.970 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe977fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:35.970 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe977fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:35.970 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe977fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:35.970 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe9777fe640 10 monclient(hunting): handle_auth_done global_id 4126 payload 824 2024-04-26T04:18:35.970 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe9777fe640 1 -- 172.21.15.186:0/2478817900 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fe97813b5a0 msgr2=0x7fe9781381c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:35.970 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe9777fe640 1 --2- 172.21.15.186:0/2478817900 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fe97813b5a0 0x7fe9781381c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:35.970 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe9777fe640 1 -- 172.21.15.186:0/2478817900 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe97813b9e0 msgr2=0x7fe97813ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:35.970 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe9777fe640 1 --2- 172.21.15.186:0/2478817900 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe97813b9e0 0x7fe97813ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:35.970 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe9777fe640 10 monclient: _finish_hunting 0 2024-04-26T04:18:35.971 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe9777fe640 1 monclient: found mon.noname-b 2024-04-26T04:18:35.971 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe9777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:35.971 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe9777fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.186:3300/0 2024-04-26T04:18:35.971 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe9777fe640 1 -- 172.21.15.186:0/2478817900 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe978140bf0 con 0x7fe9780723a0 2024-04-26T04:18:35.971 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe9777fe640 1 -- 172.21.15.186:0/2478817900 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe9780723a0 msgr2=0x7fe978137c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:35.971 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe9777fe640 1 --2- 172.21.15.186:0/2478817900 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe9780723a0 0x7fe978137c80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe96c00e890 tx=0x7fe96c00ed50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:35.971 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe95ffff640 1 -- 172.21.15.186:0/2478817900 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe96c017070 con 0x7fe9780723a0 2024-04-26T04:18:35.971 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe95ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:35.971 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe95ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T04:18:35.971 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe95ffff640 10 monclient: dump: 2024-04-26T04:18:35.971 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-04-26T04:18:35.971 INFO:teuthology.orchestra.run.smithi186.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:35.971 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:35.972 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:35.972 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:35.972 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-04-26T04:18:35.972 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:35.972 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:35.972 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:35.972 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-04-26T04:18:35.972 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe95ffff640 10 monclient: _finish_auth 0 2024-04-26T04:18:35.972 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe95ffff640 10 monclient: _check_auth_tickets 2024-04-26T04:18:35.972 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe95ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:35.972 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe95ffff640 1 -- 172.21.15.186:0/2478817900 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe96c00cb50 con 0x7fe9780723a0 2024-04-26T04:18:35.972 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe97ddf8640 5 monclient: authenticate success, global_id 4126 2024-04-26T04:18:35.972 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe97ddf8640 10 monclient: _renew_subs 2024-04-26T04:18:35.972 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe97ddf8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-04-26T04:18:35.973 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe97ddf8640 1 -- 172.21.15.186:0/2478817900 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe97809ff30 con 0x7fe9780723a0 2024-04-26T04:18:35.973 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe95ffff640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:35.973 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe95ffff640 1 -- 172.21.15.186:0/2478817900 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe96c013470 con 0x7fe9780723a0 2024-04-26T04:18:35.973 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe95ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:35.973 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe95ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T04:18:35.973 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe95ffff640 10 monclient: dump: 2024-04-26T04:18:35.973 INFO:teuthology.orchestra.run.smithi186.stderr:epoch 1 2024-04-26T04:18:35.973 INFO:teuthology.orchestra.run.smithi186.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:35.973 INFO:teuthology.orchestra.run.smithi186.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:35.973 INFO:teuthology.orchestra.run.smithi186.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:35.973 INFO:teuthology.orchestra.run.smithi186.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:35.973 INFO:teuthology.orchestra.run.smithi186.stderr:election_strategy: 1 2024-04-26T04:18:35.973 INFO:teuthology.orchestra.run.smithi186.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:35.974 INFO:teuthology.orchestra.run.smithi186.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:35.974 INFO:teuthology.orchestra.run.smithi186.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:35.974 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-04-26T04:18:35.974 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe97ddf8640 10 monclient: _renew_subs 2024-04-26T04:18:35.974 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe975ffb640 4 set_mon_vals no callback set 2024-04-26T04:18:35.974 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe97ddf8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-04-26T04:18:35.974 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe97ddf8640 1 -- 172.21.15.186:0/2478817900 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe97813f1b0 con 0x7fe9780723a0 2024-04-26T04:18:35.974 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe95ffff640 1 -- 172.21.15.186:0/2478817900 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe96c01a070 con 0x7fe9780723a0 2024-04-26T04:18:35.974 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe95ffff640 1 -- 172.21.15.186:0/2478817900 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7fe96c013c30 con 0x7fe9780723a0 2024-04-26T04:18:35.974 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe97ddf8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:35.974 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe97ddf8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:35.974 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe97ddf8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-04-26T04:18:35.974 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.960+0000 7fe97ddf8640 1 -- 172.21.15.186:0/2478817900 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe978062e10 con 0x7fe9780723a0 2024-04-26T04:18:35.974 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:35.964+0000 7fe9777fe640 1 --2- 172.21.15.186:0/2478817900 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe9780723a0 0x7fe978137c80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe96c00e890 tx=0x7fe96c00ed50 comp rx=0 tx=0).queue_received will delay after 1.71411e+09 on 0x7fe96c013610 mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) 2024-04-26T04:18:36.330 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-26T04:18:36.328+0000 7f2753209180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-26T04:18:36.384 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-26T04:18:36.384+0000 7f2753209180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-26T04:18:36.435 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-26T04:18:36.436+0000 7f2753209180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-26T04:18:36.511 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-26T04:18:36.508+0000 7f2753209180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-26T04:18:36.561 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-26T04:18:36.560+0000 7f2753209180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-26T04:18:36.637 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-26T04:18:36.636+0000 7f2753209180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-26T04:18:36.689 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-26T04:18:36.688+0000 7f2753209180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-26T04:18:36.793 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:36.792+0000 7fe95ffff640 1 -- 172.21.15.186:0/2478817900 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fe96c013610 con 0x7fe9780723a0 2024-04-26T04:18:36.793 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:36.792+0000 7fe95ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:36.793 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:36.792+0000 7fe95ffff640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:18:36.836 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-26T04:18:36.836+0000 7f2753209180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-26T04:18:36.861 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:36.860+0000 7fe97ddf8640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "70454d09-e50c-4f65-83c5-0f9bb5c0f8f6", "id": 5}] 2024-04-26T04:18:36.861 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:36.860+0000 7fe97ddf8640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "70454d09-e50c-4f65-83c5-0f9bb5c0f8f6", "id": 5}] 2024-04-26T04:18:36.861 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:36.860+0000 7fe97ddf8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-04-26T04:18:36.861 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:36.860+0000 7fe97ddf8640 1 -- 172.21.15.186:0/2478817900 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd new", "uuid": "70454d09-e50c-4f65-83c5-0f9bb5c0f8f6", "id": 5} v 0) -- 0x7fe97806e5d0 con 0x7fe9780723a0 2024-04-26T04:18:36.869 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:36.868+0000 7fe95ffff640 1 -- 172.21.15.186:0/2478817900 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "70454d09-e50c-4f65-83c5-0f9bb5c0f8f6", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7fe96c0059a0 con 0x7fe9780723a0 2024-04-26T04:18:36.869 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:36.868+0000 7fe95ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "70454d09-e50c-4f65-83c5-0f9bb5c0f8f6", "id": 5}] 2024-04-26T04:18:36.870 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:36.868+0000 7fe95ffff640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:18:36.870 INFO:teuthology.orchestra.run.smithi186.stdout:5 2024-04-26T04:18:36.871 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:36.868+0000 7fe97ddf8640 10 monclient: shutdown 2024-04-26T04:18:36.871 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:36.868+0000 7fe97ddf8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:36.871 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:36.868+0000 7fe97ddf8640 1 -- 172.21.15.186:0/2478817900 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe9780723a0 msgr2=0x7fe978137c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:36.871 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:36.868+0000 7fe97ddf8640 1 --2- 172.21.15.186:0/2478817900 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe9780723a0 0x7fe978137c80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe96c00e890 tx=0x7fe96c00ed50 comp rx=0 tx=0).stop 2024-04-26T04:18:36.871 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:36.868+0000 7fe97ddf8640 1 -- 172.21.15.186:0/2478817900 shutdown_connections 2024-04-26T04:18:36.871 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:36.868+0000 7fe97ddf8640 1 --2- 172.21.15.186:0/2478817900 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe97813b9e0 0x7fe97813ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:36.871 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:36.868+0000 7fe97ddf8640 1 --2- 172.21.15.186:0/2478817900 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fe97813b5a0 0x7fe9781381c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:36.871 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:36.868+0000 7fe97ddf8640 1 --2- 172.21.15.186:0/2478817900 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe9780723a0 0x7fe978137c80 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:36.872 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:36.868+0000 7fe97ddf8640 1 -- 172.21.15.186:0/2478817900 >> 172.21.15.186:0/2478817900 conn(0x7fe9780a6090 msgr2=0x7fe9780a3990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:36.872 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:36.868+0000 7fe97ddf8640 1 -- 172.21.15.186:0/2478817900 shutdown_connections 2024-04-26T04:18:36.872 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-26T04:18:36.868+0000 7fe97ddf8640 1 -- 172.21.15.186:0/2478817900 wait complete. 2024-04-26T04:18:36.888 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-26T04:18:36.888 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-04-26T04:18:36.890 INFO:tasks.ceph.osd.0:Started 2024-04-26T04:18:36.890 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-26T04:18:36.891 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-04-26T04:18:36.893 INFO:tasks.ceph.osd.1:Started 2024-04-26T04:18:36.893 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-26T04:18:36.893 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-04-26T04:18:36.895 INFO:tasks.ceph.osd.2:Started 2024-04-26T04:18:36.895 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-26T04:18:36.895 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-04-26T04:18:36.897 INFO:tasks.ceph.osd.3:Started 2024-04-26T04:18:36.897 INFO:tasks.ceph.osd.4:Restarting daemon 2024-04-26T04:18:36.897 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-04-26T04:18:36.899 INFO:tasks.ceph.osd.4:Started 2024-04-26T04:18:36.899 INFO:tasks.ceph.osd.5:Restarting daemon 2024-04-26T04:18:36.899 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-04-26T04:18:36.901 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-26T04:18:36.900+0000 7f2753209180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-26T04:18:36.901 INFO:tasks.ceph.osd.5:Started 2024-04-26T04:18:36.902 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T04:18:37.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.014+0000 7f2e5d5fe640 10 monclient: get_monmap_and_config 2024-04-26T04:18:37.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.014+0000 7f2e5d5fe640 10 monclient: build_initial_monmap 2024-04-26T04:18:37.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.014+0000 7f2e5d5fe640 10 monclient: monmap: 2024-04-26T04:18:37.020 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:37.020 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:37.020 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:37.020481+0000 2024-04-26T04:18:37.021 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:37.020481+0000 2024-04-26T04:18:37.021 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:37.021 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:37.021 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:18:37.021 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:18:37.021 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:18:37.021 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:18:37.021 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:18:37.021 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:37.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.014+0000 7f2e5d5fe640 1 Processor -- start 2024-04-26T04:18:37.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.014+0000 7f2e5d5fe640 1 -- start start 2024-04-26T04:18:37.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.014+0000 7f2e5d5fe640 10 monclient: init 2024-04-26T04:18:37.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.014+0000 7f2e5d5fe640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:37.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.018+0000 7f2e5d5fe640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-04-26T04:18:37.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.018+0000 7f2e5d5fe640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f2e5809fe60 0x7f2e581372f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:37.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.018+0000 7f2e5d5fe640 10 monclient(hunting): picked mon.noname-c con 0x7f2e5809fe60 addr v2:172.21.15.186:3300/0 2024-04-26T04:18:37.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.018+0000 7f2e5d5fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2e580a0240 addr v1:172.21.15.63:6789/0 2024-04-26T04:18:37.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.018+0000 7f2e5d5fe640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f2e5813c100 0x7f2e5813e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:37.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.018+0000 7f2e5d5fe640 10 monclient(hunting): picked mon.noname-a con 0x7f2e5813c100 addr v2:172.21.15.63:3300/0 2024-04-26T04:18:37.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.018+0000 7f2e5d5fe640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e58137830 con 0x7f2e580a0240 2024-04-26T04:18:37.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.018+0000 7f2e5d5fe640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:37.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.018+0000 7f2e5d5fe640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7f2e580a0c60 con 0x7f2e5813c100 2024-04-26T04:18:37.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.018+0000 7f2e5d5fe640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:37.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.018+0000 7f2e5d5fe640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f2e580a0f40 con 0x7f2e5809fe60 2024-04-26T04:18:37.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.018+0000 7f2e5d5fe640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:37.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.018+0000 7f2e5d5fe640 10 monclient(hunting): authenticate will time out at 948.728120s 2024-04-26T04:18:37.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.018+0000 7f2e577fe640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f2e5813c100 0x7f2e5813e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:37.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.018+0000 7f2e577fe640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f2e5813c100 0x7f2e5813e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:55936/0 (socket says 172.21.15.63:55936) 2024-04-26T04:18:37.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.018+0000 7f2e577fe640 1 -- 172.21.15.63:0/198661544 learned_addr learned my addr 172.21.15.63:0/198661544 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:18:37.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.018+0000 7f2e577fe640 10 monclient(hunting): get_auth_request con 0x7f2e5813c100 auth_method 0 2024-04-26T04:18:37.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.018+0000 7f2e577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:37.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.018+0000 7f2e577fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:37.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.018+0000 7f2e577fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:37.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.018+0000 7f2e577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:37.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.018+0000 7f2e577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:37.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.018+0000 7f2e577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:37.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.018+0000 7f2e56ffd640 1 --2- 172.21.15.63:0/198661544 >> v2:172.21.15.186:3300/0 conn(0x7f2e5809fe60 0x7f2e581372f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:37.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.018+0000 7f2e56ffd640 10 monclient(hunting): get_auth_request con 0x7f2e5809fe60 auth_method 0 2024-04-26T04:18:37.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.018+0000 7f2e56ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:37.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.018+0000 7f2e56ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:37.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.018+0000 7f2e56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:37.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.018+0000 7f2e56ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:37.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.018+0000 7f2e56ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:37.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.018+0000 7f2e56ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:37.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.018+0000 7f2e56ffd640 10 monclient(hunting): handle_auth_done global_id 4147 payload 274 2024-04-26T04:18:37.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.018+0000 7f2e56ffd640 1 -- 172.21.15.63:0/198661544 >> v2:172.21.15.63:3300/0 conn(0x7f2e5813c100 msgr2=0x7f2e5813e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:37.027 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.018+0000 7f2e56ffd640 1 --2- 172.21.15.63:0/198661544 >> v2:172.21.15.63:3300/0 conn(0x7f2e5813c100 0x7f2e5813e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:37.027 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.018+0000 7f2e56ffd640 1 -- 172.21.15.63:0/198661544 >> v1:172.21.15.63:6789/0 conn(0x7f2e580a0240 legacy=0x7f2e580a2a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:37.027 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.018+0000 7f2e56ffd640 10 monclient: _finish_hunting 0 2024-04-26T04:18:37.027 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.018+0000 7f2e56ffd640 1 monclient: found mon.noname-c 2024-04-26T04:18:37.027 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.018+0000 7f2e56ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:37.027 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.018+0000 7f2e56ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.186:3300/0 2024-04-26T04:18:37.027 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.018+0000 7f2e56ffd640 1 -- 172.21.15.63:0/198661544 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e580a1220 con 0x7f2e5809fe60 2024-04-26T04:18:37.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.022+0000 7f2e56ffd640 1 -- 172.21.15.63:0/198661544 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e5809fe60 msgr2=0x7f2e581372f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:37.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.022+0000 7f2e56ffd640 1 --2- 172.21.15.63:0/198661544 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e5809fe60 0x7f2e581372f0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f2e580a7010 tx=0x7f2e4801c300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f1132eb48075a718 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:37.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.022+0000 7f2e55ffb640 1 -- 172.21.15.63:0/198661544 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e480b0070 con 0x7f2e5809fe60 2024-04-26T04:18:37.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.022+0000 7f2e55ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:37.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.022+0000 7f2e55ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:18:37.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.022+0000 7f2e55ffb640 10 monclient: dump: 2024-04-26T04:18:37.029 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:37.029 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:37.029 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:37.029 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:37.029 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:37.029 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:37.030 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:37.030 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:37.030 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:37.030 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:37.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.022+0000 7f2e55ffb640 10 monclient: _finish_auth 0 2024-04-26T04:18:37.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.022+0000 7f2e55ffb640 10 monclient: _check_auth_tickets 2024-04-26T04:18:37.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.022+0000 7f2e55ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:37.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.022+0000 7f2e55ffb640 1 -- 172.21.15.63:0/198661544 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e480ab600 con 0x7f2e5809fe60 2024-04-26T04:18:37.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.022+0000 7f2e55ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:37.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.022+0000 7f2e55ffb640 1 -- 172.21.15.63:0/198661544 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e480ab9d0 con 0x7f2e5809fe60 2024-04-26T04:18:37.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.022+0000 7f2e5d5fe640 5 monclient: authenticate success, global_id 4147 2024-04-26T04:18:37.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.022+0000 7f2e5d5fe640 10 monclient: get_monmap_and_config success 2024-04-26T04:18:37.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.022+0000 7f2e5d5fe640 4 set_mon_vals no callback set 2024-04-26T04:18:37.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.022+0000 7f2e55ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:37.031 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.022+0000 7f2e55ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T04:18:37.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.022+0000 7f2e55ffb640 10 monclient: dump: 2024-04-26T04:18:37.033 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:37.033 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:37.033 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:37.033 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:37.033 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:37.033 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:37.033 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:37.033 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:37.034 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:37.034 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:37.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.022+0000 7f2e5d5fe640 10 monclient: shutdown 2024-04-26T04:18:37.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.022+0000 7f2e5d5fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:37.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.022+0000 7f2e5d5fe640 1 -- 172.21.15.63:0/198661544 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e5809fe60 msgr2=0x7f2e581372f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:37.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.022+0000 7f2e5d5fe640 1 --2- 172.21.15.63:0/198661544 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e5809fe60 0x7f2e581372f0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f2e580a7010 tx=0x7f2e4801c300 comp rx=0 tx=0).stop 2024-04-26T04:18:37.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.022+0000 7f2e5d5fe640 1 -- 172.21.15.63:0/198661544 shutdown_connections 2024-04-26T04:18:37.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.022+0000 7f2e5d5fe640 1 --2- 172.21.15.63:0/198661544 >> v2:172.21.15.63:3300/0 conn(0x7f2e5813c100 0x7f2e5813e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:37.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.022+0000 7f2e5d5fe640 1 --2- 172.21.15.63:0/198661544 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e5809fe60 0x7f2e581372f0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:37.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.022+0000 7f2e5d5fe640 1 -- 172.21.15.63:0/198661544 >> 172.21.15.63:0/198661544 conn(0x7f2e580a5fb0 msgr2=0x7f2e580a63c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:37.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.022+0000 7f2e5d5fe640 1 -- 172.21.15.63:0/198661544 shutdown_connections 2024-04-26T04:18:37.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.022+0000 7f2e5d5fe640 1 -- 172.21.15.63:0/198661544 wait complete. 2024-04-26T04:18:37.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.022+0000 7f2e5d5fe640 10 monclient: build_initial_monmap 2024-04-26T04:18:37.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.022+0000 7f2e5d5fe640 10 monclient: monmap: 2024-04-26T04:18:37.035 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:37.035 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:18:37.035 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:18:37.035 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:18:37.035 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:37.035 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:37.035 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:18:37.035 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:18:37.035 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:18:37.035 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:37.035 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.022+0000 7f2e5d5fe640 1 Processor -- start 2024-04-26T04:18:37.035 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.022+0000 7f2e5d5fe640 1 -- start start 2024-04-26T04:18:37.035 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.022+0000 7f2e5d5fe640 10 monclient: init 2024-04-26T04:18:37.039 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e5d5fe640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:37.039 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e5d5fe640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T04:18:37.039 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e5d5fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2e5809fe60 0x7f2e58139b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:37.039 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e5d5fe640 10 monclient(hunting): picked mon.noname-a con 0x7f2e5809fe60 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:18:37.039 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e5d5fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e580a0240 0x7f2e5813a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:37.039 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e5d5fe640 10 monclient(hunting): picked mon.noname-b con 0x7f2e580a0240 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:18:37.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e5d5fe640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f2e5813c100 0x7f2e5813a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:37.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e5d5fe640 10 monclient(hunting): picked mon.noname-c con 0x7f2e5813c100 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:37.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e5d5fe640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:37.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e5d5fe640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f2e581415e0 con 0x7f2e5809fe60 2024-04-26T04:18:37.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e5d5fe640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:37.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e5d5fe640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2e58141460 con 0x7f2e580a0240 2024-04-26T04:18:37.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e5d5fe640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:37.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e5d5fe640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f2e58141760 con 0x7f2e5813c100 2024-04-26T04:18:37.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e5d5fe640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:37.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e5d5fe640 10 monclient(hunting): authenticate will time out at 948.736251s 2024-04-26T04:18:37.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e56ffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2e5809fe60 0x7f2e58139b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:37.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e56ffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2e5809fe60 0x7f2e58139b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:55950/0 (socket says 172.21.15.63:55950) 2024-04-26T04:18:37.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e56ffd640 1 -- 172.21.15.63:0/3067145319 learned_addr learned my addr 172.21.15.63:0/3067145319 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:18:37.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e56ffd640 10 monclient(hunting): get_auth_request con 0x7f2e5809fe60 auth_method 0 2024-04-26T04:18:37.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e56ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:37.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e56ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:37.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:37.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e577fe640 1 --2- 172.21.15.63:0/3067145319 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f2e5813c100 0x7f2e5813a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:37.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e56ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:37.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e56ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:37.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e56ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:37.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e577fe640 10 monclient(hunting): get_auth_request con 0x7f2e5813c100 auth_method 0 2024-04-26T04:18:37.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:37.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e577fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:37.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e577fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:37.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e567fc640 1 --2- 172.21.15.63:0/3067145319 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e580a0240 0x7f2e5813a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:37.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:37.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:37.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:37.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e567fc640 10 monclient(hunting): get_auth_request con 0x7f2e580a0240 auth_method 0 2024-04-26T04:18:37.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e567fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:37.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e567fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:37.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e567fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:37.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e56ffd640 10 monclient(hunting): handle_auth_done global_id 4146 payload 824 2024-04-26T04:18:37.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e56ffd640 1 -- 172.21.15.63:0/3067145319 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f2e5813c100 msgr2=0x7f2e5813a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:37.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e56ffd640 1 --2- 172.21.15.63:0/3067145319 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f2e5813c100 0x7f2e5813a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:37.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e56ffd640 1 -- 172.21.15.63:0/3067145319 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e580a0240 msgr2=0x7f2e5813a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:37.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e56ffd640 1 --2- 172.21.15.63:0/3067145319 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e580a0240 0x7f2e5813a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:37.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e56ffd640 10 monclient: _finish_hunting 0 2024-04-26T04:18:37.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e56ffd640 1 monclient: found mon.noname-a 2024-04-26T04:18:37.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e56ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:37.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e56ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:18:37.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e56ffd640 1 -- 172.21.15.63:0/3067145319 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e580a38f0 con 0x7f2e5809fe60 2024-04-26T04:18:37.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e567fc640 1 --2- 172.21.15.63:0/3067145319 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e580a0240 0x7f2e5813a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T04:18:37.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e56ffd640 1 -- 172.21.15.63:0/3067145319 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2e5809fe60 msgr2=0x7f2e58139b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:37.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e56ffd640 1 --2- 172.21.15.63:0/3067145319 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2e5809fe60 0x7f2e58139b70 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2e480ae040 tx=0x7f2e4801ab40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:37.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e3f7fe640 1 -- 172.21.15.63:0/3067145319 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e480b0050 con 0x7f2e5809fe60 2024-04-26T04:18:37.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e3f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:37.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e3f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:18:37.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e3f7fe640 10 monclient: dump: 2024-04-26T04:18:37.043 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:37.043 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:37.043 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:37.043 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:37.043 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:37.044 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:37.044 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:37.044 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:37.044 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:37.044 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:37.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e3f7fe640 10 monclient: _finish_auth 0 2024-04-26T04:18:37.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e3f7fe640 10 monclient: _check_auth_tickets 2024-04-26T04:18:37.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e3f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:37.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e3f7fe640 1 -- 172.21.15.63:0/3067145319 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e480bc070 con 0x7f2e5809fe60 2024-04-26T04:18:37.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e3f7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:37.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e3f7fe640 1 -- 172.21.15.63:0/3067145319 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e480b5c40 con 0x7f2e5809fe60 2024-04-26T04:18:37.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e3f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:37.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e3f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:18:37.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e3f7fe640 10 monclient: dump: 2024-04-26T04:18:37.044 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:37.045 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:37.045 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:37.045 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:37.045 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:37.045 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:37.045 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:37.045 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:37.045 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:37.045 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:37.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e5d5fe640 5 monclient: authenticate success, global_id 4146 2024-04-26T04:18:37.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e5d5fe640 10 monclient: _renew_subs 2024-04-26T04:18:37.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e5d5fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:37.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e557fa640 4 set_mon_vals no callback set 2024-04-26T04:18:37.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e5d5fe640 1 -- 172.21.15.63:0/3067145319 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e580a75d0 con 0x7f2e5809fe60 2024-04-26T04:18:37.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e5d5fe640 10 monclient: _renew_subs 2024-04-26T04:18:37.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e5d5fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:37.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e5d5fe640 1 -- 172.21.15.63:0/3067145319 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e5806e660 con 0x7f2e5809fe60 2024-04-26T04:18:37.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e3f7fe640 1 -- 172.21.15.63:0/3067145319 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f2e480c1460 con 0x7f2e5809fe60 2024-04-26T04:18:37.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.026+0000 7f2e3f7fe640 1 -- 172.21.15.63:0/3067145319 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f2e480ab590 con 0x7f2e5809fe60 2024-04-26T04:18:37.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.030+0000 7f2e5d5fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:37.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.030+0000 7f2e5d5fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:37.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.030+0000 7f2e5d5fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:37.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.030+0000 7f2e5d5fe640 1 -- 172.21.15.63:0/3067145319 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e58062e10 con 0x7f2e5809fe60 2024-04-26T04:18:37.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.030+0000 7f2e3f7fe640 1 -- 172.21.15.63:0/3067145319 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f2e48016010 con 0x7f2e5809fe60 2024-04-26T04:18:37.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.030+0000 7f2e3f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:37.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.030+0000 7f2e3f7fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:18:37.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.090+0000 7f2e5d5fe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-26T04:18:37.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.090+0000 7f2e5d5fe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-26T04:18:37.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.090+0000 7f2e5d5fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:37.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.090+0000 7f2e5d5fe640 1 -- 172.21.15.63:0/3067145319 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2e58068430 con 0x7f2e5809fe60 2024-04-26T04:18:37.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.090+0000 7f2e3f7fe640 1 -- 172.21.15.63:0/3067145319 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) ==== 74+0+5605 (secure 0 0 0) 0x7f2e480b7a90 con 0x7f2e5809fe60 2024-04-26T04:18:37.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.090+0000 7f2e3f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-26T04:18:37.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.090+0000 7f2e3f7fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:18:37.095 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-26T04:18:37.095 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":8,"fsid":"cd1c6a0c-4e4b-4014-89c0-2e535479ef39","created":"2024-04-26T04:18:34.126231+0000","modified":"2024-04-26T04:18:36.862750+0000","last_up_change":"0.000000","last_in_change":"2024-04-26T04:18:36.862750+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2d37141b-8282-4c62-864a-84a75ebf8f6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"7220e45a-72bb-4d8e-93c2-32005fa2c85e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"913de4d8-31af-4394-a0ee-194cc31f822f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c492772c-ccf5-4b85-9056-6a5ee9686c0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"24f067f0-554a-4dd2-a26c-7bda3d0978f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"70454d09-e50c-4f65-83c5-0f9bb5c0f8f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T04:18:37.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.090+0000 7f2e5d5fe640 10 monclient: shutdown 2024-04-26T04:18:37.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.090+0000 7f2e5d5fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:37.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.090+0000 7f2e5d5fe640 1 -- 172.21.15.63:0/3067145319 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2e5809fe60 msgr2=0x7f2e58139b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:37.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.090+0000 7f2e5d5fe640 1 --2- 172.21.15.63:0/3067145319 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2e5809fe60 0x7f2e58139b70 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2e480ae040 tx=0x7f2e4801ab40 comp rx=0 tx=0).stop 2024-04-26T04:18:37.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.090+0000 7f2e5d5fe640 1 -- 172.21.15.63:0/3067145319 shutdown_connections 2024-04-26T04:18:37.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.090+0000 7f2e5d5fe640 1 --2- 172.21.15.63:0/3067145319 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f2e5813c100 0x7f2e5813a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:37.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.090+0000 7f2e5d5fe640 1 --2- 172.21.15.63:0/3067145319 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e580a0240 0x7f2e5813a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:37.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.090+0000 7f2e5d5fe640 1 --2- 172.21.15.63:0/3067145319 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2e5809fe60 0x7f2e58139b70 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:37.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.090+0000 7f2e5d5fe640 1 -- 172.21.15.63:0/3067145319 >> 172.21.15.63:0/3067145319 conn(0x7f2e580a5fb0 msgr2=0x7f2e58132ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:37.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.090+0000 7f2e5d5fe640 1 -- 172.21.15.63:0/3067145319 shutdown_connections 2024-04-26T04:18:37.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:37.090+0000 7f2e5d5fe640 1 -- 172.21.15.63:0/3067145319 wait complete. 2024-04-26T04:18:37.112 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-26T04:18:37.112 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-26T04:18:37.516 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-04-26T04:18:37.516+0000 7f1a6f9527c0 -1 Falling back to public interface 2024-04-26T04:18:37.517 INFO:tasks.ceph.osd.3.smithi186.stderr:2024-04-26T04:18:37.516+0000 7f79351487c0 -1 Falling back to public interface 2024-04-26T04:18:37.519 INFO:tasks.ceph.osd.0.smithi063.stderr:2024-04-26T04:18:37.514+0000 7fba62dbe7c0 -1 Falling back to public interface 2024-04-26T04:18:37.528 INFO:tasks.ceph.osd.5.smithi186.stderr:2024-04-26T04:18:37.528+0000 7ff640d3d7c0 -1 Falling back to public interface 2024-04-26T04:18:37.531 INFO:tasks.ceph.osd.2.smithi063.stderr:2024-04-26T04:18:37.526+0000 7f131e3f97c0 -1 Falling back to public interface 2024-04-26T04:18:37.542 INFO:tasks.ceph.osd.1.smithi063.stderr:2024-04-26T04:18:37.538+0000 7f603ef997c0 -1 Falling back to public interface 2024-04-26T04:18:37.564 INFO:tasks.ceph.mgr.x.smithi186.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-04-26T04:18:37.564 INFO:tasks.ceph.mgr.x.smithi186.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-04-26T04:18:37.564 INFO:tasks.ceph.mgr.x.smithi186.stderr: from numpy import show_config as show_numpy_config 2024-04-26T04:18:37.611 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-26T04:18:37.612+0000 7f2753209180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-26T04:18:37.717 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-26T04:18:37.760 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-26T04:18:37.760+0000 7f2753209180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-26T04:18:37.841 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.834+0000 7f42b9a85640 10 monclient: get_monmap_and_config 2024-04-26T04:18:37.841 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.834+0000 7f42b9a85640 10 monclient: build_initial_monmap 2024-04-26T04:18:37.841 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.834+0000 7f42b9a85640 10 monclient: monmap: 2024-04-26T04:18:37.841 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-04-26T04:18:37.841 INFO:teuthology.misc.health.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:37.841 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-04-26T04:18:37.840781+0000 2024-04-26T04:18:37.841 INFO:teuthology.misc.health.smithi063.stderr:created 2024-04-26T04:18:37.840781+0000 2024-04-26T04:18:37.842 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:37.842 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:37.842 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:18:37.842 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:18:37.842 INFO:teuthology.misc.health.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:18:37.842 INFO:teuthology.misc.health.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:18:37.842 INFO:teuthology.misc.health.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:18:37.843 INFO:teuthology.misc.health.smithi063.stderr: 2024-04-26T04:18:37.849 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.838+0000 7f42b9a85640 1 Processor -- start 2024-04-26T04:18:37.849 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.838+0000 7f42b9a85640 1 -- start start 2024-04-26T04:18:37.849 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.838+0000 7f42b9a85640 10 monclient: init 2024-04-26T04:18:37.849 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.838+0000 7f42b9a85640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:37.849 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.838+0000 7f42b9a85640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-04-26T04:18:37.849 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.838+0000 7f42b9a85640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f42b409fab0 addr v1:172.21.15.63:6789/0 2024-04-26T04:18:37.849 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.838+0000 7f42b9a85640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f42b4137e10 addr v1:172.21.15.186:6789/0 2024-04-26T04:18:37.849 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.838+0000 7f42b9a85640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f42b413b920 0x7f42b413dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:37.849 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.838+0000 7f42b9a85640 10 monclient(hunting): picked mon.noname-b con 0x7f42b413b920 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:37.850 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.838+0000 7f42b9a85640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42b4134fc0 con 0x7f42b409fab0 2024-04-26T04:18:37.850 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.838+0000 7f42b9a85640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42b409dce0 con 0x7f42b4137e10 2024-04-26T04:18:37.850 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.838+0000 7f42b9a85640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:37.850 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.838+0000 7f42b9a85640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f42b40a05a0 con 0x7f42b413b920 2024-04-26T04:18:37.850 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.838+0000 7f42b9a85640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:37.850 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.838+0000 7f42b9a85640 10 monclient(hunting): authenticate will time out at 949.549311s 2024-04-26T04:18:37.850 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.838+0000 7f42b37fe640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f42b409fab0 0x7f42b409fe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:36432/0 (socket says 172.21.15.63:36432) 2024-04-26T04:18:37.850 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.838+0000 7f42b3fff640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f42b413b920 0x7f42b413dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:37.851 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.838+0000 7f42b37fe640 1 -- 172.21.15.63:0/1383797555 learned_addr learned my addr 172.21.15.63:0/1383797555 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-04-26T04:18:37.851 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.838+0000 7f42b3fff640 10 monclient(hunting): get_auth_request con 0x7f42b413b920 auth_method 0 2024-04-26T04:18:37.851 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.838+0000 7f42b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:37.851 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.838+0000 7f42b3fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:37.851 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.838+0000 7f42b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:37.851 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.838+0000 7f42b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:37.851 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.838+0000 7f42b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:37.851 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.838+0000 7f42b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:37.851 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.838+0000 7f42b3fff640 10 monclient(hunting): handle_auth_done global_id 4136 payload 274 2024-04-26T04:18:37.851 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.838+0000 7f42b3fff640 1 -- 172.21.15.63:0/1383797555 >> v1:172.21.15.186:6789/0 conn(0x7f42b4137e10 legacy=0x7f42b413a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:37.852 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.838+0000 7f42b3fff640 1 -- 172.21.15.63:0/1383797555 >> v1:172.21.15.63:6789/0 conn(0x7f42b409fab0 legacy=0x7f42b409fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:37.852 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.838+0000 7f42b3fff640 10 monclient: _finish_hunting 0 2024-04-26T04:18:37.852 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.838+0000 7f42b3fff640 1 monclient: found mon.noname-b 2024-04-26T04:18:37.852 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.838+0000 7f42b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:37.852 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.838+0000 7f42b3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.63:3301/0 2024-04-26T04:18:37.852 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.838+0000 7f42b3fff640 1 -- 172.21.15.63:0/1383797555 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42b409dfc0 con 0x7f42b413b920 2024-04-26T04:18:37.852 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.842+0000 7f42b3fff640 1 -- 172.21.15.63:0/1383797555 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f42b413b920 msgr2=0x7f42b413dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:37.852 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.842+0000 7f42b3fff640 1 --2- 172.21.15.63:0/1383797555 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f42b413b920 0x7f42b413dd10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f42ac009dc0 tx=0x7f42ac01c5f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=71ae86e92b7df734 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:37.852 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.842+0000 7f42b27fc640 1 -- 172.21.15.63:0/1383797555 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42ac0a86d0 con 0x7f42b413b920 2024-04-26T04:18:37.852 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.842+0000 7f42b27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:37.852 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.842+0000 7f42b27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T04:18:37.852 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.842+0000 7f42b27fc640 10 monclient: dump: 2024-04-26T04:18:37.852 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-04-26T04:18:37.852 INFO:teuthology.misc.health.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:37.853 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:37.853 INFO:teuthology.misc.health.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:37.853 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:37.853 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:37.853 INFO:teuthology.misc.health.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:37.853 INFO:teuthology.misc.health.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:37.853 INFO:teuthology.misc.health.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:37.853 INFO:teuthology.misc.health.smithi063.stderr: 2024-04-26T04:18:37.853 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.842+0000 7f42b27fc640 10 monclient: _finish_auth 0 2024-04-26T04:18:37.853 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.842+0000 7f42b27fc640 10 monclient: _check_auth_tickets 2024-04-26T04:18:37.853 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.842+0000 7f42b27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:37.853 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.842+0000 7f42b9a85640 5 monclient: authenticate success, global_id 4136 2024-04-26T04:18:37.853 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.842+0000 7f42b27fc640 1 -- 172.21.15.63:0/1383797555 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42ac0a8870 con 0x7f42b413b920 2024-04-26T04:18:37.854 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.842+0000 7f42b9a85640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T04:18:37.854 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.842+0000 7f42b27fc640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:37.854 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.842+0000 7f42b27fc640 1 -- 172.21.15.63:0/1383797555 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42ac0a8c40 con 0x7f42b413b920 2024-04-26T04:18:37.854 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.842+0000 7f42b27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:37.854 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.842+0000 7f42b27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:18:37.854 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.842+0000 7f42b27fc640 10 monclient: dump: 2024-04-26T04:18:37.854 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-04-26T04:18:37.854 INFO:teuthology.misc.health.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:37.854 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:37.855 INFO:teuthology.misc.health.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:37.855 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:37.855 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:37.855 INFO:teuthology.misc.health.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:37.855 INFO:teuthology.misc.health.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:37.855 INFO:teuthology.misc.health.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:37.855 INFO:teuthology.misc.health.smithi063.stderr: 2024-04-26T04:18:37.855 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.842+0000 7f42b9a85640 10 monclient: get_monmap_and_config success 2024-04-26T04:18:37.856 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.842+0000 7f42b9a85640 4 set_mon_vals no callback set 2024-04-26T04:18:37.856 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.842+0000 7f42b9a85640 10 monclient: shutdown 2024-04-26T04:18:37.856 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.842+0000 7f42b9a85640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:37.856 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.842+0000 7f42b9a85640 1 -- 172.21.15.63:0/1383797555 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f42b413b920 msgr2=0x7f42b413dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:37.856 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.842+0000 7f42b9a85640 1 --2- 172.21.15.63:0/1383797555 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f42b413b920 0x7f42b413dd10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f42ac009dc0 tx=0x7f42ac01c5f0 comp rx=0 tx=0).stop 2024-04-26T04:18:37.856 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.842+0000 7f42b9a85640 1 -- 172.21.15.63:0/1383797555 shutdown_connections 2024-04-26T04:18:37.856 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.842+0000 7f42b9a85640 1 --2- 172.21.15.63:0/1383797555 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f42b413b920 0x7f42b413dd10 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:37.857 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.842+0000 7f42b9a85640 1 -- 172.21.15.63:0/1383797555 >> 172.21.15.63:0/1383797555 conn(0x7f42b406d090 msgr2=0x7f42b406d4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:37.857 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.842+0000 7f42b9a85640 1 -- 172.21.15.63:0/1383797555 shutdown_connections 2024-04-26T04:18:37.857 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.842+0000 7f42b9a85640 1 -- 172.21.15.63:0/1383797555 wait complete. 2024-04-26T04:18:37.857 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.842+0000 7f42b9a85640 10 monclient: build_initial_monmap 2024-04-26T04:18:37.857 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.842+0000 7f42b9a85640 10 monclient: monmap: 2024-04-26T04:18:37.857 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-04-26T04:18:37.857 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:18:37.857 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-04-26T04:18:37.857 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-04-26T04:18:37.857 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:37.857 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:37.857 INFO:teuthology.misc.health.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:18:37.857 INFO:teuthology.misc.health.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:18:37.857 INFO:teuthology.misc.health.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:18:37.858 INFO:teuthology.misc.health.smithi063.stderr: 2024-04-26T04:18:37.863 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.842+0000 7f42b9a85640 1 Processor -- start 2024-04-26T04:18:37.864 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.842+0000 7f42b9a85640 1 -- start start 2024-04-26T04:18:37.864 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.842+0000 7f42b9a85640 10 monclient: init 2024-04-26T04:18:37.864 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f42b9a85640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:37.864 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f42b9a85640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T04:18:37.864 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f42b9a85640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f42b409fab0 0x7f42b41344b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:37.864 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f42b9a85640 10 monclient(hunting): picked mon.noname-a con 0x7f42b409fab0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:18:37.864 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f42b9a85640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42b4137e10 0x7f42b409aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:37.864 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f42b9a85640 10 monclient(hunting): picked mon.noname-b con 0x7f42b4137e10 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:18:37.864 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f42b9a85640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f42b413b920 0x7f42b409af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:37.864 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f42b9a85640 10 monclient(hunting): picked mon.noname-c con 0x7f42b413b920 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:37.864 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f42b9a85640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:37.864 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f42b9a85640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f42b4074e40 con 0x7f42b409fab0 2024-04-26T04:18:37.865 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f42b9a85640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:37.865 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f42b9a85640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f42b4074cc0 con 0x7f42b4137e10 2024-04-26T04:18:37.865 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f42b9a85640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:37.865 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f42b9a85640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f42b4140a70 con 0x7f42b413b920 2024-04-26T04:18:37.865 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f42b9a85640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:37.865 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f42b9a85640 10 monclient(hunting): authenticate will time out at 949.556541s 2024-04-26T04:18:37.865 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f42b37fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f42b409fab0 0x7f42b41344b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:37.865 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f42b3fff640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f42b413b920 0x7f42b409af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:37.865 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f42b3fff640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f42b413b920 0x7f42b409af80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:41620/0 (socket says 172.21.15.63:41620) 2024-04-26T04:18:37.865 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f42b3fff640 1 -- 172.21.15.63:0/2333918891 learned_addr learned my addr 172.21.15.63:0/2333918891 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:18:37.865 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f42b37fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f42b409fab0 0x7f42b41344b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:55966/0 (socket says 172.21.15.63:55966) 2024-04-26T04:18:37.865 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f42b3fff640 10 monclient(hunting): get_auth_request con 0x7f42b413b920 auth_method 0 2024-04-26T04:18:37.866 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f42b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:37.866 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f42b3fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:37.866 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f42b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:37.866 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f42b37fe640 10 monclient(hunting): get_auth_request con 0x7f42b409fab0 auth_method 0 2024-04-26T04:18:37.866 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f42b37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:37.866 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f42b37fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:37.866 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f42b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:37.866 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f42b2ffd640 1 --2- 172.21.15.63:0/2333918891 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42b4137e10 0x7f42b409aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:37.866 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f42b2ffd640 10 monclient(hunting): get_auth_request con 0x7f42b4137e10 auth_method 0 2024-04-26T04:18:37.866 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f42b2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:37.866 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f42b2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:37.866 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f42b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:37.866 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f42b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:37.867 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f42b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:37.867 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f42b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:37.867 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f42b37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:37.867 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f42b37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:37.867 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f42b37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:37.867 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f42b2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:37.867 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f42b2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:37.867 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f42b2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:37.867 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f42b3fff640 10 monclient(hunting): handle_auth_done global_id 4139 payload 824 2024-04-26T04:18:37.867 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f42b3fff640 1 -- 172.21.15.63:0/2333918891 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42b4137e10 msgr2=0x7f42b409aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:37.867 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f42b3fff640 1 --2- 172.21.15.63:0/2333918891 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42b4137e10 0x7f42b409aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:37.867 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f42b3fff640 1 -- 172.21.15.63:0/2333918891 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f42b409fab0 msgr2=0x7f42b41344b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:37.867 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f42b3fff640 1 --2- 172.21.15.63:0/2333918891 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f42b409fab0 0x7f42b41344b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:37.868 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f42b3fff640 10 monclient: _finish_hunting 0 2024-04-26T04:18:37.868 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f42b3fff640 1 monclient: found mon.noname-c 2024-04-26T04:18:37.868 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f42b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:37.868 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f42b3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.63:3301/0 2024-04-26T04:18:37.868 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f42b3fff640 1 -- 172.21.15.63:0/2333918891 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42b4071520 con 0x7f42b413b920 2024-04-26T04:18:37.868 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f42b37fe640 1 --2- 172.21.15.63:0/2333918891 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f42b409fab0 0x7f42b41344b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T04:18:37.868 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f42b3fff640 1 -- 172.21.15.63:0/2333918891 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f42b413b920 msgr2=0x7f42b409af80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:37.868 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f42b3fff640 1 --2- 172.21.15.63:0/2333918891 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f42b413b920 0x7f42b409af80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f42ac01aca0 tx=0x7f42ac019390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:37.868 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f429bfff640 1 -- 172.21.15.63:0/2333918891 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42ac0b0810 con 0x7f42b413b920 2024-04-26T04:18:37.868 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f429bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:37.868 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f429bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:18:37.868 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f429bfff640 10 monclient: dump: 2024-04-26T04:18:37.868 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-04-26T04:18:37.868 INFO:teuthology.misc.health.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:37.869 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:37.869 INFO:teuthology.misc.health.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:37.869 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:37.869 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:37.869 INFO:teuthology.misc.health.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:37.869 INFO:teuthology.misc.health.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:37.869 INFO:teuthology.misc.health.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:37.869 INFO:teuthology.misc.health.smithi063.stderr: 2024-04-26T04:18:37.869 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f429bfff640 10 monclient: _finish_auth 0 2024-04-26T04:18:37.869 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f429bfff640 10 monclient: _check_auth_tickets 2024-04-26T04:18:37.869 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f429bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:37.869 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f429bfff640 1 -- 172.21.15.63:0/2333918891 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42ac0bf070 con 0x7f42b413b920 2024-04-26T04:18:37.869 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f429bfff640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:37.869 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f429bfff640 1 -- 172.21.15.63:0/2333918891 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42ac0b0bf0 con 0x7f42b413b920 2024-04-26T04:18:37.870 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f42b9a85640 5 monclient: authenticate success, global_id 4139 2024-04-26T04:18:37.870 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f42b9a85640 10 monclient: _renew_subs 2024-04-26T04:18:37.870 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f42b9a85640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:37.870 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f42b9a85640 1 -- 172.21.15.63:0/2333918891 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42b4071800 con 0x7f42b413b920 2024-04-26T04:18:37.870 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f429bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:37.870 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f42b1ffb640 4 set_mon_vals no callback set 2024-04-26T04:18:37.870 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f429bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:18:37.870 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f429bfff640 10 monclient: dump: 2024-04-26T04:18:37.871 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-04-26T04:18:37.871 INFO:teuthology.misc.health.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:37.871 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:37.871 INFO:teuthology.misc.health.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:37.871 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:37.871 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:37.871 INFO:teuthology.misc.health.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:37.872 INFO:teuthology.misc.health.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:37.872 INFO:teuthology.misc.health.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:37.872 INFO:teuthology.misc.health.smithi063.stderr: 2024-04-26T04:18:37.872 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f42b9a85640 10 monclient: _renew_subs 2024-04-26T04:18:37.872 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f42b9a85640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:37.872 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f42b9a85640 1 -- 172.21.15.63:0/2333918891 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f42b4071ae0 con 0x7f42b413b920 2024-04-26T04:18:37.873 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f429bfff640 1 -- 172.21.15.63:0/2333918891 <== mon.2 v2:172.21.15.63:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f42ac0c4460 con 0x7f42b413b920 2024-04-26T04:18:37.873 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.846+0000 7f429bfff640 1 -- 172.21.15.63:0/2333918891 <== mon.2 v2:172.21.15.63:3301/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f42ac0a8800 con 0x7f42b413b920 2024-04-26T04:18:37.873 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.850+0000 7f42b9a85640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:37.873 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.850+0000 7f42b9a85640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:37.873 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.850+0000 7f42b9a85640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:37.873 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.850+0000 7f42b9a85640 1 -- 172.21.15.63:0/2333918891 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42b4062e10 con 0x7f42b413b920 2024-04-26T04:18:37.874 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.850+0000 7f429bfff640 1 -- 172.21.15.63:0/2333918891 <== mon.2 v2:172.21.15.63:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f42ac0b7070 con 0x7f42b413b920 2024-04-26T04:18:37.874 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.850+0000 7f429bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:37.874 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.850+0000 7f429bfff640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:18:37.910 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-26T04:18:37.908+0000 7f2753209180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-26T04:18:37.915 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.910+0000 7f42b9a85640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-26T04:18:37.915 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.910+0000 7f42b9a85640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-26T04:18:37.915 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.910+0000 7f42b9a85640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:37.916 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.910+0000 7f42b9a85640 1 -- 172.21.15.63:0/2333918891 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f42b406ab20 con 0x7f42b413b920 2024-04-26T04:18:37.916 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.910+0000 7f429bfff640 1 -- 172.21.15.63:0/2333918891 <== mon.2 v2:172.21.15.63:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) ==== 74+0+5605 (secure 0 0 0) 0x7f42ac0ac030 con 0x7f42b413b920 2024-04-26T04:18:37.917 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.910+0000 7f429bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-26T04:18:37.917 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.910+0000 7f429bfff640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:18:37.917 INFO:teuthology.misc.health.smithi063.stdout: 2024-04-26T04:18:37.917 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":8,"fsid":"cd1c6a0c-4e4b-4014-89c0-2e535479ef39","created":"2024-04-26T04:18:34.126231+0000","modified":"2024-04-26T04:18:36.862750+0000","last_up_change":"0.000000","last_in_change":"2024-04-26T04:18:36.862750+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2d37141b-8282-4c62-864a-84a75ebf8f6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"7220e45a-72bb-4d8e-93c2-32005fa2c85e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"913de4d8-31af-4394-a0ee-194cc31f822f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c492772c-ccf5-4b85-9056-6a5ee9686c0a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"24f067f0-554a-4dd2-a26c-7bda3d0978f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"70454d09-e50c-4f65-83c5-0f9bb5c0f8f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T04:18:37.919 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.910+0000 7f42b9a85640 10 monclient: shutdown 2024-04-26T04:18:37.919 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.910+0000 7f42b9a85640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:37.919 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.910+0000 7f42b9a85640 1 -- 172.21.15.63:0/2333918891 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f42b413b920 msgr2=0x7f42b409af80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:37.919 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.910+0000 7f42b9a85640 1 --2- 172.21.15.63:0/2333918891 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f42b413b920 0x7f42b409af80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f42ac01aca0 tx=0x7f42ac019390 comp rx=0 tx=0).stop 2024-04-26T04:18:37.920 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.910+0000 7f42b9a85640 1 -- 172.21.15.63:0/2333918891 shutdown_connections 2024-04-26T04:18:37.920 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.910+0000 7f42b9a85640 1 --2- 172.21.15.63:0/2333918891 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f42b413b920 0x7f42b409af80 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:37.920 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.910+0000 7f42b9a85640 1 --2- 172.21.15.63:0/2333918891 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42b4137e10 0x7f42b409aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:37.920 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.910+0000 7f42b9a85640 1 --2- 172.21.15.63:0/2333918891 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f42b409fab0 0x7f42b41344b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:37.920 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.910+0000 7f42b9a85640 1 -- 172.21.15.63:0/2333918891 >> 172.21.15.63:0/2333918891 conn(0x7f42b406d090 msgr2=0x7f42b40a1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:37.920 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.910+0000 7f42b9a85640 1 -- 172.21.15.63:0/2333918891 shutdown_connections 2024-04-26T04:18:37.921 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:37.910+0000 7f42b9a85640 1 -- 172.21.15.63:0/2333918891 wait complete. 2024-04-26T04:18:37.933 DEBUG:teuthology.misc:0 of 6 OSDs are up 2024-04-26T04:18:37.978 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-26T04:18:37.976+0000 7f2753209180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-26T04:18:38.119 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-26T04:18:38.116+0000 7f2753209180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-26T04:18:38.738 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-26T04:18:38.736+0000 7f2753209180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-26T04:18:38.813 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-26T04:18:38.812+0000 7f2753209180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-26T04:18:38.992 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-26T04:18:38.992+0000 7f2753209180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-26T04:18:39.065 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-26T04:18:39.064+0000 7f2753209180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-26T04:18:39.348 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-26T04:18:39.348+0000 7f2753209180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-26T04:18:39.409 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-26T04:18:39.408+0000 7f2753209180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-26T04:18:40.457 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-26T04:18:40.456+0000 7f26fa751640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-26T04:18:40.462 INFO:tasks.ceph.mgr.x.smithi186.stderr:2024-04-26T04:18:40.460+0000 7f26fa751640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-26T04:18:41.289 INFO:tasks.ceph.osd.0.smithi063.stderr:2024-04-26T04:18:41.282+0000 7fba62dbe7c0 -1 osd.0 0 log_to_monitors true 2024-04-26T04:18:41.319 INFO:tasks.ceph.osd.3.smithi186.stderr:2024-04-26T04:18:41.316+0000 7f79351487c0 -1 osd.3 0 log_to_monitors true 2024-04-26T04:18:41.376 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-04-26T04:18:41.376+0000 7f1a6f9527c0 -1 osd.4 0 log_to_monitors true 2024-04-26T04:18:41.428 INFO:tasks.ceph.osd.1.smithi063.stderr:2024-04-26T04:18:41.422+0000 7f603ef997c0 -1 osd.1 0 log_to_monitors true 2024-04-26T04:18:41.435 INFO:tasks.ceph.osd.2.smithi063.stderr:2024-04-26T04:18:41.430+0000 7f131e3f97c0 -1 osd.2 0 log_to_monitors true 2024-04-26T04:18:41.439 INFO:tasks.ceph.osd.5.smithi186.stderr:2024-04-26T04:18:41.440+0000 7ff640d3d7c0 -1 osd.5 0 log_to_monitors true 2024-04-26T04:18:42.471 INFO:tasks.ceph.osd.3.smithi186.stderr:2024-04-26T04:18:42.468+0000 7f79308ce640 -1 osd.3 0 waiting for initial osdmap 2024-04-26T04:18:42.472 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-04-26T04:18:42.472+0000 7f1a6b0d8640 -1 osd.4 0 waiting for initial osdmap 2024-04-26T04:18:42.489 INFO:tasks.ceph.osd.3.smithi186.stderr:2024-04-26T04:18:42.488+0000 7f792cf1d640 -1 osd.3 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T04:18:42.490 INFO:tasks.ceph.osd.4.smithi186.stderr:2024-04-26T04:18:42.488+0000 7f1a67727640 -1 osd.4 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T04:18:43.269 INFO:tasks.ceph.osd.5.smithi186.stderr:2024-04-26T04:18:43.268+0000 7ff63c4c3640 -1 osd.5 0 waiting for initial osdmap 2024-04-26T04:18:43.289 INFO:tasks.ceph.osd.5.smithi186.stderr:2024-04-26T04:18:43.288+0000 7ff634b0a640 -1 osd.5 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T04:18:44.089 INFO:tasks.ceph.osd.1.smithi063.stderr:2024-04-26T04:18:44.082+0000 7f603a71f640 -1 osd.1 0 waiting for initial osdmap 2024-04-26T04:18:44.114 INFO:tasks.ceph.osd.1.smithi063.stderr:2024-04-26T04:18:44.110+0000 7f6032d66640 -1 osd.1 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T04:18:44.119 INFO:tasks.ceph.osd.0.smithi063.stderr:2024-04-26T04:18:44.114+0000 7fba5e544640 -1 osd.0 0 waiting for initial osdmap 2024-04-26T04:18:44.154 INFO:tasks.ceph.osd.0.smithi063.stderr:2024-04-26T04:18:44.150+0000 7fba56b8b640 -1 osd.0 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T04:18:44.169 INFO:tasks.ceph.osd.2.smithi063.stderr:2024-04-26T04:18:44.162+0000 7f1319b7f640 -1 osd.2 0 waiting for initial osdmap 2024-04-26T04:18:44.211 INFO:tasks.ceph.osd.2.smithi063.stderr:2024-04-26T04:18:44.206+0000 7f13121c6640 -1 osd.2 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-26T04:18:44.539 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-26T04:18:44.661 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.654+0000 7f74297da640 10 monclient: get_monmap_and_config 2024-04-26T04:18:44.662 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.654+0000 7f74297da640 10 monclient: build_initial_monmap 2024-04-26T04:18:44.662 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.654+0000 7f74297da640 10 monclient: monmap: 2024-04-26T04:18:44.662 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-04-26T04:18:44.662 INFO:teuthology.misc.health.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:44.662 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-04-26T04:18:44.661675+0000 2024-04-26T04:18:44.662 INFO:teuthology.misc.health.smithi063.stderr:created 2024-04-26T04:18:44.661675+0000 2024-04-26T04:18:44.662 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:44.662 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:44.662 INFO:teuthology.misc.health.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:18:44.662 INFO:teuthology.misc.health.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:18:44.662 INFO:teuthology.misc.health.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:18:44.662 INFO:teuthology.misc.health.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:18:44.662 INFO:teuthology.misc.health.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:18:44.663 INFO:teuthology.misc.health.smithi063.stderr: 2024-04-26T04:18:44.665 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.658+0000 7f74297da640 1 Processor -- start 2024-04-26T04:18:44.665 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.658+0000 7f74297da640 1 -- start start 2024-04-26T04:18:44.665 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.658+0000 7f74297da640 10 monclient: init 2024-04-26T04:18:44.665 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.658+0000 7f74297da640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:44.665 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.658+0000 7f74297da640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-04-26T04:18:44.665 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.658+0000 7f74297da640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f74240a4040 addr v1:172.21.15.186:6789/0 2024-04-26T04:18:44.665 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.658+0000 7f74297da640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f74240a4b30 0x7f742409bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:44.665 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.658+0000 7f74297da640 10 monclient(hunting): picked mon.noname-a con 0x7f74240a4b30 addr v2:172.21.15.63:3300/0 2024-04-26T04:18:44.665 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.658+0000 7f74297da640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f742413c330 0x7f742413e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:44.666 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.658+0000 7f74297da640 10 monclient(hunting): picked mon.noname-b con 0x7f742413c330 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:44.666 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.658+0000 7f74297da640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74240a51e0 con 0x7f74240a4040 2024-04-26T04:18:44.666 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.658+0000 7f74297da640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:44.666 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.658+0000 7f74297da640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7f74240a5f00 con 0x7f74240a4b30 2024-04-26T04:18:44.666 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.658+0000 7f74297da640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:44.666 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.658+0000 7f74297da640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f742409c470 con 0x7f742413c330 2024-04-26T04:18:44.666 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.658+0000 7f74297da640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:44.666 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.658+0000 7f74297da640 10 monclient(hunting): authenticate will time out at 956.369411s 2024-04-26T04:18:44.666 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.658+0000 7f74227fc640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f74240a4b30 0x7f742409bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:44.666 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.658+0000 7f74237fe640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f742413c330 0x7f742413e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:44.667 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.658+0000 7f74227fc640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f74240a4b30 0x7f742409bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:46314/0 (socket says 172.21.15.63:46314) 2024-04-26T04:18:44.667 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.658+0000 7f74227fc640 1 -- 172.21.15.63:0/2057839929 learned_addr learned my addr 172.21.15.63:0/2057839929 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:18:44.667 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.658+0000 7f74227fc640 10 monclient(hunting): get_auth_request con 0x7f74240a4b30 auth_method 0 2024-04-26T04:18:44.667 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.658+0000 7f74227fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:44.667 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.658+0000 7f74227fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:44.667 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.658+0000 7f74227fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:44.668 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.658+0000 7f74237fe640 10 monclient(hunting): get_auth_request con 0x7f742413c330 auth_method 0 2024-04-26T04:18:44.668 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.658+0000 7f74237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:44.668 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.658+0000 7f74237fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:44.668 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.658+0000 7f74237fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:44.668 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.658+0000 7f74227fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:44.668 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.658+0000 7f74227fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:44.668 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.658+0000 7f74227fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:44.668 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.658+0000 7f74237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:44.668 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.658+0000 7f74237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:44.668 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.658+0000 7f74237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:44.669 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.658+0000 7f74227fc640 10 monclient(hunting): handle_auth_done global_id 4176 payload 274 2024-04-26T04:18:44.669 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.658+0000 7f74227fc640 1 -- 172.21.15.63:0/2057839929 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f742413c330 msgr2=0x7f742413e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:44.669 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.658+0000 7f74227fc640 1 --2- 172.21.15.63:0/2057839929 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f742413c330 0x7f742413e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:44.669 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.658+0000 7f74227fc640 1 -- 172.21.15.63:0/2057839929 >> v1:172.21.15.186:6789/0 conn(0x7f74240a4040 legacy=0x7f74240a4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:44.669 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.658+0000 7f74227fc640 10 monclient: _finish_hunting 0 2024-04-26T04:18:44.669 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.658+0000 7f74227fc640 1 monclient: found mon.noname-a 2024-04-26T04:18:44.669 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.658+0000 7f74227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:44.669 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.658+0000 7f74227fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:18:44.669 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.658+0000 7f74227fc640 1 -- 172.21.15.63:0/2057839929 --> v2:172.21.15.63:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f742409b270 con 0x7f74240a4b30 2024-04-26T04:18:44.674 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.662+0000 7f74227fc640 1 -- 172.21.15.63:0/2057839929 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74240a4b30 msgr2=0x7f742409bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:44.675 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.662+0000 7f74227fc640 1 --2- 172.21.15.63:0/2057839929 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74240a4b30 0x7f742409bf30 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7414009870 tx=0x7f741401c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff048e0d8c0611c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:44.675 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.662+0000 7f7421ffb640 1 -- 172.21.15.63:0/2057839929 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74140b0070 con 0x7f74240a4b30 2024-04-26T04:18:44.675 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.662+0000 7f7421ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:44.675 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.662+0000 7f7421ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:18:44.675 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.662+0000 7f7421ffb640 10 monclient: dump: 2024-04-26T04:18:44.675 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-04-26T04:18:44.675 INFO:teuthology.misc.health.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:44.675 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:44.675 INFO:teuthology.misc.health.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:44.675 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:44.675 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:44.675 INFO:teuthology.misc.health.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:44.675 INFO:teuthology.misc.health.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:44.675 INFO:teuthology.misc.health.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:44.675 INFO:teuthology.misc.health.smithi063.stderr: 2024-04-26T04:18:44.676 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.662+0000 7f7421ffb640 10 monclient: _finish_auth 0 2024-04-26T04:18:44.676 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.662+0000 7f7421ffb640 10 monclient: _check_auth_tickets 2024-04-26T04:18:44.676 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.662+0000 7f7421ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:44.676 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.662+0000 7f7421ffb640 1 -- 172.21.15.63:0/2057839929 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74140ab600 con 0x7f74240a4b30 2024-04-26T04:18:44.676 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.662+0000 7f7421ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:44.676 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.662+0000 7f7421ffb640 1 -- 172.21.15.63:0/2057839929 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74140ab9d0 con 0x7f74240a4b30 2024-04-26T04:18:44.676 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.662+0000 7f74297da640 5 monclient: authenticate success, global_id 4176 2024-04-26T04:18:44.676 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.662+0000 7f74297da640 10 monclient: get_monmap_and_config success 2024-04-26T04:18:44.676 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.662+0000 7f74297da640 4 set_mon_vals no callback set 2024-04-26T04:18:44.676 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.662+0000 7f7421ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:44.676 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.662+0000 7f7421ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:18:44.676 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.662+0000 7f7421ffb640 10 monclient: dump: 2024-04-26T04:18:44.676 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-04-26T04:18:44.676 INFO:teuthology.misc.health.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:44.677 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:44.677 INFO:teuthology.misc.health.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:44.677 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:44.677 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:44.677 INFO:teuthology.misc.health.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:44.677 INFO:teuthology.misc.health.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:44.677 INFO:teuthology.misc.health.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:44.677 INFO:teuthology.misc.health.smithi063.stderr: 2024-04-26T04:18:44.677 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.662+0000 7f74297da640 10 monclient: shutdown 2024-04-26T04:18:44.677 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.662+0000 7f74297da640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:44.677 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.662+0000 7f74297da640 1 -- 172.21.15.63:0/2057839929 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74240a4b30 msgr2=0x7f742409bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:44.677 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.662+0000 7f74297da640 1 --2- 172.21.15.63:0/2057839929 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74240a4b30 0x7f742409bf30 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7414009870 tx=0x7f741401c3e0 comp rx=0 tx=0).stop 2024-04-26T04:18:44.677 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.662+0000 7f74297da640 1 -- 172.21.15.63:0/2057839929 shutdown_connections 2024-04-26T04:18:44.677 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.662+0000 7f74297da640 1 --2- 172.21.15.63:0/2057839929 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f742413c330 0x7f742413e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:44.677 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.662+0000 7f74297da640 1 --2- 172.21.15.63:0/2057839929 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74240a4b30 0x7f742409bf30 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:44.678 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.662+0000 7f74297da640 1 -- 172.21.15.63:0/2057839929 >> 172.21.15.63:0/2057839929 conn(0x7f74240a6140 msgr2=0x7f74240a6550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:44.678 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.662+0000 7f74297da640 1 -- 172.21.15.63:0/2057839929 shutdown_connections 2024-04-26T04:18:44.678 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.662+0000 7f74297da640 1 -- 172.21.15.63:0/2057839929 wait complete. 2024-04-26T04:18:44.678 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.662+0000 7f74297da640 10 monclient: build_initial_monmap 2024-04-26T04:18:44.678 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.662+0000 7f74297da640 10 monclient: monmap: 2024-04-26T04:18:44.678 INFO:teuthology.misc.health.smithi063.stderr:epoch 0 2024-04-26T04:18:44.678 INFO:teuthology.misc.health.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:18:44.678 INFO:teuthology.misc.health.smithi063.stderr:last_changed 0.000000 2024-04-26T04:18:44.678 INFO:teuthology.misc.health.smithi063.stderr:created 0.000000 2024-04-26T04:18:44.678 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:44.678 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:44.678 INFO:teuthology.misc.health.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:18:44.678 INFO:teuthology.misc.health.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:18:44.678 INFO:teuthology.misc.health.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:18:44.679 INFO:teuthology.misc.health.smithi063.stderr: 2024-04-26T04:18:44.679 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.662+0000 7f74297da640 1 Processor -- start 2024-04-26T04:18:44.679 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.662+0000 7f74297da640 1 -- start start 2024-04-26T04:18:44.679 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.662+0000 7f74297da640 10 monclient: init 2024-04-26T04:18:44.679 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f74297da640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:44.679 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f74297da640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T04:18:44.679 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f74297da640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74240a4040 0x7f7424136ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:44.679 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f74297da640 10 monclient(hunting): picked mon.noname-b con 0x7f74240a4040 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:18:44.679 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f74297da640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f74240a4b30 0x7f7424137020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:44.679 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f74297da640 10 monclient(hunting): picked mon.noname-c con 0x7f74240a4b30 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:44.679 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f74297da640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f742413c330 0x7f742413a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:44.679 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f74297da640 10 monclient(hunting): picked mon.noname-a con 0x7f742413c330 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:18:44.679 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f74297da640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:44.679 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f74297da640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f7424141600 con 0x7f742413c330 2024-04-26T04:18:44.679 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f74297da640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:44.680 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f74297da640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7424141480 con 0x7f74240a4040 2024-04-26T04:18:44.680 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f74297da640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:44.680 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f74297da640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f7424141780 con 0x7f74240a4b30 2024-04-26T04:18:44.680 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f74297da640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:44.680 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f74297da640 10 monclient(hunting): authenticate will time out at 956.376377s 2024-04-26T04:18:44.680 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f74227fc640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f74240a4b30 0x7f7424137020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:44.680 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f74237fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f742413c330 0x7f742413a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:44.680 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f74227fc640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f74240a4b30 0x7f7424137020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:57692/0 (socket says 172.21.15.63:57692) 2024-04-26T04:18:44.680 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f74227fc640 1 -- 172.21.15.63:0/143105438 learned_addr learned my addr 172.21.15.63:0/143105438 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:18:44.680 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f74227fc640 10 monclient(hunting): get_auth_request con 0x7f74240a4b30 auth_method 0 2024-04-26T04:18:44.680 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f74227fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:44.680 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f74227fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:44.680 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f74227fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:44.680 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f74237fe640 10 monclient(hunting): get_auth_request con 0x7f742413c330 auth_method 0 2024-04-26T04:18:44.680 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f74237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:44.681 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f74237fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:44.681 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f74237fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:44.681 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f74227fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:44.681 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f74227fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:44.681 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f74227fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:44.681 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f74237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:44.681 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f74237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:44.681 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f74237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:44.681 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f7422ffd640 1 --2- 172.21.15.63:0/143105438 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74240a4040 0x7f7424136ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:44.681 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f74227fc640 10 monclient(hunting): handle_auth_done global_id 4163 payload 824 2024-04-26T04:18:44.681 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f74227fc640 1 -- 172.21.15.63:0/143105438 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74240a4040 msgr2=0x7f7424136ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:44.681 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f74227fc640 1 --2- 172.21.15.63:0/143105438 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74240a4040 0x7f7424136ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:44.681 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f74227fc640 1 -- 172.21.15.63:0/143105438 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f742413c330 msgr2=0x7f742413a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:44.681 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f74227fc640 1 --2- 172.21.15.63:0/143105438 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f742413c330 0x7f742413a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:44.681 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f74227fc640 10 monclient: _finish_hunting 0 2024-04-26T04:18:44.682 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f74227fc640 1 monclient: found mon.noname-c 2024-04-26T04:18:44.682 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f74227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:44.682 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f74227fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.63:3301/0 2024-04-26T04:18:44.682 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f74227fc640 1 -- 172.21.15.63:0/143105438 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f742413c0b0 con 0x7f74240a4b30 2024-04-26T04:18:44.682 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f7422ffd640 10 monclient: get_auth_request con 0x7f74240a4040 auth_method 0 2024-04-26T04:18:44.682 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f7422ffd640 1 --2- 172.21.15.63:0/143105438 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74240a4040 0x7f7424136ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T04:18:44.682 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f74237fe640 1 --2- 172.21.15.63:0/143105438 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f742413c330 0x7f742413a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T04:18:44.682 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f74227fc640 1 -- 172.21.15.63:0/143105438 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f74240a4b30 msgr2=0x7f7424137020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:44.682 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f74227fc640 1 --2- 172.21.15.63:0/143105438 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f74240a4b30 0x7f7424137020 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f74140ae040 tx=0x7f741401ac20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:44.682 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f740b7fe640 1 -- 172.21.15.63:0/143105438 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74140ba020 con 0x7f74240a4b30 2024-04-26T04:18:44.682 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f740b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:44.682 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f740b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:18:44.682 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f740b7fe640 10 monclient: dump: 2024-04-26T04:18:44.683 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-04-26T04:18:44.683 INFO:teuthology.misc.health.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:44.683 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:44.683 INFO:teuthology.misc.health.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:44.683 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:44.683 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:44.683 INFO:teuthology.misc.health.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:44.683 INFO:teuthology.misc.health.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:44.683 INFO:teuthology.misc.health.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:44.683 INFO:teuthology.misc.health.smithi063.stderr: 2024-04-26T04:18:44.683 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f740b7fe640 10 monclient: _finish_auth 0 2024-04-26T04:18:44.683 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f740b7fe640 10 monclient: _check_auth_tickets 2024-04-26T04:18:44.683 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f740b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:44.683 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f740b7fe640 1 -- 172.21.15.63:0/143105438 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74140abe50 con 0x7f74240a4b30 2024-04-26T04:18:44.683 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f740b7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:44.683 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f740b7fe640 1 -- 172.21.15.63:0/143105438 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74140b0040 con 0x7f74240a4b30 2024-04-26T04:18:44.684 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f740b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:44.684 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f740b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:18:44.684 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f740b7fe640 10 monclient: dump: 2024-04-26T04:18:44.684 INFO:teuthology.misc.health.smithi063.stderr:epoch 1 2024-04-26T04:18:44.684 INFO:teuthology.misc.health.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:44.684 INFO:teuthology.misc.health.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:44.684 INFO:teuthology.misc.health.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:44.684 INFO:teuthology.misc.health.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:44.684 INFO:teuthology.misc.health.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:44.684 INFO:teuthology.misc.health.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:44.684 INFO:teuthology.misc.health.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:44.684 INFO:teuthology.misc.health.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:44.684 INFO:teuthology.misc.health.smithi063.stderr: 2024-04-26T04:18:44.684 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f74217fa640 4 set_mon_vals no callback set 2024-04-26T04:18:44.684 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f74297da640 5 monclient: authenticate success, global_id 4163 2024-04-26T04:18:44.685 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f74297da640 10 monclient: _renew_subs 2024-04-26T04:18:44.685 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f74297da640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:44.685 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f74297da640 1 -- 172.21.15.63:0/143105438 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f742413bc70 con 0x7f74240a4b30 2024-04-26T04:18:44.685 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f74297da640 10 monclient: _renew_subs 2024-04-26T04:18:44.685 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f74297da640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:44.685 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.666+0000 7f74297da640 1 -- 172.21.15.63:0/143105438 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7424120350 con 0x7f74240a4b30 2024-04-26T04:18:44.685 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.670+0000 7f740b7fe640 1 -- 172.21.15.63:0/143105438 <== mon.2 v2:172.21.15.63:3301/0 4 ==== mgrmap(e 4) ==== 43997+0+0 (secure 0 0 0) 0x7f74140bd020 con 0x7f74240a4b30 2024-04-26T04:18:44.685 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.670+0000 7f74297da640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:44.685 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.670+0000 7f74297da640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:44.685 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.670+0000 7f74297da640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:44.685 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.670+0000 7f74297da640 1 -- 172.21.15.63:0/143105438 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7424062e10 con 0x7f74240a4b30 2024-04-26T04:18:44.685 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.670+0000 7f740b7fe640 1 --2- 172.21.15.63:0/143105438 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f7410035e80 0x7f7410038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:44.685 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.670+0000 7f740b7fe640 1 -- 172.21.15.63:0/143105438 <== mon.2 v2:172.21.15.63:3301/0 5 ==== osd_map(12..12 src has 1..12) ==== 3982+0+0 (secure 0 0 0) 0x7f74140e2910 con 0x7f74240a4b30 2024-04-26T04:18:44.685 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.670+0000 7f7422ffd640 1 --2- 172.21.15.63:0/143105438 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f7410035e80 0x7f7410038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:44.685 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.670+0000 7f7422ffd640 10 monclient: get_auth_request con 0x7f7410035e80 auth_method 0 2024-04-26T04:18:44.686 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.674+0000 7f7422ffd640 1 --2- 172.21.15.63:0/143105438 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f7410035e80 0x7f7410038340 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7418003e40 tx=0x7f7418009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:44.686 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.674+0000 7f740b7fe640 1 -- 172.21.15.63:0/143105438 <== mon.2 v2:172.21.15.63:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f74140c4030 con 0x7f74240a4b30 2024-04-26T04:18:44.686 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.674+0000 7f740b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:44.686 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.674+0000 7f740b7fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:18:44.842 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.834+0000 7f74297da640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-26T04:18:44.842 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.838+0000 7f74297da640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-26T04:18:44.842 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.838+0000 7f74297da640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:44.842 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.838+0000 7f74297da640 1 -- 172.21.15.63:0/143105438 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f742413e5a0 con 0x7f74240a4b30 2024-04-26T04:18:44.842 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.838+0000 7f740b7fe640 1 -- 172.21.15.63:0/143105438 <== mon.2 v2:172.21.15.63:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) ==== 74+0+10408 (secure 0 0 0) 0x7f74140160f0 con 0x7f74240a4b30 2024-04-26T04:18:44.843 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.838+0000 7f740b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-26T04:18:44.843 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.838+0000 7f740b7fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:18:44.843 INFO:teuthology.misc.health.smithi063.stdout: 2024-04-26T04:18:44.843 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":12,"fsid":"cd1c6a0c-4e4b-4014-89c0-2e535479ef39","created":"2024-04-26T04:18:34.126231+0000","modified":"2024-04-26T04:18:44.470061+0000","last_up_change":"2024-04-26T04:18:44.470061+0000","last_in_change":"2024-04-26T04:18:36.862750+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T04:18:43.521296+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d37141b-8282-4c62-864a-84a75ebf8f6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":2758152700},{"type":"v1","addr":"172.21.15.63:6801","nonce":2758152700}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":2758152700},{"type":"v1","addr":"172.21.15.63:6803","nonce":2758152700}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":2758152700},{"type":"v1","addr":"172.21.15.63:6807","nonce":2758152700}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":2758152700},{"type":"v1","addr":"172.21.15.63:6805","nonce":2758152700}]},"public_addr":"172.21.15.63:6801/2758152700","cluster_addr":"172.21.15.63:6803/2758152700","heartbeat_back_addr":"172.21.15.63:6807/2758152700","heartbeat_front_addr":"172.21.15.63:6805/2758152700","state":["exists","up"]},{"osd":1,"uuid":"7220e45a-72bb-4d8e-93c2-32005fa2c85e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":3596874741},{"type":"v1","addr":"172.21.15.63:6817","nonce":3596874741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":3596874741},{"type":"v1","addr":"172.21.15.63:6819","nonce":3596874741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":3596874741},{"type":"v1","addr":"172.21.15.63:6823","nonce":3596874741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":3596874741},{"type":"v1","addr":"172.21.15.63:6821","nonce":3596874741}]},"public_addr":"172.21.15.63:6817/3596874741","cluster_addr":"172.21.15.63:6819/3596874741","heartbeat_back_addr":"172.21.15.63:6823/3596874741","heartbeat_front_addr":"172.21.15.63:6821/3596874741","state":["exists","up"]},{"osd":2,"uuid":"913de4d8-31af-4394-a0ee-194cc31f822f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":413829082},{"type":"v1","addr":"172.21.15.63:6809","nonce":413829082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":413829082},{"type":"v1","addr":"172.21.15.63:6811","nonce":413829082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":413829082},{"type":"v1","addr":"172.21.15.63:6815","nonce":413829082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":413829082},{"type":"v1","addr":"172.21.15.63:6813","nonce":413829082}]},"public_addr":"172.21.15.63:6809/413829082","cluster_addr":"172.21.15.63:6811/413829082","heartbeat_back_addr":"172.21.15.63:6815/413829082","heartbeat_front_addr":"172.21.15.63:6813/413829082","state":["exists","up"]},{"osd":3,"uuid":"c492772c-ccf5-4b85-9056-6a5ee9686c0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6807","nonce":4227496981},{"type":"v1","addr":"172.21.15.186:6808","nonce":4227496981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4227496981},{"type":"v1","addr":"172.21.15.186:6811","nonce":4227496981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4227496981},{"type":"v1","addr":"172.21.15.186:6815","nonce":4227496981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4227496981},{"type":"v1","addr":"172.21.15.186:6813","nonce":4227496981}]},"public_addr":"172.21.15.186:6808/4227496981","cluster_addr":"172.21.15.186:6811/4227496981","heartbeat_back_addr":"172.21.15.186:6815/4227496981","heartbeat_front_addr":"172.21.15.186:6813/4227496981","state":["exists","up"]},{"osd":4,"uuid":"24f067f0-554a-4dd2-a26c-7bda3d0978f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":3746743567},{"type":"v1","addr":"172.21.15.186:6801","nonce":3746743567}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":3746743567},{"type":"v1","addr":"172.21.15.186:6803","nonce":3746743567}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":3746743567},{"type":"v1","addr":"172.21.15.186:6809","nonce":3746743567}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":3746743567},{"type":"v1","addr":"172.21.15.186:6805","nonce":3746743567}]},"public_addr":"172.21.15.186:6801/3746743567","cluster_addr":"172.21.15.186:6803/3746743567","heartbeat_back_addr":"172.21.15.186:6809/3746743567","heartbeat_front_addr":"172.21.15.186:6805/3746743567","state":["exists","up"]},{"osd":5,"uuid":"70454d09-e50c-4f65-83c5-0f9bb5c0f8f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":3109931567},{"type":"v1","addr":"172.21.15.186:6817","nonce":3109931567}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":3109931567},{"type":"v1","addr":"172.21.15.186:6819","nonce":3109931567}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":3109931567},{"type":"v1","addr":"172.21.15.186:6823","nonce":3109931567}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":3109931567},{"type":"v1","addr":"172.21.15.186:6821","nonce":3109931567}]},"public_addr":"172.21.15.186:6817/3109931567","cluster_addr":"172.21.15.186:6819/3109931567","heartbeat_back_addr":"172.21.15.186:6823/3109931567","heartbeat_front_addr":"172.21.15.186:6821/3109931567","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T04:18:42.367922+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T04:18:42.357914+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T04:18:42.415215+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T04:18:44.845 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.838+0000 7f74297da640 1 -- 172.21.15.63:0/143105438 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f7410035e80 msgr2=0x7f7410038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:44.845 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.838+0000 7f74297da640 1 --2- 172.21.15.63:0/143105438 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f7410035e80 0x7f7410038340 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7418003e40 tx=0x7f7418009040 comp rx=0 tx=0).stop 2024-04-26T04:18:44.845 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.838+0000 7f74297da640 10 monclient: shutdown 2024-04-26T04:18:44.845 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.838+0000 7f74297da640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:44.845 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.838+0000 7f74297da640 1 -- 172.21.15.63:0/143105438 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f74240a4b30 msgr2=0x7f7424137020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:44.845 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.838+0000 7f74297da640 1 --2- 172.21.15.63:0/143105438 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f74240a4b30 0x7f7424137020 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f74140ae040 tx=0x7f741401ac20 comp rx=0 tx=0).stop 2024-04-26T04:18:44.845 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.838+0000 7f74297da640 1 -- 172.21.15.63:0/143105438 shutdown_connections 2024-04-26T04:18:44.845 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.838+0000 7f74297da640 1 --2- 172.21.15.63:0/143105438 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f7410035e80 0x7f7410038340 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:44.845 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.838+0000 7f74297da640 1 --2- 172.21.15.63:0/143105438 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f742413c330 0x7f742413a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:44.846 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.838+0000 7f74297da640 1 --2- 172.21.15.63:0/143105438 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f74240a4b30 0x7f7424137020 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:44.846 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.838+0000 7f74297da640 1 --2- 172.21.15.63:0/143105438 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74240a4040 0x7f7424136ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:44.846 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.838+0000 7f74297da640 1 -- 172.21.15.63:0/143105438 >> 172.21.15.63:0/143105438 conn(0x7f74240a6140 msgr2=0x7f74240a2ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:44.846 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.838+0000 7f74297da640 1 -- 172.21.15.63:0/143105438 shutdown_connections 2024-04-26T04:18:44.846 INFO:teuthology.misc.health.smithi063.stderr:2024-04-26T04:18:44.838+0000 7f74297da640 1 -- 172.21.15.63:0/143105438 wait complete. 2024-04-26T04:18:44.860 DEBUG:teuthology.misc:6 of 6 OSDs are up 2024-04-26T04:18:44.861 INFO:tasks.ceph:Creating RBD pool 2024-04-26T04:18:44.861 DEBUG:teuthology.orchestra.run.smithi063:> sudo ceph --cluster ceph osd pool create rbd 8 2024-04-26T04:18:44.989 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.982+0000 7f2cd10ed640 10 monclient: get_monmap_and_config 2024-04-26T04:18:44.989 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.982+0000 7f2cd10ed640 10 monclient: build_initial_monmap 2024-04-26T04:18:44.989 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.982+0000 7f2cd10ed640 10 monclient: monmap: 2024-04-26T04:18:44.989 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:44.989 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:44.989 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:44.989318+0000 2024-04-26T04:18:44.990 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:44.989318+0000 2024-04-26T04:18:44.990 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:44.990 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:44.990 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:18:44.990 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:18:44.990 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:18:44.990 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:18:44.990 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:18:44.990 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:44.992 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.986+0000 7f2cd10ed640 1 Processor -- start 2024-04-26T04:18:44.992 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.986+0000 7f2cd10ed640 1 -- start start 2024-04-26T04:18:44.992 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.986+0000 7f2cd10ed640 10 monclient: init 2024-04-26T04:18:44.992 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.986+0000 7f2cd10ed640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:44.992 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.986+0000 7f2cd10ed640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-04-26T04:18:44.993 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.986+0000 7f2cd10ed640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2ccc0a1bf0 addr v1:172.21.15.63:6789/0 2024-04-26T04:18:44.993 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.986+0000 7f2cd10ed640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2ccc0a1fd0 addr v1:172.21.15.186:6789/0 2024-04-26T04:18:44.993 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.986+0000 7f2cd10ed640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f2ccc13bd30 0x7f2ccc13e120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:44.993 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.986+0000 7f2cd10ed640 10 monclient(hunting): picked mon.noname-c con 0x7f2ccc13bd30 addr v2:172.21.15.186:3300/0 2024-04-26T04:18:44.993 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.986+0000 7f2cd10ed640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ccc1357d0 con 0x7f2ccc0a1bf0 2024-04-26T04:18:44.993 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.986+0000 7f2cd10ed640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ccc06eba0 con 0x7f2ccc0a1fd0 2024-04-26T04:18:44.993 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.986+0000 7f2cd10ed640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:44.993 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.986+0000 7f2cd10ed640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f2ccc0a1510 con 0x7f2ccc13bd30 2024-04-26T04:18:44.993 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.986+0000 7f2cd10ed640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:44.993 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.986+0000 7f2cd10ed640 10 monclient(hunting): authenticate will time out at 956.696988s 2024-04-26T04:18:44.993 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.986+0000 7f2ccad9d640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f2ccc0a1bf0 0x7f2ccc1349b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:44098/0 (socket says 172.21.15.63:44098) 2024-04-26T04:18:44.993 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.986+0000 7f2ccad9d640 1 -- 172.21.15.63:0/1058316052 learned_addr learned my addr 172.21.15.63:0/1058316052 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-04-26T04:18:44.993 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.986+0000 7f2ccb59e640 1 --2- 172.21.15.63:0/1058316052 >> v2:172.21.15.186:3300/0 conn(0x7f2ccc13bd30 0x7f2ccc13e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:44.993 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.986+0000 7f2ccb59e640 10 monclient(hunting): get_auth_request con 0x7f2ccc13bd30 auth_method 0 2024-04-26T04:18:44.993 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.986+0000 7f2ccb59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:44.994 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.986+0000 7f2ccb59e640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:44.994 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.986+0000 7f2ccb59e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:44.994 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.986+0000 7f2ccb59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:44.994 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.986+0000 7f2ccb59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:44.994 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.986+0000 7f2ccb59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:44.994 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.986+0000 7f2ccb59e640 10 monclient(hunting): handle_auth_done global_id 4192 payload 274 2024-04-26T04:18:44.994 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.986+0000 7f2ccb59e640 1 -- 172.21.15.63:0/1058316052 >> v1:172.21.15.186:6789/0 conn(0x7f2ccc0a1fd0 legacy=0x7f2ccc1350c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:44.994 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.986+0000 7f2ccb59e640 1 -- 172.21.15.63:0/1058316052 >> v1:172.21.15.63:6789/0 conn(0x7f2ccc0a1bf0 legacy=0x7f2ccc1349b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:44.994 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.986+0000 7f2ccb59e640 10 monclient: _finish_hunting 0 2024-04-26T04:18:44.994 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.986+0000 7f2ccb59e640 1 monclient: found mon.noname-c 2024-04-26T04:18:44.994 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.986+0000 7f2ccb59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:44.994 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.986+0000 7f2ccb59e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.186:3300/0 2024-04-26T04:18:44.994 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.986+0000 7f2ccb59e640 1 -- 172.21.15.63:0/1058316052 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ccc0717a0 con 0x7f2ccc13bd30 2024-04-26T04:18:44.996 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.990+0000 7f2ccb59e640 1 -- 172.21.15.63:0/1058316052 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ccc13bd30 msgr2=0x7f2ccc13e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:44.996 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.990+0000 7f2ccb59e640 1 --2- 172.21.15.63:0/1058316052 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ccc13bd30 0x7f2ccc13e120 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2cc4009d90 tx=0x7f2cc401c7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=faa9d440dcb8710b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:44.997 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.990+0000 7f2cca59c640 1 -- 172.21.15.63:0/1058316052 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cc40b2070 con 0x7f2ccc13bd30 2024-04-26T04:18:44.997 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.990+0000 7f2cca59c640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:44.997 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.990+0000 7f2cca59c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:18:44.997 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.990+0000 7f2cca59c640 10 monclient: dump: 2024-04-26T04:18:44.997 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:44.997 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:44.997 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:44.997 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:44.997 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:44.997 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:44.997 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:44.998 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:44.998 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:44.998 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:44.998 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.990+0000 7f2cca59c640 10 monclient: _finish_auth 0 2024-04-26T04:18:44.998 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.990+0000 7f2cca59c640 10 monclient: _check_auth_tickets 2024-04-26T04:18:44.998 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.990+0000 7f2cca59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:44.998 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.990+0000 7f2cca59c640 1 -- 172.21.15.63:0/1058316052 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2cc40a5840 con 0x7f2ccc13bd30 2024-04-26T04:18:44.998 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.990+0000 7f2cca59c640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:44.998 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.990+0000 7f2cca59c640 1 -- 172.21.15.63:0/1058316052 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cc40a5c10 con 0x7f2ccc13bd30 2024-04-26T04:18:44.998 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.990+0000 7f2cca59c640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:44.998 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.990+0000 7f2cca59c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T04:18:44.998 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.990+0000 7f2cca59c640 10 monclient: dump: 2024-04-26T04:18:44.998 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:44.998 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:44.999 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:44.999 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:44.999 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:44.999 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:44.999 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:44.999 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:44.999 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:44.999 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:44.999 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.990+0000 7f2cd10ed640 5 monclient: authenticate success, global_id 4192 2024-04-26T04:18:44.999 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.990+0000 7f2cd10ed640 10 monclient: get_monmap_and_config success 2024-04-26T04:18:44.999 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.990+0000 7f2cd10ed640 4 set_mon_vals no callback set 2024-04-26T04:18:45.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.994+0000 7f2cd10ed640 10 monclient: shutdown 2024-04-26T04:18:45.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.994+0000 7f2cd10ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:45.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.994+0000 7f2cd10ed640 1 -- 172.21.15.63:0/1058316052 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ccc13bd30 msgr2=0x7f2ccc13e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:45.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.994+0000 7f2cd10ed640 1 --2- 172.21.15.63:0/1058316052 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ccc13bd30 0x7f2ccc13e120 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2cc4009d90 tx=0x7f2cc401c7b0 comp rx=0 tx=0).stop 2024-04-26T04:18:45.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.994+0000 7f2cd10ed640 1 -- 172.21.15.63:0/1058316052 shutdown_connections 2024-04-26T04:18:45.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.994+0000 7f2cd10ed640 1 --2- 172.21.15.63:0/1058316052 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ccc13bd30 0x7f2ccc13e120 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:45.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.994+0000 7f2cd10ed640 1 -- 172.21.15.63:0/1058316052 >> 172.21.15.63:0/1058316052 conn(0x7f2ccc0a6410 msgr2=0x7f2ccc0a2600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:45.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.994+0000 7f2cd10ed640 1 -- 172.21.15.63:0/1058316052 shutdown_connections 2024-04-26T04:18:45.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.994+0000 7f2cd10ed640 1 -- 172.21.15.63:0/1058316052 wait complete. 2024-04-26T04:18:45.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.994+0000 7f2cd10ed640 10 monclient: build_initial_monmap 2024-04-26T04:18:45.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.994+0000 7f2cd10ed640 10 monclient: monmap: 2024-04-26T04:18:45.005 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:45.005 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:18:45.005 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:18:45.005 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:18:45.005 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:45.005 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:45.005 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:18:45.005 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:18:45.005 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:18:45.006 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:45.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.994+0000 7f2cd10ed640 1 Processor -- start 2024-04-26T04:18:45.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.994+0000 7f2cd10ed640 1 -- start start 2024-04-26T04:18:45.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.994+0000 7f2cd10ed640 10 monclient: init 2024-04-26T04:18:45.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.994+0000 7f2cd10ed640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:45.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.994+0000 7f2cd10ed640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T04:18:45.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.994+0000 7f2cd10ed640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ccc0a1bf0 0x7f2ccc1cee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:45.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.994+0000 7f2cd10ed640 10 monclient(hunting): picked mon.noname-b con 0x7f2ccc0a1bf0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:18:45.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.994+0000 7f2cd10ed640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2ccc0a1fd0 0x7f2ccc13af10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:45.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.994+0000 7f2cd10ed640 10 monclient(hunting): picked mon.noname-a con 0x7f2ccc0a1fd0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:18:45.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.994+0000 7f2cd10ed640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f2ccc13bd30 0x7f2ccc06f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:45.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.994+0000 7f2cd10ed640 10 monclient(hunting): picked mon.noname-c con 0x7f2ccc13bd30 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:45.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.994+0000 7f2cd10ed640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:45.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.994+0000 7f2cd10ed640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f2ccc0a31a0 con 0x7f2ccc0a1fd0 2024-04-26T04:18:45.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.994+0000 7f2cd10ed640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:45.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.994+0000 7f2cd10ed640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2ccc09ff30 con 0x7f2ccc0a1bf0 2024-04-26T04:18:45.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.994+0000 7f2cd10ed640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:45.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.994+0000 7f2cd10ed640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f2ccc0a3320 con 0x7f2ccc13bd30 2024-04-26T04:18:45.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.994+0000 7f2cd10ed640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:45.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.994+0000 7f2cd10ed640 10 monclient(hunting): authenticate will time out at 956.706289s 2024-04-26T04:18:45.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.994+0000 7f2cc3fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2ccc0a1fd0 0x7f2ccc13af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:45.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.994+0000 7f2ccb59e640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f2ccc13bd30 0x7f2ccc06f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:45.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.994+0000 7f2cc3fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2ccc0a1fd0 0x7f2ccc13af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:46336/0 (socket says 172.21.15.63:46336) 2024-04-26T04:18:45.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.994+0000 7f2cc3fff640 1 -- 172.21.15.63:0/3506339052 learned_addr learned my addr 172.21.15.63:0/3506339052 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:18:45.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.994+0000 7f2cc3fff640 10 monclient(hunting): get_auth_request con 0x7f2ccc0a1fd0 auth_method 0 2024-04-26T04:18:45.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.994+0000 7f2cc3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:45.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.994+0000 7f2cc3fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:45.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.994+0000 7f2cc3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:45.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.994+0000 7f2ccb59e640 10 monclient(hunting): get_auth_request con 0x7f2ccc13bd30 auth_method 0 2024-04-26T04:18:45.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.994+0000 7f2ccb59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:45.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.994+0000 7f2ccb59e640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:45.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.994+0000 7f2ccb59e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:45.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.994+0000 7f2cc3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:45.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.994+0000 7f2cc3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:45.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.994+0000 7f2cc3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:45.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.994+0000 7f2ccb59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:45.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.994+0000 7f2ccb59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:45.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.994+0000 7f2ccad9d640 1 --2- 172.21.15.63:0/3506339052 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ccc0a1bf0 0x7f2ccc1cee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:45.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.994+0000 7f2ccb59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:45.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.994+0000 7f2ccad9d640 10 monclient(hunting): get_auth_request con 0x7f2ccc0a1bf0 auth_method 0 2024-04-26T04:18:45.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.994+0000 7f2ccad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:45.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.994+0000 7f2ccad9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:45.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.994+0000 7f2ccad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:45.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.994+0000 7f2cc3fff640 10 monclient(hunting): handle_auth_done global_id 4182 payload 824 2024-04-26T04:18:45.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.994+0000 7f2cc3fff640 1 -- 172.21.15.63:0/3506339052 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f2ccc13bd30 msgr2=0x7f2ccc06f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:45.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.994+0000 7f2cc3fff640 1 --2- 172.21.15.63:0/3506339052 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f2ccc13bd30 0x7f2ccc06f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:45.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.994+0000 7f2cc3fff640 1 -- 172.21.15.63:0/3506339052 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ccc0a1bf0 msgr2=0x7f2ccc1cee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:45.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.994+0000 7f2cc3fff640 1 --2- 172.21.15.63:0/3506339052 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ccc0a1bf0 0x7f2ccc1cee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:45.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.994+0000 7f2cc3fff640 10 monclient: _finish_hunting 0 2024-04-26T04:18:45.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.994+0000 7f2cc3fff640 1 monclient: found mon.noname-a 2024-04-26T04:18:45.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.994+0000 7f2cc3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:45.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.994+0000 7f2cc3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:18:45.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.994+0000 7f2cc3fff640 1 -- 172.21.15.63:0/3506339052 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ccc1cfd20 con 0x7f2ccc0a1fd0 2024-04-26T04:18:45.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.994+0000 7f2ccb59e640 1 --2- 172.21.15.63:0/3506339052 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f2ccc13bd30 0x7f2ccc06f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T04:18:45.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.994+0000 7f2ccad9d640 1 --2- 172.21.15.63:0/3506339052 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ccc0a1bf0 0x7f2ccc1cee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T04:18:45.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.998+0000 7f2cc3fff640 1 -- 172.21.15.63:0/3506339052 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2ccc0a1fd0 msgr2=0x7f2ccc13af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:45.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.998+0000 7f2cc3fff640 1 --2- 172.21.15.63:0/3506339052 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2ccc0a1fd0 0x7f2ccc13af10 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2cb400c450 tx=0x7f2cb400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:45.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.998+0000 7f2cc37fe640 1 -- 172.21.15.63:0/3506339052 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cb4012a40 con 0x7f2ccc0a1fd0 2024-04-26T04:18:45.010 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.998+0000 7f2cc37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:45.010 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.998+0000 7f2cc37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:18:45.010 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.998+0000 7f2cc37fe640 10 monclient: dump: 2024-04-26T04:18:45.010 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:45.010 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:45.010 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:45.010 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:45.010 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:45.010 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:45.010 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:45.010 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:45.010 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:45.010 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:45.010 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.998+0000 7f2cc37fe640 10 monclient: _finish_auth 0 2024-04-26T04:18:45.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.998+0000 7f2cc37fe640 10 monclient: _check_auth_tickets 2024-04-26T04:18:45.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.998+0000 7f2cc37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:45.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.998+0000 7f2cd10ed640 5 monclient: authenticate success, global_id 4182 2024-04-26T04:18:45.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.998+0000 7f2cd10ed640 10 monclient: _renew_subs 2024-04-26T04:18:45.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.998+0000 7f2cd10ed640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:45.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.998+0000 7f2cd10ed640 1 -- 172.21.15.63:0/3506339052 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ccc138600 con 0x7f2ccc0a1fd0 2024-04-26T04:18:45.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.998+0000 7f2cc37fe640 1 -- 172.21.15.63:0/3506339052 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2cb4017070 con 0x7f2ccc0a1fd0 2024-04-26T04:18:45.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.998+0000 7f2cc37fe640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:45.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.998+0000 7f2cc37fe640 1 -- 172.21.15.63:0/3506339052 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cb4012e40 con 0x7f2ccc0a1fd0 2024-04-26T04:18:45.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.998+0000 7f2cc37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:45.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.998+0000 7f2cc37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:18:45.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.998+0000 7f2cc37fe640 10 monclient: dump: 2024-04-26T04:18:45.011 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:45.011 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:45.011 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:45.012 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:45.012 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:45.012 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:45.012 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:45.012 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:45.012 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:45.012 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:45.012 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.998+0000 7f2cd10ed640 10 monclient: _renew_subs 2024-04-26T04:18:45.012 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.998+0000 7f2cc9d9b640 4 set_mon_vals no callback set 2024-04-26T04:18:45.012 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.998+0000 7f2cd10ed640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:45.012 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.998+0000 7f2cd10ed640 1 -- 172.21.15.63:0/3506339052 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ccc139700 con 0x7f2ccc0a1fd0 2024-04-26T04:18:45.012 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.998+0000 7f2cc37fe640 1 -- 172.21.15.63:0/3506339052 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 4) ==== 43997+0+0 (secure 0 0 0) 0x7f2cb401c460 con 0x7f2ccc0a1fd0 2024-04-26T04:18:45.012 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.998+0000 7f2cd10ed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:45.012 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.998+0000 7f2cd10ed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:45.012 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.998+0000 7f2cd10ed640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:45.013 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.998+0000 7f2cd10ed640 1 -- 172.21.15.63:0/3506339052 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ccc06c870 con 0x7f2ccc0a1fd0 2024-04-26T04:18:45.013 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.998+0000 7f2cc37fe640 1 --2- 172.21.15.63:0/3506339052 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f2cac035e30 0x7f2cac0382f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:45.013 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.998+0000 7f2cc37fe640 1 -- 172.21.15.63:0/3506339052 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(12..12 src has 1..12) ==== 3982+0+0 (secure 0 0 0) 0x7f2cb40487d0 con 0x7f2ccc0a1fd0 2024-04-26T04:18:45.013 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.998+0000 7f2ccad9d640 1 --2- 172.21.15.63:0/3506339052 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f2cac035e30 0x7f2cac0382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:45.013 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:44.998+0000 7f2ccad9d640 10 monclient: get_auth_request con 0x7f2cac035e30 auth_method 0 2024-04-26T04:18:45.013 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.002+0000 7f2ccad9d640 1 --2- 172.21.15.63:0/3506339052 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f2cac035e30 0x7f2cac0382f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc007fd0 tx=0x7f2cbc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:45.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.002+0000 7f2cc37fe640 1 -- 172.21.15.63:0/3506339052 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f2cb401c9e0 con 0x7f2ccc0a1fd0 2024-04-26T04:18:45.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.002+0000 7f2cc37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:45.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.002+0000 7f2cc37fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:18:45.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.166+0000 7f2cd10ed640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-04-26T04:18:45.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.166+0000 7f2cd10ed640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-04-26T04:18:45.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.166+0000 7f2cd10ed640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:45.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.166+0000 7f2cd10ed640 1 -- 172.21.15.63:0/3506339052 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) -- 0x7f2ccc062e10 con 0x7f2ccc0a1fd0 2024-04-26T04:18:45.486 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.482+0000 7f2cc37fe640 1 -- 172.21.15.63:0/3506339052 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v13) ==== 109+0+0 (secure 0 0 0) 0x7f2cb4047500 con 0x7f2ccc0a1fd0 2024-04-26T04:18:45.486 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.482+0000 7f2cc37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-04-26T04:18:45.486 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.482+0000 7f2cc37fe640 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2024-04-26T04:18:45.486 INFO:teuthology.orchestra.run.smithi063.stderr:pool 'rbd' created 2024-04-26T04:18:45.489 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.482+0000 7f2cd10ed640 1 -- 172.21.15.63:0/3506339052 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f2cac035e30 msgr2=0x7f2cac0382f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:45.489 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.482+0000 7f2cd10ed640 1 --2- 172.21.15.63:0/3506339052 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f2cac035e30 0x7f2cac0382f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc007fd0 tx=0x7f2cbc009040 comp rx=0 tx=0).stop 2024-04-26T04:18:45.489 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.482+0000 7f2cd10ed640 10 monclient: shutdown 2024-04-26T04:18:45.489 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.482+0000 7f2cd10ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:45.489 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.482+0000 7f2cd10ed640 1 -- 172.21.15.63:0/3506339052 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2ccc0a1fd0 msgr2=0x7f2ccc13af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:45.489 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.482+0000 7f2cd10ed640 1 --2- 172.21.15.63:0/3506339052 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2ccc0a1fd0 0x7f2ccc13af10 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2cb400c450 tx=0x7f2cb400c910 comp rx=0 tx=0).stop 2024-04-26T04:18:45.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.482+0000 7f2cd10ed640 1 -- 172.21.15.63:0/3506339052 shutdown_connections 2024-04-26T04:18:45.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.482+0000 7f2cd10ed640 1 --2- 172.21.15.63:0/3506339052 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f2cac035e30 0x7f2cac0382f0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:45.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.482+0000 7f2cd10ed640 1 --2- 172.21.15.63:0/3506339052 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f2ccc13bd30 0x7f2ccc06f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:45.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.482+0000 7f2cd10ed640 1 --2- 172.21.15.63:0/3506339052 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2ccc0a1fd0 0x7f2ccc13af10 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:45.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.482+0000 7f2cd10ed640 1 --2- 172.21.15.63:0/3506339052 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ccc0a1bf0 0x7f2ccc1cee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:45.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.482+0000 7f2cd10ed640 1 -- 172.21.15.63:0/3506339052 >> 172.21.15.63:0/3506339052 conn(0x7f2ccc0a6410 msgr2=0x7f2ccc134f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:45.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.482+0000 7f2cd10ed640 1 -- 172.21.15.63:0/3506339052 shutdown_connections 2024-04-26T04:18:45.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.482+0000 7f2cd10ed640 1 -- 172.21.15.63:0/3506339052 wait complete. 2024-04-26T04:18:45.517 DEBUG:teuthology.orchestra.run.smithi063:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-26T04:18:45.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.638+0000 7f390c951640 10 monclient: get_monmap_and_config 2024-04-26T04:18:45.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.638+0000 7f390c951640 10 monclient: build_initial_monmap 2024-04-26T04:18:45.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.638+0000 7f390c951640 10 monclient: monmap: 2024-04-26T04:18:45.645 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:45.645 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:45.645 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:45.644669+0000 2024-04-26T04:18:45.645 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:45.644669+0000 2024-04-26T04:18:45.645 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:45.645 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:45.645 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:18:45.645 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:18:45.645 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:18:45.645 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:18:45.645 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:18:45.645 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:45.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.642+0000 7f390c951640 1 Processor -- start 2024-04-26T04:18:45.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.642+0000 7f390c951640 1 -- start start 2024-04-26T04:18:45.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.642+0000 7f390c951640 10 monclient: init 2024-04-26T04:18:45.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.642+0000 7f390c951640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:45.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.642+0000 7f390c951640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-04-26T04:18:45.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.642+0000 7f390c951640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f39080a1c60 addr v1:172.21.15.63:6789/0 2024-04-26T04:18:45.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.642+0000 7f390c951640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f390809b370 0x7f39080a2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:45.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.642+0000 7f390c951640 10 monclient(hunting): picked mon.noname-c con 0x7f390809b370 addr v2:172.21.15.186:3300/0 2024-04-26T04:18:45.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.642+0000 7f390c951640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f39080a2580 0x7f390813aee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:45.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.642+0000 7f390c951640 10 monclient(hunting): picked mon.noname-b con 0x7f39080a2580 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:45.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.642+0000 7f390c951640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f390809b130 con 0x7f39080a1c60 2024-04-26T04:18:45.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.642+0000 7f390c951640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:45.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.642+0000 7f390c951640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f39080a29d0 con 0x7f390809b370 2024-04-26T04:18:45.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.642+0000 7f390c951640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:45.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.642+0000 7f390c951640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f39080a3850 con 0x7f39080a2580 2024-04-26T04:18:45.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.642+0000 7f390c951640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:45.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.642+0000 7f390c951640 10 monclient(hunting): authenticate will time out at 957.353252s 2024-04-26T04:18:45.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.642+0000 7f390659c640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f39080a1c60 0x7f390809ddc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:44108/0 (socket says 172.21.15.63:44108) 2024-04-26T04:18:45.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.642+0000 7f390659c640 1 -- 172.21.15.63:0/2287340294 learned_addr learned my addr 172.21.15.63:0/2287340294 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-04-26T04:18:45.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.642+0000 7f3906d9d640 1 --2- 172.21.15.63:0/2287340294 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f39080a2580 0x7f390813aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:45.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.642+0000 7f3906d9d640 10 monclient(hunting): get_auth_request con 0x7f39080a2580 auth_method 0 2024-04-26T04:18:45.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.642+0000 7f3906d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:45.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.642+0000 7f3906d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:45.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.642+0000 7f3906d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:45.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.642+0000 7f3906d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:45.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.642+0000 7f3906d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:45.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.642+0000 7f3906d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:45.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.642+0000 7f3905d9b640 1 --2- 172.21.15.63:0/2287340294 >> v2:172.21.15.186:3300/0 conn(0x7f390809b370 0x7f39080a2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:45.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.642+0000 7f3905d9b640 10 monclient(hunting): get_auth_request con 0x7f390809b370 auth_method 0 2024-04-26T04:18:45.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.642+0000 7f3905d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:45.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.642+0000 7f3905d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:45.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.642+0000 7f3905d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:45.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.642+0000 7f3906d9d640 10 monclient(hunting): handle_auth_done global_id 4169 payload 274 2024-04-26T04:18:45.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.642+0000 7f3906d9d640 1 -- 172.21.15.63:0/2287340294 >> v2:172.21.15.186:3300/0 conn(0x7f390809b370 msgr2=0x7f39080a2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:45.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.642+0000 7f3906d9d640 1 --2- 172.21.15.63:0/2287340294 >> v2:172.21.15.186:3300/0 conn(0x7f390809b370 0x7f39080a2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:45.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.642+0000 7f3906d9d640 1 -- 172.21.15.63:0/2287340294 >> v1:172.21.15.63:6789/0 conn(0x7f39080a1c60 legacy=0x7f390809ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:45.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.642+0000 7f3906d9d640 10 monclient: _finish_hunting 0 2024-04-26T04:18:45.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.642+0000 7f3906d9d640 1 monclient: found mon.noname-b 2024-04-26T04:18:45.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.642+0000 7f3906d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:45.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.642+0000 7f3906d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.63:3301/0 2024-04-26T04:18:45.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.642+0000 7f3906d9d640 1 -- 172.21.15.63:0/2287340294 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f390813c430 con 0x7f39080a2580 2024-04-26T04:18:45.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.646+0000 7f3906d9d640 1 -- 172.21.15.63:0/2287340294 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f39080a2580 msgr2=0x7f390813aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:45.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.646+0000 7f3906d9d640 1 --2- 172.21.15.63:0/2287340294 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f39080a2580 0x7f390813aee0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f3900009d90 tx=0x7f390001c7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96396284ca02ef15 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:45.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.646+0000 7f390559a640 1 -- 172.21.15.63:0/2287340294 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39000ac070 con 0x7f39080a2580 2024-04-26T04:18:45.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.646+0000 7f390559a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:45.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.646+0000 7f390559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T04:18:45.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.646+0000 7f390559a640 10 monclient: dump: 2024-04-26T04:18:45.661 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:45.661 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:45.661 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:45.661 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:45.662 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:45.662 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:45.662 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:45.662 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:45.662 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:45.662 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:45.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.646+0000 7f390559a640 10 monclient: _finish_auth 0 2024-04-26T04:18:45.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.646+0000 7f390559a640 10 monclient: _check_auth_tickets 2024-04-26T04:18:45.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.646+0000 7f390559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:45.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.646+0000 7f390c951640 5 monclient: authenticate success, global_id 4169 2024-04-26T04:18:45.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.646+0000 7f390c951640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T04:18:45.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.646+0000 7f390559a640 1 -- 172.21.15.63:0/2287340294 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39000a6890 con 0x7f39080a2580 2024-04-26T04:18:45.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.646+0000 7f390559a640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:45.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.646+0000 7f390c951640 10 monclient: get_monmap_and_config success 2024-04-26T04:18:45.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.646+0000 7f390c951640 4 set_mon_vals no callback set 2024-04-26T04:18:45.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.646+0000 7f390559a640 1 -- 172.21.15.63:0/2287340294 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39000a6c60 con 0x7f39080a2580 2024-04-26T04:18:45.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.646+0000 7f390559a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:45.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.646+0000 7f390559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:18:45.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.646+0000 7f390559a640 10 monclient: dump: 2024-04-26T04:18:45.663 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:45.663 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:45.663 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:45.663 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:45.663 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:45.663 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:45.663 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:45.663 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:45.664 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:45.664 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:45.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.646+0000 7f390c951640 10 monclient: shutdown 2024-04-26T04:18:45.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.646+0000 7f390c951640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:45.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.646+0000 7f390c951640 1 -- 172.21.15.63:0/2287340294 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f39080a2580 msgr2=0x7f390813aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:45.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.646+0000 7f390c951640 1 --2- 172.21.15.63:0/2287340294 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f39080a2580 0x7f390813aee0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f3900009d90 tx=0x7f390001c7f0 comp rx=0 tx=0).stop 2024-04-26T04:18:45.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.646+0000 7f390c951640 1 -- 172.21.15.63:0/2287340294 shutdown_connections 2024-04-26T04:18:45.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.646+0000 7f390c951640 1 --2- 172.21.15.63:0/2287340294 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f39080a2580 0x7f390813aee0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:45.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.646+0000 7f390c951640 1 --2- 172.21.15.63:0/2287340294 >> v2:172.21.15.186:3300/0 conn(0x7f390809b370 0x7f39080a2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:45.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.646+0000 7f390c951640 1 -- 172.21.15.63:0/2287340294 >> 172.21.15.63:0/2287340294 conn(0x7f39080a6a90 msgr2=0x7f39080a2d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:45.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.646+0000 7f390c951640 1 -- 172.21.15.63:0/2287340294 shutdown_connections 2024-04-26T04:18:45.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.646+0000 7f390c951640 1 -- 172.21.15.63:0/2287340294 wait complete. 2024-04-26T04:18:45.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f390c951640 10 monclient: build_initial_monmap 2024-04-26T04:18:45.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f390c951640 10 monclient: monmap: 2024-04-26T04:18:45.664 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:45.665 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:18:45.665 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:18:45.665 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:18:45.665 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:45.665 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:45.665 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:18:45.665 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:18:45.665 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:18:45.665 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:45.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f390c951640 1 Processor -- start 2024-04-26T04:18:45.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f390c951640 1 -- start start 2024-04-26T04:18:45.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f390c951640 10 monclient: init 2024-04-26T04:18:45.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f390c951640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:45.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f390c951640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T04:18:45.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f390c951640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f39080a1c60 0x7f3908136cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:45.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f390c951640 10 monclient(hunting): picked mon.noname-a con 0x7f39080a1c60 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:18:45.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f390c951640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f39080a2580 0x7f3908137210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:45.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f390c951640 10 monclient(hunting): picked mon.noname-c con 0x7f39080a2580 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:45.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f390c951640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f39081ceb40 0x7f3908120cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:45.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f390c951640 10 monclient(hunting): picked mon.noname-b con 0x7f39081ceb40 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:18:45.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f390c951640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:45.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f390c951640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f390813c000 con 0x7f39080a1c60 2024-04-26T04:18:45.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f390c951640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:45.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f390c951640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f390813be80 con 0x7f39081ceb40 2024-04-26T04:18:45.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f390c951640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:45.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f390c951640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f390813c180 con 0x7f39080a2580 2024-04-26T04:18:45.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f390c951640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:45.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f390c951640 10 monclient(hunting): authenticate will time out at 957.360924s 2024-04-26T04:18:45.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f390659c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f39080a1c60 0x7f3908136cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:45.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f3905d9b640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f39080a2580 0x7f3908137210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:45.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f390659c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f39080a1c60 0x7f3908136cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:46344/0 (socket says 172.21.15.63:46344) 2024-04-26T04:18:45.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f3905d9b640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f39080a2580 0x7f3908137210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:57706/0 (socket says 172.21.15.63:57706) 2024-04-26T04:18:45.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f390659c640 1 -- 172.21.15.63:0/1278782131 learned_addr learned my addr 172.21.15.63:0/1278782131 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:18:45.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f390659c640 10 monclient(hunting): get_auth_request con 0x7f39080a1c60 auth_method 0 2024-04-26T04:18:45.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f390659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:45.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f390659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:45.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f390659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:45.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f3905d9b640 10 monclient(hunting): get_auth_request con 0x7f39080a2580 auth_method 0 2024-04-26T04:18:45.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f3905d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:45.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f3905d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:45.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f3905d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:45.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f3905d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:45.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f3905d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:45.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f3905d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:45.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f390659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:45.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f390659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:45.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f390659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:45.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f3906d9d640 1 --2- 172.21.15.63:0/1278782131 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f39081ceb40 0x7f3908120cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:45.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f3906d9d640 10 monclient(hunting): get_auth_request con 0x7f39081ceb40 auth_method 0 2024-04-26T04:18:45.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f3906d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:45.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f3906d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:45.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f3906d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:45.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f3905d9b640 10 monclient(hunting): handle_auth_done global_id 4172 payload 824 2024-04-26T04:18:45.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f3905d9b640 1 -- 172.21.15.63:0/1278782131 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f39081ceb40 msgr2=0x7f3908120cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:45.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f3905d9b640 1 --2- 172.21.15.63:0/1278782131 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f39081ceb40 0x7f3908120cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:45.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f3905d9b640 1 -- 172.21.15.63:0/1278782131 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f39080a1c60 msgr2=0x7f3908136cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:45.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f3905d9b640 1 --2- 172.21.15.63:0/1278782131 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f39080a1c60 0x7f3908136cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:45.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f3905d9b640 10 monclient: _finish_hunting 0 2024-04-26T04:18:45.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f3905d9b640 1 monclient: found mon.noname-c 2024-04-26T04:18:45.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f3905d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:45.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f3905d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.63:3301/0 2024-04-26T04:18:45.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f3905d9b640 1 -- 172.21.15.63:0/1278782131 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3908124480 con 0x7f39080a2580 2024-04-26T04:18:45.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f390659c640 1 --2- 172.21.15.63:0/1278782131 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f39080a1c60 0x7f3908136cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T04:18:45.672 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f3905d9b640 1 -- 172.21.15.63:0/1278782131 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f39080a2580 msgr2=0x7f3908137210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:45.672 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f3905d9b640 1 --2- 172.21.15.63:0/1278782131 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f39080a2580 0x7f3908137210 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f38f800d550 tx=0x7f38f800da10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:45.672 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f38f6ffd640 1 -- 172.21.15.63:0/1278782131 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38f8018020 con 0x7f39080a2580 2024-04-26T04:18:45.672 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f38f6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:45.672 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f38f6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:18:45.673 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f38f6ffd640 10 monclient: dump: 2024-04-26T04:18:45.673 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:45.673 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:45.673 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:45.673 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:45.673 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:45.673 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:45.673 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:45.673 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:45.673 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:45.673 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:45.673 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f38f6ffd640 10 monclient: _finish_auth 0 2024-04-26T04:18:45.673 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f38f6ffd640 10 monclient: _check_auth_tickets 2024-04-26T04:18:45.673 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f38f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:45.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f38f6ffd640 1 -- 172.21.15.63:0/1278782131 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38f8014830 con 0x7f39080a2580 2024-04-26T04:18:45.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f38f6ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:45.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f38f6ffd640 1 -- 172.21.15.63:0/1278782131 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38f8014c20 con 0x7f39080a2580 2024-04-26T04:18:45.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f38f6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:45.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f38f6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:18:45.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f38f6ffd640 10 monclient: dump: 2024-04-26T04:18:45.674 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:45.674 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:45.674 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:45.674 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:45.674 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:45.674 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:45.674 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:45.675 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:45.675 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:45.675 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:45.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.650+0000 7f3904d99640 4 set_mon_vals no callback set 2024-04-26T04:18:45.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.654+0000 7f390c951640 5 monclient: authenticate success, global_id 4172 2024-04-26T04:18:45.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.654+0000 7f390c951640 10 monclient: _renew_subs 2024-04-26T04:18:45.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.654+0000 7f390c951640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:45.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.654+0000 7f390c951640 1 -- 172.21.15.63:0/1278782131 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3908122cf0 con 0x7f39080a2580 2024-04-26T04:18:45.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.654+0000 7f390c951640 10 monclient: _renew_subs 2024-04-26T04:18:45.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.654+0000 7f390c951640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:45.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.654+0000 7f390c951640 1 -- 172.21.15.63:0/1278782131 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f39081241a0 con 0x7f39080a2580 2024-04-26T04:18:45.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.654+0000 7f38f6ffd640 1 -- 172.21.15.63:0/1278782131 <== mon.2 v2:172.21.15.63:3301/0 4 ==== mgrmap(e 4) ==== 43997+0+0 (secure 0 0 0) 0x7f38f801b020 con 0x7f39080a2580 2024-04-26T04:18:45.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.654+0000 7f390c951640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:45.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.654+0000 7f390c951640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:45.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.654+0000 7f390c951640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:45.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.654+0000 7f390c951640 1 -- 172.21.15.63:0/1278782131 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3908062e10 con 0x7f39080a2580 2024-04-26T04:18:45.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.654+0000 7f38f6ffd640 1 --2- 172.21.15.63:0/1278782131 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f38e8035e30 0x7f38e80382f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:45.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.654+0000 7f38f6ffd640 1 -- 172.21.15.63:0/1278782131 <== mon.2 v2:172.21.15.63:3301/0 5 ==== osd_map(13..13 src has 1..13) ==== 4335+0+0 (secure 0 0 0) 0x7f38f8048fb0 con 0x7f39080a2580 2024-04-26T04:18:45.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.654+0000 7f390659c640 1 --2- 172.21.15.63:0/1278782131 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f38e8035e30 0x7f38e80382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:45.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.654+0000 7f390659c640 10 monclient: get_auth_request con 0x7f38e8035e30 auth_method 0 2024-04-26T04:18:45.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.658+0000 7f390659c640 1 --2- 172.21.15.63:0/1278782131 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f38e8035e30 0x7f38e80382f0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f38fc007d60 tx=0x7f38fc007cb0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:45.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.658+0000 7f38f6ffd640 1 -- 172.21.15.63:0/1278782131 <== mon.2 v2:172.21.15.63:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f38f8022020 con 0x7f39080a2580 2024-04-26T04:18:45.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.658+0000 7f38f6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:45.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.658+0000 7f38f6ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:18:45.827 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.822+0000 7f390c951640 10 start_mon_command cmd=[{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-04-26T04:18:45.827 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.822+0000 7f390c951640 10 monclient: _send_command 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-04-26T04:18:45.827 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.822+0000 7f390c951640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:45.828 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:45.822+0000 7f390c951640 1 -- 172.21.15.63:0/1278782131 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) -- 0x7f3908122a40 con 0x7f39080a2580 2024-04-26T04:18:46.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.490+0000 7f38f6ffd640 1 -- 172.21.15.63:0/1278782131 <== mon.2 v2:172.21.15.63:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v14) ==== 173+0+0 (secure 0 0 0) 0x7f38f8013370 con 0x7f39080a2580 2024-04-26T04:18:46.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.490+0000 7f38f6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-04-26T04:18:46.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.490+0000 7f38f6ffd640 10 monclient: _finish_command 2 = system:0 enabled application 'rbd' on pool 'rbd' 2024-04-26T04:18:46.499 INFO:teuthology.orchestra.run.smithi063.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-26T04:18:46.500 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.494+0000 7f390c951640 1 -- 172.21.15.63:0/1278782131 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f38e8035e30 msgr2=0x7f38e80382f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:46.500 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.494+0000 7f390c951640 1 --2- 172.21.15.63:0/1278782131 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f38e8035e30 0x7f38e80382f0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f38fc007d60 tx=0x7f38fc007cb0 comp rx=0 tx=0).stop 2024-04-26T04:18:46.500 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.494+0000 7f390c951640 10 monclient: shutdown 2024-04-26T04:18:46.501 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.494+0000 7f390c951640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:46.501 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.494+0000 7f390c951640 1 -- 172.21.15.63:0/1278782131 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f39080a2580 msgr2=0x7f3908137210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:46.501 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.494+0000 7f390c951640 1 --2- 172.21.15.63:0/1278782131 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f39080a2580 0x7f3908137210 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f38f800d550 tx=0x7f38f800da10 comp rx=0 tx=0).stop 2024-04-26T04:18:46.501 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.494+0000 7f390c951640 1 -- 172.21.15.63:0/1278782131 shutdown_connections 2024-04-26T04:18:46.501 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.494+0000 7f390c951640 1 --2- 172.21.15.63:0/1278782131 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f38e8035e30 0x7f38e80382f0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:46.501 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.494+0000 7f390c951640 1 --2- 172.21.15.63:0/1278782131 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f39081ceb40 0x7f3908120cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:46.501 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.494+0000 7f390c951640 1 --2- 172.21.15.63:0/1278782131 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f39080a2580 0x7f3908137210 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:46.501 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.494+0000 7f390c951640 1 --2- 172.21.15.63:0/1278782131 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f39080a1c60 0x7f3908136cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:46.501 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.494+0000 7f390c951640 1 -- 172.21.15.63:0/1278782131 >> 172.21.15.63:0/1278782131 conn(0x7f39080a6a90 msgr2=0x7f390809e5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:46.501 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.494+0000 7f390c951640 1 -- 172.21.15.63:0/1278782131 shutdown_connections 2024-04-26T04:18:46.501 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.494+0000 7f390c951640 1 -- 172.21.15.63:0/1278782131 wait complete. 2024-04-26T04:18:46.526 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-26T04:18:46.526 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-26T04:18:46.526 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-26T04:18:46.526 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-26T04:18:46.527 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-26T04:18:46.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.638+0000 7f3dcfbcb640 10 monclient: get_monmap_and_config 2024-04-26T04:18:46.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.638+0000 7f3dcfbcb640 10 monclient: build_initial_monmap 2024-04-26T04:18:46.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.638+0000 7f3dcfbcb640 10 monclient: monmap: 2024-04-26T04:18:46.646 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:46.646 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:46.646 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:46.645272+0000 2024-04-26T04:18:46.646 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:46.645272+0000 2024-04-26T04:18:46.646 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:46.646 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:46.646 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:18:46.646 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:18:46.646 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:18:46.646 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:18:46.646 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:18:46.646 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:46.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.642+0000 7f3dcfbcb640 1 Processor -- start 2024-04-26T04:18:46.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.642+0000 7f3dcfbcb640 1 -- start start 2024-04-26T04:18:46.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.642+0000 7f3dcfbcb640 10 monclient: init 2024-04-26T04:18:46.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.642+0000 7f3dcfbcb640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:46.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.642+0000 7f3dcfbcb640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-04-26T04:18:46.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.642+0000 7f3dcfbcb640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f3dc80a0bb0 0x7f3dc809eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:46.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.642+0000 7f3dcfbcb640 10 monclient(hunting): picked mon.noname-b con 0x7f3dc80a0bb0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:46.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.642+0000 7f3dcfbcb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3dc809d7b0 addr v1:172.21.15.186:6789/0 2024-04-26T04:18:46.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.642+0000 7f3dcfbcb640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f3dc809e2a0 0x7f3dc813b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:46.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.642+0000 7f3dcfbcb640 10 monclient(hunting): picked mon.noname-c con 0x7f3dc809e2a0 addr v2:172.21.15.186:3300/0 2024-04-26T04:18:46.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.642+0000 7f3dcfbcb640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3dc80a1930 con 0x7f3dc809d7b0 2024-04-26T04:18:46.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.642+0000 7f3dcfbcb640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:46.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.642+0000 7f3dcfbcb640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f3dc80a0f90 con 0x7f3dc809e2a0 2024-04-26T04:18:46.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.642+0000 7f3dcfbcb640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:46.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.642+0000 7f3dcfbcb640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f3dc8099240 con 0x7f3dc80a0bb0 2024-04-26T04:18:46.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.642+0000 7f3dcfbcb640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:46.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.642+0000 7f3dcfbcb640 10 monclient(hunting): authenticate will time out at 958.352732s 2024-04-26T04:18:46.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.642+0000 7f3dcd967640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f3dc80a0bb0 0x7f3dc809eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:46.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.642+0000 7f3dcd967640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f3dc80a0bb0 0x7f3dc809eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:57716/0 (socket says 172.21.15.63:57716) 2024-04-26T04:18:46.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.642+0000 7f3dcd967640 1 -- 172.21.15.63:0/4096478084 learned_addr learned my addr 172.21.15.63:0/4096478084 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:18:46.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.642+0000 7f3dcd967640 10 monclient(hunting): get_auth_request con 0x7f3dc80a0bb0 auth_method 0 2024-04-26T04:18:46.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.642+0000 7f3dcd967640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:46.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.642+0000 7f3dcd967640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:46.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.642+0000 7f3dcd967640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:46.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.642+0000 7f3dce168640 1 --2- 172.21.15.63:0/4096478084 >> v2:172.21.15.186:3300/0 conn(0x7f3dc809e2a0 0x7f3dc813b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:46.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.642+0000 7f3dcd967640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:46.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.642+0000 7f3dcd967640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:46.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.642+0000 7f3dcd967640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:46.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.642+0000 7f3dce168640 10 monclient(hunting): get_auth_request con 0x7f3dc809e2a0 auth_method 0 2024-04-26T04:18:46.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.642+0000 7f3dce168640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:46.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.642+0000 7f3dce168640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:46.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.642+0000 7f3dce168640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:46.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.642+0000 7f3dce168640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:46.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.642+0000 7f3dce168640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:46.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.642+0000 7f3dce168640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:46.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.642+0000 7f3dcd967640 10 monclient(hunting): handle_auth_done global_id 4181 payload 274 2024-04-26T04:18:46.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.642+0000 7f3dcd967640 1 -- 172.21.15.63:0/4096478084 >> v2:172.21.15.186:3300/0 conn(0x7f3dc809e2a0 msgr2=0x7f3dc813b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:46.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.642+0000 7f3dcd967640 1 --2- 172.21.15.63:0/4096478084 >> v2:172.21.15.186:3300/0 conn(0x7f3dc809e2a0 0x7f3dc813b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:46.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.642+0000 7f3dcd967640 1 -- 172.21.15.63:0/4096478084 >> v1:172.21.15.186:6789/0 conn(0x7f3dc809d7b0 legacy=0x7f3dc809db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:46.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.642+0000 7f3dcd967640 10 monclient: _finish_hunting 0 2024-04-26T04:18:46.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.642+0000 7f3dcd967640 1 monclient: found mon.noname-b 2024-04-26T04:18:46.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.642+0000 7f3dcd967640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:46.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.642+0000 7f3dcd967640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.63:3301/0 2024-04-26T04:18:46.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.642+0000 7f3dcd967640 1 -- 172.21.15.63:0/4096478084 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dc809b820 con 0x7f3dc80a0bb0 2024-04-26T04:18:46.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.646+0000 7f3dcd967640 1 -- 172.21.15.63:0/4096478084 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f3dc80a0bb0 msgr2=0x7f3dc809eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:46.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.646+0000 7f3dcd967640 1 --2- 172.21.15.63:0/4096478084 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f3dc80a0bb0 0x7f3dc809eb60 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3dc4009870 tx=0x7f3dc401c3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c358ee39f99fe3a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:46.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.646+0000 7f3dcc965640 1 -- 172.21.15.63:0/4096478084 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dc40b0070 con 0x7f3dc80a0bb0 2024-04-26T04:18:46.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.646+0000 7f3dcc965640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:46.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.646+0000 7f3dcc965640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T04:18:46.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.646+0000 7f3dcc965640 10 monclient: dump: 2024-04-26T04:18:46.656 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:46.656 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:46.656 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:46.656 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:46.656 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:46.656 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:46.657 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:46.657 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:46.657 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:46.657 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:46.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.646+0000 7f3dcc965640 10 monclient: _finish_auth 0 2024-04-26T04:18:46.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.646+0000 7f3dcc965640 10 monclient: _check_auth_tickets 2024-04-26T04:18:46.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.646+0000 7f3dcc965640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:46.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.646+0000 7f3dcc965640 1 -- 172.21.15.63:0/4096478084 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3dc40ab600 con 0x7f3dc80a0bb0 2024-04-26T04:18:46.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.646+0000 7f3dcc965640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:46.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.646+0000 7f3dcc965640 1 -- 172.21.15.63:0/4096478084 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dc40ab9d0 con 0x7f3dc80a0bb0 2024-04-26T04:18:46.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.646+0000 7f3dcc965640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:46.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.646+0000 7f3dcc965640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:18:46.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.646+0000 7f3dcc965640 10 monclient: dump: 2024-04-26T04:18:46.658 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:46.658 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:46.658 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:46.658 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:46.658 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:46.658 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:46.658 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:46.658 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:46.658 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:46.658 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:46.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.646+0000 7f3dcfbcb640 5 monclient: authenticate success, global_id 4181 2024-04-26T04:18:46.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.646+0000 7f3dcfbcb640 10 monclient: get_monmap_and_config success 2024-04-26T04:18:46.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.646+0000 7f3dcfbcb640 4 set_mon_vals no callback set 2024-04-26T04:18:46.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.646+0000 7f3dcfbcb640 10 monclient: shutdown 2024-04-26T04:18:46.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.646+0000 7f3dcfbcb640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:46.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.646+0000 7f3dcfbcb640 1 -- 172.21.15.63:0/4096478084 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f3dc80a0bb0 msgr2=0x7f3dc809eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:46.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.646+0000 7f3dcfbcb640 1 --2- 172.21.15.63:0/4096478084 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f3dc80a0bb0 0x7f3dc809eb60 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3dc4009870 tx=0x7f3dc401c3b0 comp rx=0 tx=0).stop 2024-04-26T04:18:46.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.646+0000 7f3dcfbcb640 1 -- 172.21.15.63:0/4096478084 shutdown_connections 2024-04-26T04:18:46.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.646+0000 7f3dcfbcb640 1 --2- 172.21.15.63:0/4096478084 >> v2:172.21.15.186:3300/0 conn(0x7f3dc809e2a0 0x7f3dc813b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:46.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.646+0000 7f3dcfbcb640 1 --2- 172.21.15.63:0/4096478084 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f3dc80a0bb0 0x7f3dc809eb60 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:46.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.646+0000 7f3dcfbcb640 1 -- 172.21.15.63:0/4096478084 >> 172.21.15.63:0/4096478084 conn(0x7f3dc80a1e70 msgr2=0x7f3dc80a2280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:46.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.646+0000 7f3dcfbcb640 1 -- 172.21.15.63:0/4096478084 shutdown_connections 2024-04-26T04:18:46.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.646+0000 7f3dcfbcb640 1 -- 172.21.15.63:0/4096478084 wait complete. 2024-04-26T04:18:46.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.646+0000 7f3dcfbcb640 10 monclient: build_initial_monmap 2024-04-26T04:18:46.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.646+0000 7f3dcfbcb640 10 monclient: monmap: 2024-04-26T04:18:46.660 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:46.660 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:18:46.660 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:18:46.660 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:18:46.660 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:46.660 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:46.660 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:18:46.661 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:18:46.661 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:18:46.661 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:46.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dcfbcb640 1 Processor -- start 2024-04-26T04:18:46.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dcfbcb640 1 -- start start 2024-04-26T04:18:46.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dcfbcb640 10 monclient: init 2024-04-26T04:18:46.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dcfbcb640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:46.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dcfbcb640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T04:18:46.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dcfbcb640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f3dc809d7b0 0x7f3dc81ceb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:46.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dcfbcb640 10 monclient(hunting): picked mon.noname-c con 0x7f3dc809d7b0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:46.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dcfbcb640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3dc809e2a0 0x7f3dc8098430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:46.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dcfbcb640 10 monclient(hunting): picked mon.noname-b con 0x7f3dc809e2a0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:18:46.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dcfbcb640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3dc80a0bb0 0x7f3dc8098970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:46.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dcfbcb640 10 monclient(hunting): picked mon.noname-a con 0x7f3dc80a0bb0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:18:46.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dcfbcb640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:46.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dcfbcb640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f3dc813e220 con 0x7f3dc80a0bb0 2024-04-26T04:18:46.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dcfbcb640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:46.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dcfbcb640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3dc813e0a0 con 0x7f3dc809e2a0 2024-04-26T04:18:46.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dcfbcb640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:46.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dcfbcb640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f3dc813e3a0 con 0x7f3dc809d7b0 2024-04-26T04:18:46.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dcfbcb640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:46.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dcfbcb640 10 monclient(hunting): authenticate will time out at 958.360731s 2024-04-26T04:18:46.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dcd967640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f3dc809d7b0 0x7f3dc81ceb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:46.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dce168640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3dc80a0bb0 0x7f3dc8098970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:46.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dcd967640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f3dc809d7b0 0x7f3dc81ceb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:57718/0 (socket says 172.21.15.63:57718) 2024-04-26T04:18:46.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dcd967640 1 -- 172.21.15.63:0/555975847 learned_addr learned my addr 172.21.15.63:0/555975847 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:18:46.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dce168640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3dc80a0bb0 0x7f3dc8098970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:46350/0 (socket says 172.21.15.63:46350) 2024-04-26T04:18:46.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dcd967640 10 monclient(hunting): get_auth_request con 0x7f3dc809d7b0 auth_method 0 2024-04-26T04:18:46.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dcd967640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:46.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dcd967640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:46.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dcd967640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:46.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dce168640 10 monclient(hunting): get_auth_request con 0x7f3dc80a0bb0 auth_method 0 2024-04-26T04:18:46.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dce168640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:46.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dce168640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:46.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dce168640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:46.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dcd166640 1 --2- 172.21.15.63:0/555975847 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3dc809e2a0 0x7f3dc8098430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:46.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dcd166640 10 monclient(hunting): get_auth_request con 0x7f3dc809e2a0 auth_method 0 2024-04-26T04:18:46.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dcd166640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:46.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dcd166640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:46.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dcd166640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:46.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dcd967640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:46.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dcd967640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:46.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dcd967640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:46.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dce168640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:46.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dce168640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:46.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dce168640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:46.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dcd166640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:46.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dcd166640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:46.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dcd166640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:46.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dcd967640 10 monclient(hunting): handle_auth_done global_id 4184 payload 824 2024-04-26T04:18:46.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dcd967640 1 -- 172.21.15.63:0/555975847 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3dc809e2a0 msgr2=0x7f3dc8098430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:46.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dcd967640 1 --2- 172.21.15.63:0/555975847 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3dc809e2a0 0x7f3dc8098430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:46.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dcd967640 1 -- 172.21.15.63:0/555975847 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3dc80a0bb0 msgr2=0x7f3dc8098970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:46.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dcd967640 1 --2- 172.21.15.63:0/555975847 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3dc80a0bb0 0x7f3dc8098970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:46.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dcd967640 10 monclient: _finish_hunting 0 2024-04-26T04:18:46.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dcd967640 1 monclient: found mon.noname-c 2024-04-26T04:18:46.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dcd967640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:46.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dcd967640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.63:3301/0 2024-04-26T04:18:46.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dcd967640 1 -- 172.21.15.63:0/555975847 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dc806f5b0 con 0x7f3dc809d7b0 2024-04-26T04:18:46.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dce168640 1 --2- 172.21.15.63:0/555975847 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3dc80a0bb0 0x7f3dc8098970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T04:18:46.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dcd967640 1 -- 172.21.15.63:0/555975847 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f3dc809d7b0 msgr2=0x7f3dc81ceb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:46.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dcd967640 1 --2- 172.21.15.63:0/555975847 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f3dc809d7b0 0x7f3dc81ceb10 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3dc401aa60 tx=0x7f3dc4019150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:46.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dbe7fc640 1 -- 172.21.15.63:0/555975847 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dc40bb020 con 0x7f3dc809d7b0 2024-04-26T04:18:46.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dbe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:46.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dbe7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:18:46.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dbe7fc640 10 monclient: dump: 2024-04-26T04:18:46.669 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:46.669 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:46.669 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:46.669 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:46.670 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:46.670 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:46.670 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:46.670 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:46.670 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:46.670 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:46.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dbe7fc640 10 monclient: _finish_auth 0 2024-04-26T04:18:46.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dbe7fc640 10 monclient: _check_auth_tickets 2024-04-26T04:18:46.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dbe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:46.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dbe7fc640 1 -- 172.21.15.63:0/555975847 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3dc40b1040 con 0x7f3dc809d7b0 2024-04-26T04:18:46.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dbe7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:46.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dbe7fc640 1 -- 172.21.15.63:0/555975847 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dc40b0070 con 0x7f3dc809d7b0 2024-04-26T04:18:46.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dcfbcb640 5 monclient: authenticate success, global_id 4184 2024-04-26T04:18:46.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dcfbcb640 10 monclient: _renew_subs 2024-04-26T04:18:46.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dcfbcb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:46.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dcfbcb640 1 -- 172.21.15.63:0/555975847 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3dc809ba60 con 0x7f3dc809d7b0 2024-04-26T04:18:46.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dbe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:46.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dbffff640 4 set_mon_vals no callback set 2024-04-26T04:18:46.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dbe7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:18:46.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dbe7fc640 10 monclient: dump: 2024-04-26T04:18:46.671 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:46.671 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:46.671 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:46.671 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:46.671 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:46.671 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:46.671 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:46.671 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:46.672 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:46.672 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:46.672 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.650+0000 7f3dcfbcb640 10 monclient: _renew_subs 2024-04-26T04:18:46.672 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.654+0000 7f3dcfbcb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:46.672 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.654+0000 7f3dcfbcb640 1 -- 172.21.15.63:0/555975847 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3dc806f9f0 con 0x7f3dc809d7b0 2024-04-26T04:18:46.672 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.654+0000 7f3dbe7fc640 1 -- 172.21.15.63:0/555975847 <== mon.2 v2:172.21.15.63:3301/0 4 ==== mgrmap(e 4) ==== 43997+0+0 (secure 0 0 0) 0x7f3dc40be020 con 0x7f3dc809d7b0 2024-04-26T04:18:46.672 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.654+0000 7f3dcfbcb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:46.672 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.654+0000 7f3dcfbcb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:46.672 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.654+0000 7f3dcfbcb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:46.672 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.654+0000 7f3dcfbcb640 1 -- 172.21.15.63:0/555975847 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3dc8062e10 con 0x7f3dc809d7b0 2024-04-26T04:18:46.672 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.654+0000 7f3dbe7fc640 1 --2- 172.21.15.63:0/555975847 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f3dac035e80 0x7f3dac038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:46.672 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.654+0000 7f3dbe7fc640 1 -- 172.21.15.63:0/555975847 <== mon.2 v2:172.21.15.63:3301/0 5 ==== osd_map(14..14 src has 1..14) ==== 4346+0+0 (secure 0 0 0) 0x7f3dc40e3010 con 0x7f3dc809d7b0 2024-04-26T04:18:46.672 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.654+0000 7f3dcd166640 1 --2- 172.21.15.63:0/555975847 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f3dac035e80 0x7f3dac038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:46.672 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.654+0000 7f3dcd166640 10 monclient: get_auth_request con 0x7f3dac035e80 auth_method 0 2024-04-26T04:18:46.673 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.658+0000 7f3dcd166640 1 --2- 172.21.15.63:0/555975847 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f3dac035e80 0x7f3dac038340 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3dc806f230 tx=0x7f3db8007e40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:46.673 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.658+0000 7f3dbe7fc640 1 -- 172.21.15.63:0/555975847 <== mon.2 v2:172.21.15.63:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f3dc40c5030 con 0x7f3dc809d7b0 2024-04-26T04:18:46.673 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.658+0000 7f3dbe7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:46.673 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.658+0000 7f3dbe7fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:18:46.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.866+0000 7f3dcfbcb640 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2024-04-26T04:18:46.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.866+0000 7f3dcfbcb640 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2024-04-26T04:18:46.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.866+0000 7f3dcfbcb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:46.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.866+0000 7f3dcfbcb640 1 -- 172.21.15.63:0/555975847 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f3dc8068430 con 0x7f3dc809d7b0 2024-04-26T04:18:46.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.866+0000 7f3dbe7fc640 1 -- 172.21.15.63:0/555975847 <== mon.2 v2:172.21.15.63:3301/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v4) ==== 74+0+84086 (secure 0 0 0) 0x7f3dc40160f0 con 0x7f3dc809d7b0 2024-04-26T04:18:46.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.866+0000 7f3dbe7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2024-04-26T04:18:46.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.866+0000 7f3dbe7fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:18:46.873 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-26T04:18:46.875 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.870+0000 7f3dcfbcb640 1 -- 172.21.15.63:0/555975847 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f3dac035e80 msgr2=0x7f3dac038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:46.875 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.870+0000 7f3dcfbcb640 1 --2- 172.21.15.63:0/555975847 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f3dac035e80 0x7f3dac038340 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3dc806f230 tx=0x7f3db8007e40 comp rx=0 tx=0).stop 2024-04-26T04:18:46.875 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.870+0000 7f3dcfbcb640 10 monclient: shutdown 2024-04-26T04:18:46.875 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.870+0000 7f3dcfbcb640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:46.875 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.870+0000 7f3dcfbcb640 1 -- 172.21.15.63:0/555975847 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f3dc809d7b0 msgr2=0x7f3dc81ceb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:46.875 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.870+0000 7f3dcfbcb640 1 --2- 172.21.15.63:0/555975847 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f3dc809d7b0 0x7f3dc81ceb10 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3dc401aa60 tx=0x7f3dc4019150 comp rx=0 tx=0).stop 2024-04-26T04:18:46.876 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.870+0000 7f3dcfbcb640 1 -- 172.21.15.63:0/555975847 shutdown_connections 2024-04-26T04:18:46.876 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.870+0000 7f3dcfbcb640 1 --2- 172.21.15.63:0/555975847 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f3dac035e80 0x7f3dac038340 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:46.876 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.870+0000 7f3dcfbcb640 1 --2- 172.21.15.63:0/555975847 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3dc80a0bb0 0x7f3dc8098970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:46.876 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.870+0000 7f3dcfbcb640 1 --2- 172.21.15.63:0/555975847 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3dc809e2a0 0x7f3dc8098430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:46.876 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.870+0000 7f3dcfbcb640 1 --2- 172.21.15.63:0/555975847 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f3dc809d7b0 0x7f3dc81ceb10 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:46.876 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.870+0000 7f3dcfbcb640 1 -- 172.21.15.63:0/555975847 >> 172.21.15.63:0/555975847 conn(0x7f3dc80a1e70 msgr2=0x7f3dc809c6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:46.876 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.870+0000 7f3dcfbcb640 1 -- 172.21.15.63:0/555975847 shutdown_connections 2024-04-26T04:18:46.876 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:46.870+0000 7f3dcfbcb640 1 -- 172.21.15.63:0/555975847 wait complete. 2024-04-26T04:18:46.891 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":4,"flags":0,"active_gid":4105,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2583706639},{"type":"v1","addr":"172.21.15.186:6825","nonce":2583706639}]},"active_addr":"172.21.15.186:6825/2583706639","active_change":"2024-04-26T04:18:39.431164+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.186:0","nonce":969427663}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.186:0","nonce":3830768028}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.186:0","nonce":3371660837}]}]} 2024-04-26T04:18:46.895 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-26T04:18:46.895 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-26T04:18:46.896 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T04:18:47.012 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.006+0000 7f2cf5d63640 10 monclient: get_monmap_and_config 2024-04-26T04:18:47.012 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.006+0000 7f2cf5d63640 10 monclient: build_initial_monmap 2024-04-26T04:18:47.012 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.006+0000 7f2cf5d63640 10 monclient: monmap: 2024-04-26T04:18:47.012 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:47.012 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:47.012 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:47.012463+0000 2024-04-26T04:18:47.013 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:47.012463+0000 2024-04-26T04:18:47.013 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:47.013 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:47.013 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:18:47.013 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:18:47.013 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:18:47.013 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:18:47.013 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:18:47.013 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:47.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.006+0000 7f2cf5d63640 1 Processor -- start 2024-04-26T04:18:47.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.006+0000 7f2cf5d63640 1 -- start start 2024-04-26T04:18:47.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.006+0000 7f2cf5d63640 10 monclient: init 2024-04-26T04:18:47.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.010+0000 7f2cf5d63640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:47.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.010+0000 7f2cf5d63640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-04-26T04:18:47.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.010+0000 7f2cf5d63640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f2cf006be90 0x7f2cf00a48c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:47.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.010+0000 7f2cf5d63640 10 monclient(hunting): picked mon.noname-b con 0x7f2cf006be90 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:47.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.010+0000 7f2cf5d63640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2cf00a4e00 addr v1:172.21.15.63:6789/0 2024-04-26T04:18:47.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.010+0000 7f2cf5d63640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f2cf00a5200 0x7f2cf013b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:47.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.010+0000 7f2cf5d63640 10 monclient(hunting): picked mon.noname-a con 0x7f2cf00a5200 addr v2:172.21.15.63:3300/0 2024-04-26T04:18:47.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.010+0000 7f2cf5d63640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2cf006f330 con 0x7f2cf00a4e00 2024-04-26T04:18:47.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.010+0000 7f2cf5d63640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:47.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.010+0000 7f2cf5d63640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7f2cf006c270 con 0x7f2cf00a5200 2024-04-26T04:18:47.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.010+0000 7f2cf5d63640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:47.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.010+0000 7f2cf5d63640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f2cf00707e0 con 0x7f2cf006be90 2024-04-26T04:18:47.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.010+0000 7f2cf5d63640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:47.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.010+0000 7f2cf5d63640 10 monclient(hunting): authenticate will time out at 958.720305s 2024-04-26T04:18:47.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.010+0000 7f2ceeffd640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f2cf00a4e00 0x7f2cf01381a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:44122/0 (socket says 172.21.15.63:44122) 2024-04-26T04:18:47.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.010+0000 7f2ceeffd640 1 -- 172.21.15.63:0/2044105305 learned_addr learned my addr 172.21.15.63:0/2044105305 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-04-26T04:18:47.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.010+0000 7f2cef7fe640 1 --2- 172.21.15.63:0/2044105305 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f2cf006be90 0x7f2cf00a48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:47.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.010+0000 7f2cef7fe640 10 monclient(hunting): get_auth_request con 0x7f2cf006be90 auth_method 0 2024-04-26T04:18:47.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.010+0000 7f2cef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:47.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.010+0000 7f2cef7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:47.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.010+0000 7f2cef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:47.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.010+0000 7f2ceffff640 1 --2- 172.21.15.63:0/2044105305 >> v2:172.21.15.63:3300/0 conn(0x7f2cf00a5200 0x7f2cf013b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:47.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.010+0000 7f2ceffff640 10 monclient(hunting): get_auth_request con 0x7f2cf00a5200 auth_method 0 2024-04-26T04:18:47.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.010+0000 7f2ceffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:47.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.010+0000 7f2ceffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:47.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.010+0000 7f2ceffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:47.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.010+0000 7f2cef7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:47.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.010+0000 7f2cef7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:47.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.010+0000 7f2cef7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:47.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.010+0000 7f2ceffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:47.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.010+0000 7f2ceffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:47.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.010+0000 7f2ceffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:47.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.010+0000 7f2ceffff640 10 monclient(hunting): handle_auth_done global_id 4197 payload 274 2024-04-26T04:18:47.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.010+0000 7f2ceffff640 1 -- 172.21.15.63:0/2044105305 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f2cf006be90 msgr2=0x7f2cf00a48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:47.022 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.010+0000 7f2ceffff640 1 --2- 172.21.15.63:0/2044105305 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f2cf006be90 0x7f2cf00a48c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:47.022 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.010+0000 7f2ceffff640 1 -- 172.21.15.63:0/2044105305 >> v1:172.21.15.63:6789/0 conn(0x7f2cf00a4e00 legacy=0x7f2cf01381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:47.022 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.010+0000 7f2ceffff640 10 monclient: _finish_hunting 0 2024-04-26T04:18:47.022 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.010+0000 7f2ceffff640 1 monclient: found mon.noname-a 2024-04-26T04:18:47.022 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.010+0000 7f2ceffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:47.022 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.010+0000 7f2ceffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:18:47.022 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.010+0000 7f2ceffff640 1 -- 172.21.15.63:0/2044105305 --> v2:172.21.15.63:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cf0070ac0 con 0x7f2cf00a5200 2024-04-26T04:18:47.031 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.014+0000 7f2ceffff640 1 -- 172.21.15.63:0/2044105305 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2cf00a5200 msgr2=0x7f2cf013b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:47.031 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.014+0000 7f2ceffff640 1 --2- 172.21.15.63:0/2044105305 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2cf00a5200 0x7f2cf013b8d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2ce8009d90 tx=0x7f2ce801c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2202738795d461da server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:47.031 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.014+0000 7f2cee7fc640 1 -- 172.21.15.63:0/2044105305 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ce80b2070 con 0x7f2cf00a5200 2024-04-26T04:18:47.031 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.014+0000 7f2cee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:47.031 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.014+0000 7f2cee7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:18:47.031 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.014+0000 7f2cee7fc640 10 monclient: dump: 2024-04-26T04:18:47.031 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:47.031 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:47.031 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:47.031 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:47.031 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:47.031 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:47.032 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:47.032 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:47.032 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:47.032 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:47.032 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.014+0000 7f2cee7fc640 10 monclient: _finish_auth 0 2024-04-26T04:18:47.032 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.014+0000 7f2cee7fc640 10 monclient: _check_auth_tickets 2024-04-26T04:18:47.032 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.014+0000 7f2cee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:47.032 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.014+0000 7f2cee7fc640 1 -- 172.21.15.63:0/2044105305 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ce80a5800 con 0x7f2cf00a5200 2024-04-26T04:18:47.032 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.014+0000 7f2cee7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:47.032 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.014+0000 7f2cee7fc640 1 -- 172.21.15.63:0/2044105305 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ce80a5bd0 con 0x7f2cf00a5200 2024-04-26T04:18:47.032 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.014+0000 7f2cf5d63640 5 monclient: authenticate success, global_id 4197 2024-04-26T04:18:47.032 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.014+0000 7f2cf5d63640 10 monclient: get_monmap_and_config success 2024-04-26T04:18:47.032 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.014+0000 7f2cf5d63640 4 set_mon_vals no callback set 2024-04-26T04:18:47.032 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.014+0000 7f2cee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:47.032 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.014+0000 7f2cee7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:18:47.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.014+0000 7f2cee7fc640 10 monclient: dump: 2024-04-26T04:18:47.033 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:47.033 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:47.033 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:47.033 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:47.033 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:47.033 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:47.033 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:47.033 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:47.033 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:47.033 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:47.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.018+0000 7f2cf5d63640 10 monclient: shutdown 2024-04-26T04:18:47.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.018+0000 7f2cf5d63640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:47.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.018+0000 7f2cf5d63640 1 -- 172.21.15.63:0/2044105305 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2cf00a5200 msgr2=0x7f2cf013b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:47.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.018+0000 7f2cf5d63640 1 --2- 172.21.15.63:0/2044105305 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2cf00a5200 0x7f2cf013b8d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2ce8009d90 tx=0x7f2ce801c7c0 comp rx=0 tx=0).stop 2024-04-26T04:18:47.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.018+0000 7f2cf5d63640 1 -- 172.21.15.63:0/2044105305 shutdown_connections 2024-04-26T04:18:47.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.018+0000 7f2cf5d63640 1 --2- 172.21.15.63:0/2044105305 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2cf00a5200 0x7f2cf013b8d0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:47.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.018+0000 7f2cf5d63640 1 --2- 172.21.15.63:0/2044105305 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f2cf006be90 0x7f2cf00a48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:47.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.018+0000 7f2cf5d63640 1 -- 172.21.15.63:0/2044105305 >> 172.21.15.63:0/2044105305 conn(0x7f2cf00a60a0 msgr2=0x7f2cf00a24b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:47.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.018+0000 7f2cf5d63640 1 -- 172.21.15.63:0/2044105305 shutdown_connections 2024-04-26T04:18:47.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.018+0000 7f2cf5d63640 1 -- 172.21.15.63:0/2044105305 wait complete. 2024-04-26T04:18:47.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.018+0000 7f2cf5d63640 10 monclient: build_initial_monmap 2024-04-26T04:18:47.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.018+0000 7f2cf5d63640 10 monclient: monmap: 2024-04-26T04:18:47.034 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:47.034 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:18:47.034 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:18:47.034 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:18:47.034 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:47.035 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:47.035 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:18:47.035 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:18:47.035 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:18:47.035 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:47.035 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.018+0000 7f2cf5d63640 1 Processor -- start 2024-04-26T04:18:47.035 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.018+0000 7f2cf5d63640 1 -- start start 2024-04-26T04:18:47.035 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.018+0000 7f2cf5d63640 10 monclient: init 2024-04-26T04:18:47.035 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.018+0000 7f2cf5d63640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:47.035 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.018+0000 7f2cf5d63640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T04:18:47.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.018+0000 7f2cf5d63640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2cf006be90 0x7f2cf01ced80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:47.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.018+0000 7f2cf5d63640 10 monclient(hunting): picked mon.noname-a con 0x7f2cf006be90 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:18:47.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.018+0000 7f2cf5d63640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f2cf00a4e00 0x7f2cf0139700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:47.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.018+0000 7f2cf5d63640 10 monclient(hunting): picked mon.noname-c con 0x7f2cf00a4e00 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:47.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.018+0000 7f2cf5d63640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2cf00a5200 0x7f2cf0139c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:47.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.018+0000 7f2cf5d63640 10 monclient(hunting): picked mon.noname-b con 0x7f2cf00a5200 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:18:47.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.018+0000 7f2cf5d63640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:47.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.018+0000 7f2cf5d63640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f2cf0071be0 con 0x7f2cf006be90 2024-04-26T04:18:47.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.018+0000 7f2cf5d63640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:47.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.018+0000 7f2cf5d63640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2cf0071a60 con 0x7f2cf00a5200 2024-04-26T04:18:47.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.018+0000 7f2cf5d63640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:47.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.018+0000 7f2cf5d63640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f2cf013e630 con 0x7f2cf00a4e00 2024-04-26T04:18:47.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.018+0000 7f2cf5d63640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:47.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.018+0000 7f2cf5d63640 10 monclient(hunting): authenticate will time out at 958.730027s 2024-04-26T04:18:47.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.018+0000 7f2cef7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2cf006be90 0x7f2cf01ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:47.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.018+0000 7f2cef7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2cf006be90 0x7f2cf01ced80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:46362/0 (socket says 172.21.15.63:46362) 2024-04-26T04:18:47.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.018+0000 7f2cef7fe640 1 -- 172.21.15.63:0/2782847643 learned_addr learned my addr 172.21.15.63:0/2782847643 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:18:47.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.018+0000 7f2ceeffd640 1 --2- 172.21.15.63:0/2782847643 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f2cf00a4e00 0x7f2cf0139700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:47.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.018+0000 7f2cef7fe640 10 monclient(hunting): get_auth_request con 0x7f2cf006be90 auth_method 0 2024-04-26T04:18:47.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.018+0000 7f2cef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:47.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.018+0000 7f2cef7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:47.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.018+0000 7f2cef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:47.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.018+0000 7f2ceeffd640 10 monclient(hunting): get_auth_request con 0x7f2cf00a4e00 auth_method 0 2024-04-26T04:18:47.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.018+0000 7f2ceeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:47.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.018+0000 7f2ceeffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:47.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.018+0000 7f2ceeffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:47.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.018+0000 7f2cef7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:47.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.018+0000 7f2cef7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:47.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.018+0000 7f2ceffff640 1 --2- 172.21.15.63:0/2782847643 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2cf00a5200 0x7f2cf0139c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:47.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.018+0000 7f2cef7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:47.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.018+0000 7f2ceffff640 10 monclient(hunting): get_auth_request con 0x7f2cf00a5200 auth_method 0 2024-04-26T04:18:47.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.018+0000 7f2ceffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:47.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.018+0000 7f2ceffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:47.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.018+0000 7f2ceffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:47.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.018+0000 7f2ceeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:47.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.018+0000 7f2ceeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:47.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.018+0000 7f2ceeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:47.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.018+0000 7f2ceffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:47.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.018+0000 7f2ceffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:47.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.018+0000 7f2ceffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:47.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.018+0000 7f2cef7fe640 10 monclient(hunting): handle_auth_done global_id 4200 payload 824 2024-04-26T04:18:47.039 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.018+0000 7f2cef7fe640 1 -- 172.21.15.63:0/2782847643 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f2cf00a4e00 msgr2=0x7f2cf0139700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:47.039 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.018+0000 7f2cef7fe640 1 --2- 172.21.15.63:0/2782847643 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f2cf00a4e00 0x7f2cf0139700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:47.039 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.018+0000 7f2cef7fe640 1 -- 172.21.15.63:0/2782847643 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2cf00a5200 msgr2=0x7f2cf0139c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:47.039 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.018+0000 7f2cef7fe640 1 --2- 172.21.15.63:0/2782847643 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2cf00a5200 0x7f2cf0139c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:47.039 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.018+0000 7f2cef7fe640 10 monclient: _finish_hunting 0 2024-04-26T04:18:47.039 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.018+0000 7f2cef7fe640 1 monclient: found mon.noname-a 2024-04-26T04:18:47.039 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.018+0000 7f2cef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:47.039 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.018+0000 7f2cef7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:18:47.039 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.018+0000 7f2cef7fe640 1 -- 172.21.15.63:0/2782847643 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cf01270c0 con 0x7f2cf006be90 2024-04-26T04:18:47.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.022+0000 7f2cef7fe640 1 -- 172.21.15.63:0/2782847643 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2cf006be90 msgr2=0x7f2cf01ced80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:47.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.022+0000 7f2cef7fe640 1 --2- 172.21.15.63:0/2782847643 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2cf006be90 0x7f2cf01ced80 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f2ce400e870 tx=0x7f2ce400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:47.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.022+0000 7f2cd7fff640 1 -- 172.21.15.63:0/2782847643 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ce4017070 con 0x7f2cf006be90 2024-04-26T04:18:47.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.022+0000 7f2cd7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:47.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.022+0000 7f2cd7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:18:47.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.022+0000 7f2cd7fff640 10 monclient: dump: 2024-04-26T04:18:47.040 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:47.040 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:47.040 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:47.040 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:47.040 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:47.040 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:47.040 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:47.041 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:47.041 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:47.041 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:47.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.022+0000 7f2cd7fff640 10 monclient: _finish_auth 0 2024-04-26T04:18:47.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.022+0000 7f2cd7fff640 10 monclient: _check_auth_tickets 2024-04-26T04:18:47.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.022+0000 7f2cd7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:47.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.022+0000 7f2cd7fff640 1 -- 172.21.15.63:0/2782847643 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ce400ee60 con 0x7f2cf006be90 2024-04-26T04:18:47.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.022+0000 7f2cd7fff640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:47.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.022+0000 7f2cd7fff640 1 -- 172.21.15.63:0/2782847643 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ce400cd30 con 0x7f2cf006be90 2024-04-26T04:18:47.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.022+0000 7f2cf5d63640 5 monclient: authenticate success, global_id 4200 2024-04-26T04:18:47.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.022+0000 7f2cf5d63640 10 monclient: _renew_subs 2024-04-26T04:18:47.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.022+0000 7f2cf5d63640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:47.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.022+0000 7f2cf5d63640 1 -- 172.21.15.63:0/2782847643 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cf01256b0 con 0x7f2cf006be90 2024-04-26T04:18:47.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.022+0000 7f2cd7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:47.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.022+0000 7f2cd7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:18:47.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.022+0000 7f2cd7fff640 10 monclient: dump: 2024-04-26T04:18:47.042 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:47.042 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:47.042 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:47.042 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:47.042 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:47.042 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:47.042 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:47.043 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:47.043 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:47.043 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:47.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.022+0000 7f2ced7fa640 4 set_mon_vals no callback set 2024-04-26T04:18:47.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.022+0000 7f2cf5d63640 10 monclient: _renew_subs 2024-04-26T04:18:47.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.022+0000 7f2cf5d63640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:47.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.022+0000 7f2cf5d63640 1 -- 172.21.15.63:0/2782847643 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cf0124dd0 con 0x7f2cf006be90 2024-04-26T04:18:47.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.022+0000 7f2cf5d63640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:47.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.022+0000 7f2cf5d63640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:47.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.022+0000 7f2cf5d63640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:47.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.022+0000 7f2cf5d63640 1 -- 172.21.15.63:0/2782847643 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2cf0062e10 con 0x7f2cf006be90 2024-04-26T04:18:47.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.022+0000 7f2cd7fff640 1 -- 172.21.15.63:0/2782847643 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 4) ==== 43997+0+0 (secure 0 0 0) 0x7f2ce401a070 con 0x7f2cf006be90 2024-04-26T04:18:47.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.022+0000 7f2cd7fff640 1 --2- 172.21.15.63:0/2782847643 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f2cdc035e80 0x7f2cdc038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:47.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.022+0000 7f2cd7fff640 1 -- 172.21.15.63:0/2782847643 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 4346+0+0 (secure 0 0 0) 0x7f2ce40489f0 con 0x7f2cf006be90 2024-04-26T04:18:47.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.022+0000 7f2ceeffd640 1 --2- 172.21.15.63:0/2782847643 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f2cdc035e80 0x7f2cdc038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:47.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.022+0000 7f2ceeffd640 10 monclient: get_auth_request con 0x7f2cdc035e80 auth_method 0 2024-04-26T04:18:47.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.026+0000 7f2ceeffd640 1 --2- 172.21.15.63:0/2782847643 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f2cdc035e80 0x7f2cdc038340 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f2ce0007ca0 tx=0x7f2ce0007bf0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:47.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.026+0000 7f2cd7fff640 1 -- 172.21.15.63:0/2782847643 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f2ce4011350 con 0x7f2cf006be90 2024-04-26T04:18:47.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.030+0000 7f2cd7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:47.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.030+0000 7f2cd7fff640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:18:47.202 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.194+0000 7f2cf5d63640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-26T04:18:47.203 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.194+0000 7f2cf5d63640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-26T04:18:47.203 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.194+0000 7f2cf5d63640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:47.203 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.194+0000 7f2cf5d63640 1 -- 172.21.15.63:0/2782847643 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2cf0068260 con 0x7f2cf006be90 2024-04-26T04:18:47.203 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.194+0000 7f2cd7fff640 1 -- 172.21.15.63:0/2782847643 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) ==== 74+0+12258 (secure 0 0 0) 0x7f2ce401ad40 con 0x7f2cf006be90 2024-04-26T04:18:47.203 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.194+0000 7f2cd7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-26T04:18:47.203 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.194+0000 7f2cd7fff640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:18:47.204 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-26T04:18:47.204 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":14,"fsid":"cd1c6a0c-4e4b-4014-89c0-2e535479ef39","created":"2024-04-26T04:18:34.126231+0000","modified":"2024-04-26T04:18:46.485385+0000","last_up_change":"2024-04-26T04:18:44.470061+0000","last_in_change":"2024-04-26T04:18:36.862750+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T04:18:43.521296+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-26T04:18:45.211226+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d37141b-8282-4c62-864a-84a75ebf8f6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":2758152700},{"type":"v1","addr":"172.21.15.63:6801","nonce":2758152700}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":2758152700},{"type":"v1","addr":"172.21.15.63:6803","nonce":2758152700}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":2758152700},{"type":"v1","addr":"172.21.15.63:6807","nonce":2758152700}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":2758152700},{"type":"v1","addr":"172.21.15.63:6805","nonce":2758152700}]},"public_addr":"172.21.15.63:6801/2758152700","cluster_addr":"172.21.15.63:6803/2758152700","heartbeat_back_addr":"172.21.15.63:6807/2758152700","heartbeat_front_addr":"172.21.15.63:6805/2758152700","state":["exists","up"]},{"osd":1,"uuid":"7220e45a-72bb-4d8e-93c2-32005fa2c85e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":3596874741},{"type":"v1","addr":"172.21.15.63:6817","nonce":3596874741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":3596874741},{"type":"v1","addr":"172.21.15.63:6819","nonce":3596874741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":3596874741},{"type":"v1","addr":"172.21.15.63:6823","nonce":3596874741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":3596874741},{"type":"v1","addr":"172.21.15.63:6821","nonce":3596874741}]},"public_addr":"172.21.15.63:6817/3596874741","cluster_addr":"172.21.15.63:6819/3596874741","heartbeat_back_addr":"172.21.15.63:6823/3596874741","heartbeat_front_addr":"172.21.15.63:6821/3596874741","state":["exists","up"]},{"osd":2,"uuid":"913de4d8-31af-4394-a0ee-194cc31f822f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":413829082},{"type":"v1","addr":"172.21.15.63:6809","nonce":413829082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":413829082},{"type":"v1","addr":"172.21.15.63:6811","nonce":413829082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":413829082},{"type":"v1","addr":"172.21.15.63:6815","nonce":413829082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":413829082},{"type":"v1","addr":"172.21.15.63:6813","nonce":413829082}]},"public_addr":"172.21.15.63:6809/413829082","cluster_addr":"172.21.15.63:6811/413829082","heartbeat_back_addr":"172.21.15.63:6815/413829082","heartbeat_front_addr":"172.21.15.63:6813/413829082","state":["exists","up"]},{"osd":3,"uuid":"c492772c-ccf5-4b85-9056-6a5ee9686c0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6807","nonce":4227496981},{"type":"v1","addr":"172.21.15.186:6808","nonce":4227496981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4227496981},{"type":"v1","addr":"172.21.15.186:6811","nonce":4227496981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4227496981},{"type":"v1","addr":"172.21.15.186:6815","nonce":4227496981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4227496981},{"type":"v1","addr":"172.21.15.186:6813","nonce":4227496981}]},"public_addr":"172.21.15.186:6808/4227496981","cluster_addr":"172.21.15.186:6811/4227496981","heartbeat_back_addr":"172.21.15.186:6815/4227496981","heartbeat_front_addr":"172.21.15.186:6813/4227496981","state":["exists","up"]},{"osd":4,"uuid":"24f067f0-554a-4dd2-a26c-7bda3d0978f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":3746743567},{"type":"v1","addr":"172.21.15.186:6801","nonce":3746743567}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":3746743567},{"type":"v1","addr":"172.21.15.186:6803","nonce":3746743567}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":3746743567},{"type":"v1","addr":"172.21.15.186:6809","nonce":3746743567}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":3746743567},{"type":"v1","addr":"172.21.15.186:6805","nonce":3746743567}]},"public_addr":"172.21.15.186:6801/3746743567","cluster_addr":"172.21.15.186:6803/3746743567","heartbeat_back_addr":"172.21.15.186:6809/3746743567","heartbeat_front_addr":"172.21.15.186:6805/3746743567","state":["exists","up"]},{"osd":5,"uuid":"70454d09-e50c-4f65-83c5-0f9bb5c0f8f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":3109931567},{"type":"v1","addr":"172.21.15.186:6817","nonce":3109931567}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":3109931567},{"type":"v1","addr":"172.21.15.186:6819","nonce":3109931567}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":3109931567},{"type":"v1","addr":"172.21.15.186:6823","nonce":3109931567}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":3109931567},{"type":"v1","addr":"172.21.15.186:6821","nonce":3109931567}]},"public_addr":"172.21.15.186:6817/3109931567","cluster_addr":"172.21.15.186:6819/3109931567","heartbeat_back_addr":"172.21.15.186:6823/3109931567","heartbeat_front_addr":"172.21.15.186:6821/3109931567","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T04:18:42.278922+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T04:18:42.387279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T04:18:42.394303+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T04:18:42.367922+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T04:18:42.357914+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T04:18:42.415215+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T04:18:47.205 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.198+0000 7f2cf5d63640 1 -- 172.21.15.63:0/2782847643 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f2cdc035e80 msgr2=0x7f2cdc038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:47.205 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.198+0000 7f2cf5d63640 1 --2- 172.21.15.63:0/2782847643 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f2cdc035e80 0x7f2cdc038340 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f2ce0007ca0 tx=0x7f2ce0007bf0 comp rx=0 tx=0).stop 2024-04-26T04:18:47.205 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.198+0000 7f2cf5d63640 10 monclient: shutdown 2024-04-26T04:18:47.205 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.198+0000 7f2cf5d63640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:47.205 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.198+0000 7f2cf5d63640 1 -- 172.21.15.63:0/2782847643 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2cf006be90 msgr2=0x7f2cf01ced80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:47.205 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.198+0000 7f2cf5d63640 1 --2- 172.21.15.63:0/2782847643 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2cf006be90 0x7f2cf01ced80 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f2ce400e870 tx=0x7f2ce400ed30 comp rx=0 tx=0).stop 2024-04-26T04:18:47.205 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.198+0000 7f2cf5d63640 1 -- 172.21.15.63:0/2782847643 shutdown_connections 2024-04-26T04:18:47.205 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.198+0000 7f2cf5d63640 1 --2- 172.21.15.63:0/2782847643 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f2cdc035e80 0x7f2cdc038340 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:47.205 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.198+0000 7f2cf5d63640 1 --2- 172.21.15.63:0/2782847643 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2cf00a5200 0x7f2cf0139c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:47.205 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.198+0000 7f2cf5d63640 1 --2- 172.21.15.63:0/2782847643 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f2cf00a4e00 0x7f2cf0139700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:47.205 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.198+0000 7f2cf5d63640 1 --2- 172.21.15.63:0/2782847643 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2cf006be90 0x7f2cf01ced80 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:47.205 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.198+0000 7f2cf5d63640 1 -- 172.21.15.63:0/2782847643 >> 172.21.15.63:0/2782847643 conn(0x7f2cf00a60a0 msgr2=0x7f2cf00a0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:47.206 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.198+0000 7f2cf5d63640 1 -- 172.21.15.63:0/2782847643 shutdown_connections 2024-04-26T04:18:47.206 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.198+0000 7f2cf5d63640 1 -- 172.21.15.63:0/2782847643 wait complete. 2024-04-26T04:18:47.221 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-26T04:18:47.222 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T04:18:47.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.342+0000 7f45ce5a4640 10 monclient: get_monmap_and_config 2024-04-26T04:18:47.349 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.342+0000 7f45ce5a4640 10 monclient: build_initial_monmap 2024-04-26T04:18:47.349 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.342+0000 7f45ce5a4640 10 monclient: monmap: 2024-04-26T04:18:47.349 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:47.349 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:47.349 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:47.348759+0000 2024-04-26T04:18:47.349 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:47.348759+0000 2024-04-26T04:18:47.349 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:47.349 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:47.349 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:18:47.349 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:18:47.349 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:18:47.349 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:18:47.349 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:18:47.350 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:47.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.342+0000 7f45ce5a4640 1 Processor -- start 2024-04-26T04:18:47.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.346+0000 7f45ce5a4640 1 -- start start 2024-04-26T04:18:47.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.346+0000 7f45ce5a4640 10 monclient: init 2024-04-26T04:18:47.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.346+0000 7f45ce5a4640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:47.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.346+0000 7f45ce5a4640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-04-26T04:18:47.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.346+0000 7f45ce5a4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f45c80a2cd0 addr v1:172.21.15.63:6789/0 2024-04-26T04:18:47.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.346+0000 7f45ce5a4640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f45c80a30b0 0x7f45c813c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:47.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.346+0000 7f45ce5a4640 10 monclient(hunting): picked mon.noname-c con 0x7f45c80a30b0 addr v2:172.21.15.186:3300/0 2024-04-26T04:18:47.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.346+0000 7f45ce5a4640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f45c813c540 0x7f45c813e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:47.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.346+0000 7f45ce5a4640 10 monclient(hunting): picked mon.noname-b con 0x7f45c813c540 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:47.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.346+0000 7f45ce5a4640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45c80a4ee0 con 0x7f45c80a2cd0 2024-04-26T04:18:47.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.346+0000 7f45ce5a4640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:47.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.346+0000 7f45ce5a4640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f45c80a0fb0 con 0x7f45c80a30b0 2024-04-26T04:18:47.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.346+0000 7f45ce5a4640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:47.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.346+0000 7f45ce5a4640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f45c80a5380 con 0x7f45c813c540 2024-04-26T04:18:47.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.346+0000 7f45ce5a4640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:47.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.346+0000 7f45ce5a4640 10 monclient(hunting): authenticate will time out at 959.056949s 2024-04-26T04:18:47.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.346+0000 7f45c7fff640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f45c80a2cd0 0x7f45c80a47d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:44126/0 (socket says 172.21.15.63:44126) 2024-04-26T04:18:47.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.346+0000 7f45c7fff640 1 -- 172.21.15.63:0/2924546261 learned_addr learned my addr 172.21.15.63:0/2924546261 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-04-26T04:18:47.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.346+0000 7f45ccb41640 1 --2- 172.21.15.63:0/2924546261 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f45c813c540 0x7f45c813e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:47.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.346+0000 7f45ccb41640 10 monclient(hunting): get_auth_request con 0x7f45c813c540 auth_method 0 2024-04-26T04:18:47.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.346+0000 7f45ccb41640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:47.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.346+0000 7f45ccb41640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:47.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.346+0000 7f45ccb41640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:47.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.346+0000 7f45c77fe640 1 --2- 172.21.15.63:0/2924546261 >> v2:172.21.15.186:3300/0 conn(0x7f45c80a30b0 0x7f45c813c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:47.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.346+0000 7f45ccb41640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:47.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.346+0000 7f45ccb41640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:47.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.346+0000 7f45ccb41640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:47.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.346+0000 7f45c77fe640 10 monclient(hunting): get_auth_request con 0x7f45c80a30b0 auth_method 0 2024-04-26T04:18:47.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.346+0000 7f45c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:47.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.346+0000 7f45c77fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:47.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.346+0000 7f45c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:47.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.346+0000 7f45ccb41640 10 monclient(hunting): handle_auth_done global_id 4196 payload 274 2024-04-26T04:18:47.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.346+0000 7f45ccb41640 1 -- 172.21.15.63:0/2924546261 >> v2:172.21.15.186:3300/0 conn(0x7f45c80a30b0 msgr2=0x7f45c813c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:47.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.346+0000 7f45ccb41640 1 --2- 172.21.15.63:0/2924546261 >> v2:172.21.15.186:3300/0 conn(0x7f45c80a30b0 0x7f45c813c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:47.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.346+0000 7f45ccb41640 1 -- 172.21.15.63:0/2924546261 >> v1:172.21.15.63:6789/0 conn(0x7f45c80a2cd0 legacy=0x7f45c80a47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:47.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.346+0000 7f45ccb41640 10 monclient: _finish_hunting 0 2024-04-26T04:18:47.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.346+0000 7f45ccb41640 1 monclient: found mon.noname-b 2024-04-26T04:18:47.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.346+0000 7f45ccb41640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:47.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.346+0000 7f45ccb41640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.63:3301/0 2024-04-26T04:18:47.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.346+0000 7f45ccb41640 1 -- 172.21.15.63:0/2924546261 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45c80a1290 con 0x7f45c813c540 2024-04-26T04:18:47.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.346+0000 7f45c77fe640 1 --2- 172.21.15.63:0/2924546261 >> v2:172.21.15.186:3300/0 conn(0x7f45c80a30b0 0x7f45c813c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T04:18:47.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.350+0000 7f45ccb41640 1 -- 172.21.15.63:0/2924546261 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f45c813c540 msgr2=0x7f45c813e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:47.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.350+0000 7f45ccb41640 1 --2- 172.21.15.63:0/2924546261 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f45c813c540 0x7f45c813e930 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f45c00097c0 tx=0x7f45c001c3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2fbd0986888548a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:47.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.350+0000 7f45c6ffd640 1 -- 172.21.15.63:0/2924546261 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c00b0070 con 0x7f45c813c540 2024-04-26T04:18:47.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.350+0000 7f45c6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:47.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.350+0000 7f45c6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T04:18:47.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.350+0000 7f45c6ffd640 10 monclient: dump: 2024-04-26T04:18:47.360 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:47.360 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:47.360 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:47.360 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:47.360 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:47.360 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:47.361 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:47.361 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:47.361 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:47.361 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:47.361 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.350+0000 7f45c6ffd640 10 monclient: _finish_auth 0 2024-04-26T04:18:47.361 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.350+0000 7f45c6ffd640 10 monclient: _check_auth_tickets 2024-04-26T04:18:47.361 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.350+0000 7f45c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:47.361 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.350+0000 7f45ce5a4640 5 monclient: authenticate success, global_id 4196 2024-04-26T04:18:47.361 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.350+0000 7f45ce5a4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T04:18:47.361 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.350+0000 7f45c6ffd640 1 -- 172.21.15.63:0/2924546261 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45c00ab600 con 0x7f45c813c540 2024-04-26T04:18:47.361 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.350+0000 7f45c6ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:47.361 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.350+0000 7f45ce5a4640 10 monclient: get_monmap_and_config success 2024-04-26T04:18:47.361 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.350+0000 7f45ce5a4640 4 set_mon_vals no callback set 2024-04-26T04:18:47.361 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.350+0000 7f45c6ffd640 1 -- 172.21.15.63:0/2924546261 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c00ab9d0 con 0x7f45c813c540 2024-04-26T04:18:47.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.350+0000 7f45c6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:47.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.350+0000 7f45c6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:18:47.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.350+0000 7f45c6ffd640 10 monclient: dump: 2024-04-26T04:18:47.362 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:47.362 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:47.362 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:47.362 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:47.362 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:47.362 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:47.362 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:47.362 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:47.362 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:47.362 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:47.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.350+0000 7f45ce5a4640 10 monclient: shutdown 2024-04-26T04:18:47.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.350+0000 7f45ce5a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:47.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.350+0000 7f45ce5a4640 1 -- 172.21.15.63:0/2924546261 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f45c813c540 msgr2=0x7f45c813e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:47.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.350+0000 7f45ce5a4640 1 --2- 172.21.15.63:0/2924546261 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f45c813c540 0x7f45c813e930 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f45c00097c0 tx=0x7f45c001c3b0 comp rx=0 tx=0).stop 2024-04-26T04:18:47.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.350+0000 7f45ce5a4640 1 -- 172.21.15.63:0/2924546261 shutdown_connections 2024-04-26T04:18:47.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.350+0000 7f45ce5a4640 1 --2- 172.21.15.63:0/2924546261 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f45c813c540 0x7f45c813e930 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:47.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.350+0000 7f45ce5a4640 1 --2- 172.21.15.63:0/2924546261 >> v2:172.21.15.186:3300/0 conn(0x7f45c80a30b0 0x7f45c813c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:47.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.350+0000 7f45ce5a4640 1 -- 172.21.15.63:0/2924546261 >> 172.21.15.63:0/2924546261 conn(0x7f45c80a5f40 msgr2=0x7f45c80a6320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:47.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.350+0000 7f45ce5a4640 1 -- 172.21.15.63:0/2924546261 shutdown_connections 2024-04-26T04:18:47.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.350+0000 7f45ce5a4640 1 -- 172.21.15.63:0/2924546261 wait complete. 2024-04-26T04:18:47.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.350+0000 7f45ce5a4640 10 monclient: build_initial_monmap 2024-04-26T04:18:47.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.350+0000 7f45ce5a4640 10 monclient: monmap: 2024-04-26T04:18:47.363 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:47.363 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:18:47.363 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:18:47.363 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:18:47.364 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:47.364 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:47.364 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:18:47.364 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:18:47.364 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:18:47.364 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:47.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.350+0000 7f45ce5a4640 1 Processor -- start 2024-04-26T04:18:47.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.350+0000 7f45ce5a4640 1 -- start start 2024-04-26T04:18:47.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.350+0000 7f45ce5a4640 10 monclient: init 2024-04-26T04:18:47.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.350+0000 7f45ce5a4640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:47.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.350+0000 7f45ce5a4640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T04:18:47.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.350+0000 7f45ce5a4640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f45c80a2cd0 0x7f45c8135a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:47.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.350+0000 7f45ce5a4640 10 monclient(hunting): picked mon.noname-a con 0x7f45c80a2cd0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:18:47.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.354+0000 7f45ce5a4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45c80a30b0 0x7f45c806fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:47.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.354+0000 7f45ce5a4640 10 monclient(hunting): picked mon.noname-b con 0x7f45c80a30b0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:18:47.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.354+0000 7f45ce5a4640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f45c813c540 0x7f45c8070340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:47.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.354+0000 7f45ce5a4640 10 monclient(hunting): picked mon.noname-c con 0x7f45c813c540 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:47.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.354+0000 7f45ce5a4640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:47.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.354+0000 7f45ce5a4640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f45c8141a20 con 0x7f45c80a2cd0 2024-04-26T04:18:47.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.354+0000 7f45ce5a4640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:47.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.354+0000 7f45ce5a4640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f45c81418a0 con 0x7f45c80a30b0 2024-04-26T04:18:47.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.354+0000 7f45ce5a4640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:47.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.354+0000 7f45ce5a4640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f45c8141ba0 con 0x7f45c813c540 2024-04-26T04:18:47.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.354+0000 7f45ce5a4640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:47.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.354+0000 7f45ce5a4640 10 monclient(hunting): authenticate will time out at 959.064064s 2024-04-26T04:18:47.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.354+0000 7f45c7fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f45c80a2cd0 0x7f45c8135a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:47.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.354+0000 7f45ccb41640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f45c813c540 0x7f45c8070340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:47.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.354+0000 7f45c7fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f45c80a2cd0 0x7f45c8135a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:46378/0 (socket says 172.21.15.63:46378) 2024-04-26T04:18:47.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.354+0000 7f45c7fff640 1 -- 172.21.15.63:0/3281242247 learned_addr learned my addr 172.21.15.63:0/3281242247 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:18:47.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.354+0000 7f45c7fff640 10 monclient(hunting): get_auth_request con 0x7f45c80a2cd0 auth_method 0 2024-04-26T04:18:47.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.354+0000 7f45c7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:47.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.354+0000 7f45c7fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:47.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.354+0000 7f45c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:47.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.354+0000 7f45ccb41640 10 monclient(hunting): get_auth_request con 0x7f45c813c540 auth_method 0 2024-04-26T04:18:47.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.354+0000 7f45ccb41640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:47.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.354+0000 7f45ccb41640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:47.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.354+0000 7f45ccb41640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:47.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.354+0000 7f45c7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:47.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.354+0000 7f45c7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:47.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.354+0000 7f45c7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:47.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.354+0000 7f45ccb41640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:47.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.354+0000 7f45ccb41640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:47.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.354+0000 7f45ccb41640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:47.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.354+0000 7f45c77fe640 1 --2- 172.21.15.63:0/3281242247 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45c80a30b0 0x7f45c806fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:47.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.354+0000 7f45c77fe640 10 monclient(hunting): get_auth_request con 0x7f45c80a30b0 auth_method 0 2024-04-26T04:18:47.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.354+0000 7f45c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:47.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.354+0000 7f45c77fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:47.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.354+0000 7f45c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:47.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.354+0000 7f45c7fff640 10 monclient(hunting): handle_auth_done global_id 4203 payload 824 2024-04-26T04:18:47.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.354+0000 7f45c7fff640 1 -- 172.21.15.63:0/3281242247 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f45c813c540 msgr2=0x7f45c8070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:47.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.354+0000 7f45c7fff640 1 --2- 172.21.15.63:0/3281242247 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f45c813c540 0x7f45c8070340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:47.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.354+0000 7f45c7fff640 1 -- 172.21.15.63:0/3281242247 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45c80a30b0 msgr2=0x7f45c806fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:47.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.354+0000 7f45c7fff640 1 --2- 172.21.15.63:0/3281242247 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45c80a30b0 0x7f45c806fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:47.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.354+0000 7f45c7fff640 10 monclient: _finish_hunting 0 2024-04-26T04:18:47.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.354+0000 7f45c7fff640 1 monclient: found mon.noname-a 2024-04-26T04:18:47.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.354+0000 7f45c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:47.368 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.354+0000 7f45c7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:18:47.368 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.354+0000 7f45c7fff640 1 -- 172.21.15.63:0/3281242247 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45c8120390 con 0x7f45c80a2cd0 2024-04-26T04:18:47.368 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.354+0000 7f45ccb41640 1 --2- 172.21.15.63:0/3281242247 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f45c813c540 0x7f45c8070340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T04:18:47.368 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.354+0000 7f45c77fe640 1 --2- 172.21.15.63:0/3281242247 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45c80a30b0 0x7f45c806fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T04:18:47.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.354+0000 7f45c7fff640 1 -- 172.21.15.63:0/3281242247 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f45c80a2cd0 msgr2=0x7f45c8135a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:47.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.354+0000 7f45c7fff640 1 --2- 172.21.15.63:0/3281242247 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f45c80a2cd0 0x7f45c8135a00 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f45b800cc40 tx=0x7f45b800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:47.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.354+0000 7f45c4ff9640 1 -- 172.21.15.63:0/3281242247 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45b8012400 con 0x7f45c80a2cd0 2024-04-26T04:18:47.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.354+0000 7f45c4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:47.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.354+0000 7f45c4ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:18:47.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.354+0000 7f45c4ff9640 10 monclient: dump: 2024-04-26T04:18:47.371 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:47.371 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:47.371 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:47.371 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:47.371 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:47.371 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:47.371 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:47.371 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:47.371 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:47.371 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:47.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.354+0000 7f45c4ff9640 10 monclient: _finish_auth 0 2024-04-26T04:18:47.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.354+0000 7f45c4ff9640 10 monclient: _check_auth_tickets 2024-04-26T04:18:47.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.354+0000 7f45c4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:47.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.354+0000 7f45ce5a4640 5 monclient: authenticate success, global_id 4203 2024-04-26T04:18:47.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.354+0000 7f45ce5a4640 10 monclient: _renew_subs 2024-04-26T04:18:47.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.354+0000 7f45ce5a4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:47.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.354+0000 7f45ce5a4640 1 -- 172.21.15.63:0/3281242247 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45c8072380 con 0x7f45c80a2cd0 2024-04-26T04:18:47.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.354+0000 7f45c4ff9640 1 -- 172.21.15.63:0/3281242247 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45b80125a0 con 0x7f45c80a2cd0 2024-04-26T04:18:47.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.354+0000 7f45c4ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:47.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.354+0000 7f45c4ff9640 1 -- 172.21.15.63:0/3281242247 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45b8012990 con 0x7f45c80a2cd0 2024-04-26T04:18:47.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.354+0000 7f45c4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:47.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.354+0000 7f45c4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:18:47.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.354+0000 7f45c4ff9640 10 monclient: dump: 2024-04-26T04:18:47.372 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:47.372 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:47.372 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:47.372 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:47.373 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:47.373 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:47.373 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:47.373 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:47.373 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:47.373 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:47.373 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.354+0000 7f45c67fc640 4 set_mon_vals no callback set 2024-04-26T04:18:47.373 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.354+0000 7f45ce5a4640 10 monclient: _renew_subs 2024-04-26T04:18:47.373 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.354+0000 7f45ce5a4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:47.373 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.354+0000 7f45ce5a4640 1 -- 172.21.15.63:0/3281242247 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45c80a37d0 con 0x7f45c80a2cd0 2024-04-26T04:18:47.373 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.358+0000 7f45ce5a4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:47.373 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.358+0000 7f45ce5a4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:47.373 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.358+0000 7f45ce5a4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:47.373 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.358+0000 7f45ce5a4640 1 -- 172.21.15.63:0/3281242247 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45c8062e10 con 0x7f45c80a2cd0 2024-04-26T04:18:47.373 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.358+0000 7f45c4ff9640 1 -- 172.21.15.63:0/3281242247 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 4) ==== 43997+0+0 (secure 0 0 0) 0x7f45b8012bb0 con 0x7f45c80a2cd0 2024-04-26T04:18:47.374 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.358+0000 7f45c4ff9640 1 --2- 172.21.15.63:0/3281242247 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f45b4035e30 0x7f45b40382f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:47.374 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.358+0000 7f45c4ff9640 1 -- 172.21.15.63:0/3281242247 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 4346+0+0 (secure 0 0 0) 0x7f45b8049d90 con 0x7f45c80a2cd0 2024-04-26T04:18:47.374 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.358+0000 7f45c77fe640 1 --2- 172.21.15.63:0/3281242247 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f45b4035e30 0x7f45b40382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:47.374 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.358+0000 7f45c77fe640 10 monclient: get_auth_request con 0x7f45b4035e30 auth_method 0 2024-04-26T04:18:47.374 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.362+0000 7f45c77fe640 1 --2- 172.21.15.63:0/3281242247 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f45b4035e30 0x7f45b40382f0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f45c8071650 tx=0x7f45bc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:47.374 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.362+0000 7f45c4ff9640 1 -- 172.21.15.63:0/3281242247 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f45b8025e10 con 0x7f45c80a2cd0 2024-04-26T04:18:47.374 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.362+0000 7f45c4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:47.374 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.362+0000 7f45c4ff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:18:47.534 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.530+0000 7f45ce5a4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-26T04:18:47.534 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.530+0000 7f45ce5a4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-26T04:18:47.534 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.530+0000 7f45ce5a4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:47.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.530+0000 7f45ce5a4640 1 -- 172.21.15.63:0/3281242247 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f45c80683d0 con 0x7f45c80a2cd0 2024-04-26T04:18:47.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.530+0000 7f45c4ff9640 1 -- 172.21.15.63:0/3281242247 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) ==== 74+0+12245 (secure 0 0 0) 0x7f45b8047a70 con 0x7f45c80a2cd0 2024-04-26T04:18:47.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.530+0000 7f45c4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-26T04:18:47.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.530+0000 7f45c4ff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:18:47.536 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-26T04:18:47.536 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":15,"fsid":"cd1c6a0c-4e4b-4014-89c0-2e535479ef39","created":"2024-04-26T04:18:34.126231+0000","modified":"2024-04-26T04:18:47.487705+0000","last_up_change":"2024-04-26T04:18:44.470061+0000","last_in_change":"2024-04-26T04:18:36.862750+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T04:18:43.521296+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-26T04:18:45.211226+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d37141b-8282-4c62-864a-84a75ebf8f6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":2758152700},{"type":"v1","addr":"172.21.15.63:6801","nonce":2758152700}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":2758152700},{"type":"v1","addr":"172.21.15.63:6803","nonce":2758152700}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":2758152700},{"type":"v1","addr":"172.21.15.63:6807","nonce":2758152700}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":2758152700},{"type":"v1","addr":"172.21.15.63:6805","nonce":2758152700}]},"public_addr":"172.21.15.63:6801/2758152700","cluster_addr":"172.21.15.63:6803/2758152700","heartbeat_back_addr":"172.21.15.63:6807/2758152700","heartbeat_front_addr":"172.21.15.63:6805/2758152700","state":["exists","up"]},{"osd":1,"uuid":"7220e45a-72bb-4d8e-93c2-32005fa2c85e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":3596874741},{"type":"v1","addr":"172.21.15.63:6817","nonce":3596874741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":3596874741},{"type":"v1","addr":"172.21.15.63:6819","nonce":3596874741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":3596874741},{"type":"v1","addr":"172.21.15.63:6823","nonce":3596874741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":3596874741},{"type":"v1","addr":"172.21.15.63:6821","nonce":3596874741}]},"public_addr":"172.21.15.63:6817/3596874741","cluster_addr":"172.21.15.63:6819/3596874741","heartbeat_back_addr":"172.21.15.63:6823/3596874741","heartbeat_front_addr":"172.21.15.63:6821/3596874741","state":["exists","up"]},{"osd":2,"uuid":"913de4d8-31af-4394-a0ee-194cc31f822f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":413829082},{"type":"v1","addr":"172.21.15.63:6809","nonce":413829082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":413829082},{"type":"v1","addr":"172.21.15.63:6811","nonce":413829082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":413829082},{"type":"v1","addr":"172.21.15.63:6815","nonce":413829082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":413829082},{"type":"v1","addr":"172.21.15.63:6813","nonce":413829082}]},"public_addr":"172.21.15.63:6809/413829082","cluster_addr":"172.21.15.63:6811/413829082","heartbeat_back_addr":"172.21.15.63:6815/413829082","heartbeat_front_addr":"172.21.15.63:6813/413829082","state":["exists","up"]},{"osd":3,"uuid":"c492772c-ccf5-4b85-9056-6a5ee9686c0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6807","nonce":4227496981},{"type":"v1","addr":"172.21.15.186:6808","nonce":4227496981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4227496981},{"type":"v1","addr":"172.21.15.186:6811","nonce":4227496981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4227496981},{"type":"v1","addr":"172.21.15.186:6815","nonce":4227496981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4227496981},{"type":"v1","addr":"172.21.15.186:6813","nonce":4227496981}]},"public_addr":"172.21.15.186:6808/4227496981","cluster_addr":"172.21.15.186:6811/4227496981","heartbeat_back_addr":"172.21.15.186:6815/4227496981","heartbeat_front_addr":"172.21.15.186:6813/4227496981","state":["exists","up"]},{"osd":4,"uuid":"24f067f0-554a-4dd2-a26c-7bda3d0978f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":3746743567},{"type":"v1","addr":"172.21.15.186:6801","nonce":3746743567}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":3746743567},{"type":"v1","addr":"172.21.15.186:6803","nonce":3746743567}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":3746743567},{"type":"v1","addr":"172.21.15.186:6809","nonce":3746743567}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":3746743567},{"type":"v1","addr":"172.21.15.186:6805","nonce":3746743567}]},"public_addr":"172.21.15.186:6801/3746743567","cluster_addr":"172.21.15.186:6803/3746743567","heartbeat_back_addr":"172.21.15.186:6809/3746743567","heartbeat_front_addr":"172.21.15.186:6805/3746743567","state":["exists","up"]},{"osd":5,"uuid":"70454d09-e50c-4f65-83c5-0f9bb5c0f8f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":3109931567},{"type":"v1","addr":"172.21.15.186:6817","nonce":3109931567}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":3109931567},{"type":"v1","addr":"172.21.15.186:6819","nonce":3109931567}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":3109931567},{"type":"v1","addr":"172.21.15.186:6823","nonce":3109931567}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":3109931567},{"type":"v1","addr":"172.21.15.186:6821","nonce":3109931567}]},"public_addr":"172.21.15.186:6817/3109931567","cluster_addr":"172.21.15.186:6819/3109931567","heartbeat_back_addr":"172.21.15.186:6823/3109931567","heartbeat_front_addr":"172.21.15.186:6821/3109931567","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T04:18:42.278922+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T04:18:42.387279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T04:18:42.394303+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T04:18:42.367922+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T04:18:42.357914+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T04:18:42.415215+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T04:18:47.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.530+0000 7f45ce5a4640 1 -- 172.21.15.63:0/3281242247 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f45b4035e30 msgr2=0x7f45b40382f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:47.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.530+0000 7f45ce5a4640 1 --2- 172.21.15.63:0/3281242247 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f45b4035e30 0x7f45b40382f0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f45c8071650 tx=0x7f45bc008040 comp rx=0 tx=0).stop 2024-04-26T04:18:47.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.530+0000 7f45ce5a4640 10 monclient: shutdown 2024-04-26T04:18:47.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.530+0000 7f45ce5a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:47.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.530+0000 7f45ce5a4640 1 -- 172.21.15.63:0/3281242247 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f45c80a2cd0 msgr2=0x7f45c8135a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:47.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.530+0000 7f45ce5a4640 1 --2- 172.21.15.63:0/3281242247 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f45c80a2cd0 0x7f45c8135a00 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f45b800cc40 tx=0x7f45b800a430 comp rx=0 tx=0).stop 2024-04-26T04:18:47.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.530+0000 7f45ce5a4640 1 -- 172.21.15.63:0/3281242247 shutdown_connections 2024-04-26T04:18:47.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.530+0000 7f45ce5a4640 1 --2- 172.21.15.63:0/3281242247 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f45b4035e30 0x7f45b40382f0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:47.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.530+0000 7f45ce5a4640 1 --2- 172.21.15.63:0/3281242247 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f45c813c540 0x7f45c8070340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:47.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.530+0000 7f45ce5a4640 1 --2- 172.21.15.63:0/3281242247 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45c80a30b0 0x7f45c806fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:47.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.530+0000 7f45ce5a4640 1 --2- 172.21.15.63:0/3281242247 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f45c80a2cd0 0x7f45c8135a00 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:47.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.530+0000 7f45ce5a4640 1 -- 172.21.15.63:0/3281242247 >> 172.21.15.63:0/3281242247 conn(0x7f45c80a5f40 msgr2=0x7f45c80a1a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:47.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.534+0000 7f45ce5a4640 1 -- 172.21.15.63:0/3281242247 shutdown_connections 2024-04-26T04:18:47.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.534+0000 7f45ce5a4640 1 -- 172.21.15.63:0/3281242247 wait complete. 2024-04-26T04:18:47.555 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-26T04:18:47.557 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-26T04:18:47.557 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-26T04:18:47.558 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-26T04:18:47.558 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-04-26T04:18:47.559 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-04-26T04:18:47.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7fbd6e941640 10 monclient: get_monmap_and_config 2024-04-26T04:18:47.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7f84da408640 10 monclient: get_monmap_and_config 2024-04-26T04:18:47.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7f84da408640 10 monclient: build_initial_monmap 2024-04-26T04:18:47.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7f84da408640 10 monclient: monmap: 2024-04-26T04:18:47.679 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:47.680 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:47.680 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:47.678528+0000 2024-04-26T04:18:47.680 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:47.678528+0000 2024-04-26T04:18:47.680 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:47.680 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:47.680 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:18:47.681 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:18:47.681 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:18:47.681 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:18:47.682 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:18:47.682 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:47.683 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7fbd6e941640 10 monclient: build_initial_monmap 2024-04-26T04:18:47.683 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7fbd6e941640 10 monclient: monmap: 2024-04-26T04:18:47.683 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:47.683 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:47.683 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:47.678477+0000 2024-04-26T04:18:47.683 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:47.678477+0000 2024-04-26T04:18:47.684 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:47.685 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:47.685 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:18:47.685 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:18:47.685 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:18:47.686 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:18:47.686 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:18:47.686 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:47.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7f84da408640 1 Processor -- start 2024-04-26T04:18:47.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7f84da408640 1 -- start start 2024-04-26T04:18:47.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7f84da408640 10 monclient: init 2024-04-26T04:18:47.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7f84da408640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:47.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7f84da408640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-04-26T04:18:47.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7f84da408640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f84d40a2cd0 0x7f84d40a30b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:47.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7f84da408640 10 monclient(hunting): picked mon.noname-a con 0x7f84d40a2cd0 addr v2:172.21.15.63:3300/0 2024-04-26T04:18:47.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7f84da408640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f84d40a47d0 addr v1:172.21.15.63:6789/0 2024-04-26T04:18:47.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7f84da408640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f84d40a4c00 addr v1:172.21.15.186:6789/0 2024-04-26T04:18:47.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7f84da408640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84d4137b00 con 0x7f84d40a47d0 2024-04-26T04:18:47.690 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7f84da408640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84d40a0fb0 con 0x7f84d40a4c00 2024-04-26T04:18:47.690 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7f84da408640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:47.690 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7f84da408640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7f84d40a35f0 con 0x7f84d40a2cd0 2024-04-26T04:18:47.690 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7f84da408640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:47.690 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7f84da408640 10 monclient(hunting): authenticate will time out at 959.386019s 2024-04-26T04:18:47.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7f84d3fff640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f84d40a2cd0 0x7f84d40a30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:47.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7f84d3fff640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f84d40a2cd0 0x7f84d40a30b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:46380/0 (socket says 172.21.15.63:46380) 2024-04-26T04:18:47.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7f84d3fff640 1 -- 172.21.15.63:0/2689054401 learned_addr learned my addr 172.21.15.63:0/2689054401 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:18:47.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7f84d3fff640 10 monclient(hunting): get_auth_request con 0x7f84d40a2cd0 auth_method 0 2024-04-26T04:18:47.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7f84d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:47.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7f84d3fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:47.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7f84d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:47.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7fbd6e941640 1 Processor -- start 2024-04-26T04:18:47.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7f84d3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:47.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7f84d3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:47.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7f84d3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:47.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7fbd6e941640 1 -- start start 2024-04-26T04:18:47.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7fbd6e941640 10 monclient: init 2024-04-26T04:18:47.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7fbd6e941640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:47.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7fbd6e941640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-04-26T04:18:47.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7fbd6e941640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7fbd680a2cd0 0x7fbd680a30b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:47.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7fbd6e941640 10 monclient(hunting): picked mon.noname-a con 0x7fbd680a2cd0 addr v2:172.21.15.63:3300/0 2024-04-26T04:18:47.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7fbd6e941640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbd680a47d0 addr v1:172.21.15.63:6789/0 2024-04-26T04:18:47.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7fbd6e941640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fbd680a4c00 0x7fbd681373f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:47.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7fbd6e941640 10 monclient(hunting): picked mon.noname-c con 0x7fbd680a4c00 addr v2:172.21.15.186:3300/0 2024-04-26T04:18:47.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7fbd6e941640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd680a0fb0 con 0x7fbd680a47d0 2024-04-26T04:18:47.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7fbd6e941640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:47.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7fbd6e941640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7fbd680a35f0 con 0x7fbd680a2cd0 2024-04-26T04:18:47.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7fbd6e941640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:47.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7fbd6e941640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fbd680a5380 con 0x7fbd680a4c00 2024-04-26T04:18:47.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7fbd6e941640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:47.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7fbd6e941640 10 monclient(hunting): authenticate will time out at 959.387077s 2024-04-26T04:18:47.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7fbd67fff640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7fbd680a2cd0 0x7fbd680a30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:47.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7fbd67fff640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7fbd680a2cd0 0x7fbd680a30b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:46390/0 (socket says 172.21.15.63:46390) 2024-04-26T04:18:47.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7fbd67fff640 1 -- 172.21.15.63:0/45622161 learned_addr learned my addr 172.21.15.63:0/45622161 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:18:47.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7fbd67fff640 10 monclient(hunting): get_auth_request con 0x7fbd680a2cd0 auth_method 0 2024-04-26T04:18:47.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7fbd67fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:47.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7fbd67fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:47.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7fbd67fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:47.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7f84d3fff640 10 monclient(hunting): handle_auth_done global_id 4206 payload 274 2024-04-26T04:18:47.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7f84d3fff640 1 -- 172.21.15.63:0/2689054401 >> v1:172.21.15.186:6789/0 conn(0x7f84d40a4c00 legacy=0x7f84d41373f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:47.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7f84d3fff640 1 -- 172.21.15.63:0/2689054401 >> v1:172.21.15.63:6789/0 conn(0x7f84d40a47d0 legacy=0x7f84d4136ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:47.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7f84d3fff640 10 monclient: _finish_hunting 0 2024-04-26T04:18:47.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7f84d3fff640 1 monclient: found mon.noname-a 2024-04-26T04:18:47.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7f84d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:47.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7f84d3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:18:47.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7f84d3fff640 1 -- 172.21.15.63:0/2689054401 --> v2:172.21.15.63:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84d40a5380 con 0x7f84d40a2cd0 2024-04-26T04:18:47.702 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7fbd67fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:47.702 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7fbd67fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:47.702 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7fbd67fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:47.702 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7fbd67fff640 10 monclient(hunting): handle_auth_done global_id 4209 payload 274 2024-04-26T04:18:47.702 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7fbd67fff640 1 -- 172.21.15.63:0/45622161 >> v2:172.21.15.186:3300/0 conn(0x7fbd680a4c00 msgr2=0x7fbd681373f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:47.703 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7fbd67fff640 1 --2- 172.21.15.63:0/45622161 >> v2:172.21.15.186:3300/0 conn(0x7fbd680a4c00 0x7fbd681373f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:47.703 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7fbd67fff640 1 -- 172.21.15.63:0/45622161 >> v1:172.21.15.63:6789/0 conn(0x7fbd680a47d0 legacy=0x7fbd68136ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:47.703 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7fbd67fff640 10 monclient: _finish_hunting 0 2024-04-26T04:18:47.703 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7fbd67fff640 1 monclient: found mon.noname-a 2024-04-26T04:18:47.703 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7fbd67fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:47.703 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7fbd67fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:18:47.703 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.674+0000 7fbd67fff640 1 -- 172.21.15.63:0/45622161 --> v2:172.21.15.63:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd680a1290 con 0x7fbd680a2cd0 2024-04-26T04:18:47.703 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.682+0000 7fbd67fff640 1 -- 172.21.15.63:0/45622161 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbd680a2cd0 msgr2=0x7fbd680a30b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:47.703 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.682+0000 7fbd67fff640 1 --2- 172.21.15.63:0/45622161 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbd680a2cd0 0x7fbd680a30b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fbd68068310 tx=0x7fbd5801c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a7f8a624cb07fef server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:47.704 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.682+0000 7f84d3fff640 1 -- 172.21.15.63:0/2689054401 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f84d40a2cd0 msgr2=0x7f84d40a30b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:47.704 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.682+0000 7f84d3fff640 1 --2- 172.21.15.63:0/2689054401 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f84d40a2cd0 0x7f84d40a30b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f84c4009870 tx=0x7f84c401c4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96debba416698fb4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:47.704 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.682+0000 7fbd66ffd640 1 -- 172.21.15.63:0/45622161 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd580a5650 con 0x7fbd680a2cd0 2024-04-26T04:18:47.704 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.682+0000 7fbd66ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:47.704 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.682+0000 7fbd66ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:18:47.704 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.682+0000 7fbd66ffd640 10 monclient: dump: 2024-04-26T04:18:47.705 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:47.705 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:47.705 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:47.705 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:47.705 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:47.705 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:47.705 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:47.705 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:47.705 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:47.705 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:47.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.682+0000 7fbd66ffd640 10 monclient: _finish_auth 0 2024-04-26T04:18:47.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.682+0000 7fbd66ffd640 10 monclient: _check_auth_tickets 2024-04-26T04:18:47.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.682+0000 7f84d2ffd640 1 -- 172.21.15.63:0/2689054401 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84c40b1070 con 0x7f84d40a2cd0 2024-04-26T04:18:47.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.682+0000 7f84d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:47.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.682+0000 7f84d2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:18:47.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.682+0000 7f84d2ffd640 10 monclient: dump: 2024-04-26T04:18:47.706 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:47.706 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:47.707 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:47.707 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:47.707 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:47.707 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:47.707 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:47.707 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:47.707 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:47.707 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:47.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.682+0000 7f84d2ffd640 10 monclient: _finish_auth 0 2024-04-26T04:18:47.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.682+0000 7fbd66ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:47.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.682+0000 7fbd6e941640 5 monclient: authenticate success, global_id 4209 2024-04-26T04:18:47.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.682+0000 7fbd6e941640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T04:18:47.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.682+0000 7f84d2ffd640 10 monclient: _check_auth_tickets 2024-04-26T04:18:47.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.682+0000 7f84d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:47.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.682+0000 7f84d2ffd640 1 -- 172.21.15.63:0/2689054401 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84c40ac630 con 0x7f84d40a2cd0 2024-04-26T04:18:47.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.682+0000 7f84d2ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:47.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.682+0000 7f84d2ffd640 1 -- 172.21.15.63:0/2689054401 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84c40a8050 con 0x7f84d40a2cd0 2024-04-26T04:18:47.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.682+0000 7f84da408640 5 monclient: authenticate success, global_id 4206 2024-04-26T04:18:47.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.682+0000 7f84da408640 10 monclient: get_monmap_and_config success 2024-04-26T04:18:47.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.682+0000 7f84da408640 4 set_mon_vals no callback set 2024-04-26T04:18:47.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.682+0000 7f84d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:47.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.682+0000 7f84d2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:18:47.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.682+0000 7f84d2ffd640 10 monclient: dump: 2024-04-26T04:18:47.709 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:47.709 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:47.709 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:47.709 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:47.709 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:47.710 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:47.710 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:47.711 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:47.711 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:47.711 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:47.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.682+0000 7fbd66ffd640 1 -- 172.21.15.63:0/45622161 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd580a57f0 con 0x7fbd680a2cd0 2024-04-26T04:18:47.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.682+0000 7fbd66ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:47.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.682+0000 7f84da408640 10 monclient: shutdown 2024-04-26T04:18:47.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.682+0000 7f84da408640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:47.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.682+0000 7f84da408640 1 -- 172.21.15.63:0/2689054401 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f84d40a2cd0 msgr2=0x7f84d40a30b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:47.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.682+0000 7f84da408640 1 --2- 172.21.15.63:0/2689054401 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f84d40a2cd0 0x7f84d40a30b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f84c4009870 tx=0x7f84c401c4c0 comp rx=0 tx=0).stop 2024-04-26T04:18:47.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.682+0000 7fbd6e941640 10 monclient: get_monmap_and_config success 2024-04-26T04:18:47.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.682+0000 7fbd6e941640 4 set_mon_vals no callback set 2024-04-26T04:18:47.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.682+0000 7fbd6e941640 10 monclient: shutdown 2024-04-26T04:18:47.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.682+0000 7fbd6e941640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:47.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.682+0000 7fbd6e941640 1 -- 172.21.15.63:0/45622161 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbd680a2cd0 msgr2=0x7fbd680a30b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:47.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.682+0000 7fbd6e941640 1 --2- 172.21.15.63:0/45622161 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbd680a2cd0 0x7fbd680a30b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fbd68068310 tx=0x7fbd5801c990 comp rx=0 tx=0).stop 2024-04-26T04:18:47.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.682+0000 7fbd6e941640 1 -- 172.21.15.63:0/45622161 shutdown_connections 2024-04-26T04:18:47.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.682+0000 7fbd6e941640 1 --2- 172.21.15.63:0/45622161 >> v2:172.21.15.186:3300/0 conn(0x7fbd680a4c00 0x7fbd681373f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:47.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.682+0000 7fbd6e941640 1 --2- 172.21.15.63:0/45622161 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbd680a2cd0 0x7fbd680a30b0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:47.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.682+0000 7fbd6e941640 1 -- 172.21.15.63:0/45622161 >> 172.21.15.63:0/45622161 conn(0x7fbd680a5f40 msgr2=0x7fbd680a6320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:47.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.682+0000 7f84da408640 1 -- 172.21.15.63:0/2689054401 shutdown_connections 2024-04-26T04:18:47.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.682+0000 7f84da408640 1 --2- 172.21.15.63:0/2689054401 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f84d40a2cd0 0x7f84d40a30b0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:47.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.682+0000 7f84da408640 1 -- 172.21.15.63:0/2689054401 >> 172.21.15.63:0/2689054401 conn(0x7f84d40a5f40 msgr2=0x7f84d40a6320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:47.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.682+0000 7f84da408640 1 -- 172.21.15.63:0/2689054401 shutdown_connections 2024-04-26T04:18:47.714 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.682+0000 7fbd6e941640 1 -- 172.21.15.63:0/45622161 shutdown_connections 2024-04-26T04:18:47.714 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.682+0000 7f84da408640 1 -- 172.21.15.63:0/2689054401 wait complete. 2024-04-26T04:18:47.714 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.682+0000 7fbd6e941640 1 -- 172.21.15.63:0/45622161 wait complete. 2024-04-26T04:18:47.714 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.682+0000 7fbd6e941640 10 monclient: build_initial_monmap 2024-04-26T04:18:47.715 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.682+0000 7fbd6e941640 10 monclient: monmap: 2024-04-26T04:18:47.715 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:47.715 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:18:47.715 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:18:47.715 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:18:47.715 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:47.715 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:47.715 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:18:47.715 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:18:47.715 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:18:47.715 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:47.715 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.682+0000 7fbd6e941640 1 Processor -- start 2024-04-26T04:18:47.716 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.682+0000 7f84da408640 10 monclient: build_initial_monmap 2024-04-26T04:18:47.716 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.682+0000 7f84da408640 10 monclient: monmap: 2024-04-26T04:18:47.716 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:47.716 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:18:47.716 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:18:47.716 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:18:47.716 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:47.716 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:47.716 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:18:47.717 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:18:47.717 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:18:47.717 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:47.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.682+0000 7f84da408640 1 Processor -- start 2024-04-26T04:18:47.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.682+0000 7f84da408640 1 -- start start 2024-04-26T04:18:47.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.682+0000 7f84da408640 10 monclient: init 2024-04-26T04:18:47.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7f84da408640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:47.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7f84da408640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T04:18:47.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7f84da408640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f84d40a2cd0 0x7f84d413f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:47.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7f84da408640 10 monclient(hunting): picked mon.noname-c con 0x7f84d40a2cd0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:47.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7f84da408640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84d40a47d0 0x7f84d413f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:47.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7f84da408640 10 monclient(hunting): picked mon.noname-b con 0x7f84d40a47d0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:18:47.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7f84da408640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f84d40a4c00 0x7f84d4137db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:47.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7f84da408640 10 monclient(hunting): picked mon.noname-a con 0x7f84d40a4c00 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:18:47.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7f84da408640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:47.718 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7f84da408640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f84d4142be0 con 0x7f84d40a4c00 2024-04-26T04:18:47.718 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7f84da408640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:47.718 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7f84da408640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f84d4142a60 con 0x7f84d40a47d0 2024-04-26T04:18:47.718 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7f84da408640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:47.718 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7f84da408640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f84d4142d60 con 0x7f84d40a2cd0 2024-04-26T04:18:47.718 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7f84da408640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:47.718 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7f84da408640 10 monclient(hunting): authenticate will time out at 959.396472s 2024-04-26T04:18:47.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.682+0000 7fbd6e941640 1 -- start start 2024-04-26T04:18:47.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.682+0000 7fbd6e941640 10 monclient: init 2024-04-26T04:18:47.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7fbd6e941640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:47.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7fbd6e941640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T04:18:47.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7fbd6e941640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbd680a2cd0 0x7fbd6808fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:47.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7fbd6e941640 10 monclient(hunting): picked mon.noname-a con 0x7fbd680a2cd0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:18:47.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7fbd6e941640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbd680a47d0 0x7fbd6813ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:47.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7fbd6e941640 10 monclient(hunting): picked mon.noname-b con 0x7fbd680a47d0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:18:47.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7fbd6e941640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fbd680a4c00 0x7fbd6813bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:47.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7fbd6e941640 10 monclient(hunting): picked mon.noname-c con 0x7fbd680a4c00 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:47.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7fbd6e941640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:47.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7fbd6e941640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fbd68141a50 con 0x7fbd680a2cd0 2024-04-26T04:18:47.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7fbd6e941640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:47.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7fbd6e941640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbd68137b40 con 0x7fbd680a47d0 2024-04-26T04:18:47.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7fbd6e941640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:47.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7fbd6e941640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7fbd68141bd0 con 0x7fbd680a4c00 2024-04-26T04:18:47.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7fbd6e941640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:47.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7fbd6e941640 10 monclient(hunting): authenticate will time out at 959.396526s 2024-04-26T04:18:47.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7fbd67fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbd680a2cd0 0x7fbd6808fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:47.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7fbd67fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbd680a2cd0 0x7fbd6808fb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:46400/0 (socket says 172.21.15.63:46400) 2024-04-26T04:18:47.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7fbd67fff640 1 -- 172.21.15.63:0/4093175473 learned_addr learned my addr 172.21.15.63:0/4093175473 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:18:47.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7fbd67fff640 10 monclient(hunting): get_auth_request con 0x7fbd680a2cd0 auth_method 0 2024-04-26T04:18:47.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7fbd67fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:47.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7fbd67fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:47.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7fbd67fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:47.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7fbd67fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:47.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7fbd67fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:47.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7fbd677fe640 1 --2- 172.21.15.63:0/4093175473 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbd680a47d0 0x7fbd6813ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:47.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7fbd67fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:47.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7fbd677fe640 10 monclient(hunting): get_auth_request con 0x7fbd680a47d0 auth_method 0 2024-04-26T04:18:47.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7fbd677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:47.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7fbd677fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:47.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7fbd677fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:47.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7f84d3fff640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f84d40a2cd0 0x7f84d413f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:47.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7f84d89a5640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f84d40a4c00 0x7f84d4137db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:47.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7f84d3fff640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f84d40a2cd0 0x7f84d413f290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:57780/0 (socket says 172.21.15.63:57780) 2024-04-26T04:18:47.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7f84d3fff640 1 -- 172.21.15.63:0/1623322705 learned_addr learned my addr 172.21.15.63:0/1623322705 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:18:47.722 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7f84d3fff640 10 monclient(hunting): get_auth_request con 0x7f84d40a2cd0 auth_method 0 2024-04-26T04:18:47.722 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7f84d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:47.722 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7f84d3fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:47.722 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7f84d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:47.722 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7f84d89a5640 10 monclient(hunting): get_auth_request con 0x7f84d40a4c00 auth_method 0 2024-04-26T04:18:47.722 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7f84d89a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:47.722 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7f84d89a5640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:47.722 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7f84d89a5640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:47.722 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7f84d3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:47.722 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7f84d3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:47.722 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7f84d3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:47.722 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7f84d89a5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:47.722 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7f84d89a5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:47.722 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7f84d89a5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:47.723 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7f84d3fff640 10 monclient(hunting): handle_auth_done global_id 4202 payload 824 2024-04-26T04:18:47.723 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7f84d3fff640 1 -- 172.21.15.63:0/1623322705 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84d40a47d0 msgr2=0x7f84d413f7d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T04:18:47.723 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7f84d3fff640 1 --2- 172.21.15.63:0/1623322705 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84d40a47d0 0x7f84d413f7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:47.723 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7f84d3fff640 1 -- 172.21.15.63:0/1623322705 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f84d40a4c00 msgr2=0x7f84d4137db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:47.723 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7f84d3fff640 1 --2- 172.21.15.63:0/1623322705 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f84d40a4c00 0x7f84d4137db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:47.723 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7f84d3fff640 10 monclient: _finish_hunting 0 2024-04-26T04:18:47.723 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7f84d3fff640 1 monclient: found mon.noname-c 2024-04-26T04:18:47.723 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7f84d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:47.723 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7f84d3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.63:3301/0 2024-04-26T04:18:47.723 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7f84d3fff640 1 -- 172.21.15.63:0/1623322705 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84d413a490 con 0x7f84d40a2cd0 2024-04-26T04:18:47.724 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7fbd6cede640 1 --2- 172.21.15.63:0/4093175473 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fbd680a4c00 0x7fbd6813bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:47.724 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7fbd677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:47.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7fbd677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:47.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7fbd677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:47.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7fbd67fff640 10 monclient(hunting): handle_auth_done global_id 4215 payload 824 2024-04-26T04:18:47.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7fbd67fff640 1 -- 172.21.15.63:0/4093175473 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fbd680a4c00 msgr2=0x7fbd6813bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:47.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7fbd67fff640 1 --2- 172.21.15.63:0/4093175473 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fbd680a4c00 0x7fbd6813bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:47.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7fbd67fff640 1 -- 172.21.15.63:0/4093175473 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbd680a47d0 msgr2=0x7fbd6813ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:47.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7fbd67fff640 1 --2- 172.21.15.63:0/4093175473 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbd680a47d0 0x7fbd6813ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:47.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7fbd67fff640 10 monclient: _finish_hunting 0 2024-04-26T04:18:47.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7fbd67fff640 1 monclient: found mon.noname-a 2024-04-26T04:18:47.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7fbd67fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:47.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7fbd67fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:18:47.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7fbd67fff640 1 -- 172.21.15.63:0/4093175473 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd6813daf0 con 0x7fbd680a2cd0 2024-04-26T04:18:47.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7fbd6cede640 10 monclient: get_auth_request con 0x7fbd680a4c00 auth_method 0 2024-04-26T04:18:47.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7fbd6cede640 1 --2- 172.21.15.63:0/4093175473 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fbd680a4c00 0x7fbd6813bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T04:18:47.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7f84d3fff640 1 -- 172.21.15.63:0/1623322705 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f84d40a2cd0 msgr2=0x7f84d413f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:47.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7f84d3fff640 1 --2- 172.21.15.63:0/1623322705 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f84d40a2cd0 0x7f84d413f290 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f84c401ab70 tx=0x7f84c4019260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:47.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7f84d0ff9640 1 -- 172.21.15.63:0/1623322705 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84c40b1050 con 0x7f84d40a2cd0 2024-04-26T04:18:47.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7f84d0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:47.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7f84d0ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:18:47.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7f84d0ff9640 10 monclient: dump: 2024-04-26T04:18:47.726 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:47.726 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:47.727 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:47.727 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:47.727 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:47.727 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:47.727 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:47.727 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:47.727 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:47.727 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:47.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7f84d0ff9640 10 monclient: _finish_auth 0 2024-04-26T04:18:47.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7f84d0ff9640 10 monclient: _check_auth_tickets 2024-04-26T04:18:47.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7f84d0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:47.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7f84d0ff9640 1 -- 172.21.15.63:0/1623322705 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84c40bb070 con 0x7f84d40a2cd0 2024-04-26T04:18:47.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7f84d0ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:47.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7f84d0ff9640 1 -- 172.21.15.63:0/1623322705 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84c40a8040 con 0x7f84d40a2cd0 2024-04-26T04:18:47.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7f84d0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:47.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7f84d0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:18:47.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7f84d0ff9640 10 monclient: dump: 2024-04-26T04:18:47.728 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:47.728 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:47.728 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:47.728 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:47.728 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:47.728 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:47.728 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:47.728 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:47.729 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:47.729 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:47.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7f84d27fc640 4 set_mon_vals no callback set 2024-04-26T04:18:47.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.690+0000 7fbd67fff640 1 -- 172.21.15.63:0/4093175473 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbd680a2cd0 msgr2=0x7fbd6808fb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:47.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.690+0000 7fbd67fff640 1 --2- 172.21.15.63:0/4093175473 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbd680a2cd0 0x7fbd6808fb10 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fbd5801b040 tx=0x7fbd58019730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:47.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.690+0000 7fbd64ff9640 1 -- 172.21.15.63:0/4093175473 DispatchQueue::entry inject delay of 0.002000 2024-04-26T04:18:47.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7f84da408640 5 monclient: authenticate success, global_id 4202 2024-04-26T04:18:47.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7f84da408640 10 monclient: _renew_subs 2024-04-26T04:18:47.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.686+0000 7f84da408640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:47.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.690+0000 7f84da408640 1 -- 172.21.15.63:0/1623322705 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84d4071f30 con 0x7f84d40a2cd0 2024-04-26T04:18:47.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.690+0000 7f84da408640 10 monclient: _renew_subs 2024-04-26T04:18:47.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.690+0000 7f84da408640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:47.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.690+0000 7f84da408640 1 -- 172.21.15.63:0/1623322705 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f84d41395a0 con 0x7f84d40a2cd0 2024-04-26T04:18:47.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.690+0000 7f84d0ff9640 1 -- 172.21.15.63:0/1623322705 <== mon.2 v2:172.21.15.63:3301/0 4 ==== mgrmap(e 4) ==== 43997+0+0 (secure 0 0 0) 0x7f84c40aba90 con 0x7f84d40a2cd0 2024-04-26T04:18:47.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.690+0000 7f84da408640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:47.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.690+0000 7f84da408640 1 -- 172.21.15.63:0/1623322705 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f84d4072ad0 con 0x7f84d40a2cd0 2024-04-26T04:18:47.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.690+0000 7f84d0ff9640 1 --2- 172.21.15.63:0/1623322705 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f84c0035eb0 0x7f84c0038370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:47.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.690+0000 7f84d0ff9640 1 -- 172.21.15.63:0/1623322705 <== mon.2 v2:172.21.15.63:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7f84c40e2ca0 con 0x7f84d40a2cd0 2024-04-26T04:18:47.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.690+0000 7f84d0ff9640 1 --2- 172.21.15.63:0/1623322705 >> [v2:172.21.15.63:6808/413829082,v1:172.21.15.63:6809/413829082] conn(0x7f84c003bda0 0x7f84c003e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:47.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.690+0000 7f84d0ff9640 1 -- 172.21.15.63:0/1623322705 --> [v2:172.21.15.63:6808/413829082,v1:172.21.15.63:6809/413829082] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f84c40a5e90 con 0x7f84c003bda0 2024-04-26T04:18:47.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.690+0000 7f84d0ff9640 1 -- 172.21.15.63:0/1623322705 <== mon.2 v2:172.21.15.63:3301/0 6 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (secure 0 0 0) 0x7f84c40e81e0 con 0x7f84d40a2cd0 2024-04-26T04:18:47.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.690+0000 7f84d0ff9640 10 monclient: handle_get_version_reply finishing 1 version 15 2024-04-26T04:18:47.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.690+0000 7f84d89a5640 1 --2- 172.21.15.63:0/1623322705 >> [v2:172.21.15.63:6808/413829082,v1:172.21.15.63:6809/413829082] conn(0x7f84c003bda0 0x7f84c003e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:47.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.690+0000 7f84d89a5640 10 monclient: get_auth_request con 0x7f84c003bda0 auth_method 0 2024-04-26T04:18:47.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.690+0000 7f84d37fe640 1 --2- 172.21.15.63:0/1623322705 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f84c0035eb0 0x7f84c0038370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:47.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.690+0000 7f84d37fe640 10 monclient: get_auth_request con 0x7f84c0035eb0 auth_method 0 2024-04-26T04:18:47.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.690+0000 7fbd64ff9640 1 -- 172.21.15.63:0/4093175473 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd580bc020 con 0x7fbd680a2cd0 2024-04-26T04:18:47.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.690+0000 7fbd64ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:47.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.690+0000 7fbd64ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:18:47.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.690+0000 7fbd64ff9640 10 monclient: dump: 2024-04-26T04:18:47.734 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:47.734 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:47.734 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:47.734 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:47.734 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:47.734 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:47.734 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:47.734 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:47.734 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:47.735 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:47.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.690+0000 7fbd64ff9640 10 monclient: _finish_auth 0 2024-04-26T04:18:47.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.690+0000 7fbd64ff9640 10 monclient: _check_auth_tickets 2024-04-26T04:18:47.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.690+0000 7fbd64ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:47.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.690+0000 7fbd64ff9640 1 -- 172.21.15.63:0/4093175473 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd58007e20 con 0x7fbd680a2cd0 2024-04-26T04:18:47.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.690+0000 7fbd64ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:47.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.690+0000 7fbd64ff9640 1 -- 172.21.15.63:0/4093175473 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd580b86b0 con 0x7fbd680a2cd0 2024-04-26T04:18:47.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.690+0000 7fbd64ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:47.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.690+0000 7fbd64ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:18:47.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.690+0000 7fbd64ff9640 10 monclient: dump: 2024-04-26T04:18:47.736 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:47.736 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:47.736 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:47.736 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:47.736 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:47.736 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:47.736 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:47.736 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:47.736 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:47.736 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:47.736 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.690+0000 7fbd6e941640 5 monclient: authenticate success, global_id 4215 2024-04-26T04:18:47.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.690+0000 7fbd6e941640 10 monclient: _renew_subs 2024-04-26T04:18:47.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.690+0000 7fbd6e941640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:47.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.690+0000 7fbd667fc640 4 set_mon_vals no callback set 2024-04-26T04:18:47.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.690+0000 7fbd6e941640 1 -- 172.21.15.63:0/4093175473 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd680a37d0 con 0x7fbd680a2cd0 2024-04-26T04:18:47.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.690+0000 7fbd6e941640 10 monclient: _renew_subs 2024-04-26T04:18:47.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.690+0000 7fbd6e941640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:47.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.690+0000 7fbd6e941640 1 -- 172.21.15.63:0/4093175473 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd6813d6b0 con 0x7fbd680a2cd0 2024-04-26T04:18:47.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.690+0000 7fbd6e941640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:47.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.690+0000 7fbd6e941640 1 -- 172.21.15.63:0/4093175473 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fbd68062e10 con 0x7fbd680a2cd0 2024-04-26T04:18:47.738 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.690+0000 7fbd64ff9640 1 -- 172.21.15.63:0/4093175473 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 4) ==== 43997+0+0 (secure 0 0 0) 0x7fbd580bf020 con 0x7fbd680a2cd0 2024-04-26T04:18:47.738 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.694+0000 7fbd64ff9640 1 --2- 172.21.15.63:0/4093175473 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fbd48035e80 0x7fbd48038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:47.738 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.694+0000 7fbd64ff9640 1 -- 172.21.15.63:0/4093175473 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7fbd580e3d50 con 0x7fbd680a2cd0 2024-04-26T04:18:47.738 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.694+0000 7fbd64ff9640 1 --2- 172.21.15.63:0/4093175473 >> [v2:172.21.15.63:6816/3596874741,v1:172.21.15.63:6817/3596874741] conn(0x7fbd4803bd70 0x7fbd4803e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:47.738 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.694+0000 7fbd64ff9640 1 -- 172.21.15.63:0/4093175473 --> [v2:172.21.15.63:6816/3596874741,v1:172.21.15.63:6817/3596874741] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbd5800ae60 con 0x7fbd4803bd70 2024-04-26T04:18:47.738 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.694+0000 7fbd64ff9640 1 -- 172.21.15.63:0/4093175473 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (secure 0 0 0) 0x7fbd580e62d0 con 0x7fbd680a2cd0 2024-04-26T04:18:47.738 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.694+0000 7fbd64ff9640 10 monclient: handle_get_version_reply finishing 1 version 15 2024-04-26T04:18:47.738 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.694+0000 7fbd677fe640 1 --2- 172.21.15.63:0/4093175473 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fbd48035e80 0x7fbd48038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:47.738 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.694+0000 7fbd6cede640 1 --2- 172.21.15.63:0/4093175473 >> [v2:172.21.15.63:6816/3596874741,v1:172.21.15.63:6817/3596874741] conn(0x7fbd4803bd70 0x7fbd4803e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:47.738 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.694+0000 7fbd677fe640 10 monclient: get_auth_request con 0x7fbd48035e80 auth_method 0 2024-04-26T04:18:47.738 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.694+0000 7fbd6cede640 10 monclient: get_auth_request con 0x7fbd4803bd70 auth_method 0 2024-04-26T04:18:47.739 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.694+0000 7f84d37fe640 1 --2- 172.21.15.63:0/1623322705 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f84c0035eb0 0x7f84c0038370 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f84d413ffe0 tx=0x7f84c80079c0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:47.739 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.694+0000 7f84d89a5640 1 --2- 172.21.15.63:0/1623322705 >> [v2:172.21.15.63:6808/413829082,v1:172.21.15.63:6809/413829082] conn(0x7f84c003bda0 0x7f84c003e1c0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:47.739 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.694+0000 7fbd677fe640 1 --2- 172.21.15.63:0/4093175473 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fbd48035e80 0x7fbd48038340 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fbd680a24d0 tx=0x7fbd50007e40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:47.740 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.694+0000 7f84d0ff9640 1 -- 172.21.15.63:0/1623322705 <== osd.2 v2:172.21.15.63:6808/413829082 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f84c40a5e90 con 0x7f84c003bda0 2024-04-26T04:18:47.740 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.698+0000 7fbd6cede640 1 --2- 172.21.15.63:0/4093175473 >> [v2:172.21.15.63:6816/3596874741,v1:172.21.15.63:6817/3596874741] conn(0x7fbd4803bd70 0x7fbd4803e190 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:47.740 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.698+0000 7fbd64ff9640 1 -- 172.21.15.63:0/4093175473 <== osd.1 v2:172.21.15.63:6816/3596874741 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbd5800ae60 con 0x7fbd4803bd70 2024-04-26T04:18:47.740 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.714+0000 7fbd6e941640 1 -- 172.21.15.63:0/4093175473 --> [v2:172.21.15.63:6816/3596874741,v1:172.21.15.63:6817/3596874741] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fbd68090050 con 0x7fbd4803bd70 2024-04-26T04:18:47.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.714+0000 7fbd64ff9640 1 -- 172.21.15.63:0/4093175473 <== osd.1 v2:172.21.15.63:6816/3596874741 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fbd68090050 con 0x7fbd4803bd70 2024-04-26T04:18:47.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.714+0000 7fbd6e941640 1 -- 172.21.15.63:0/4093175473 >> [v2:172.21.15.63:6816/3596874741,v1:172.21.15.63:6817/3596874741] conn(0x7fbd4803bd70 msgr2=0x7fbd4803e190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:47.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.714+0000 7fbd6e941640 1 --2- 172.21.15.63:0/4093175473 >> [v2:172.21.15.63:6816/3596874741,v1:172.21.15.63:6817/3596874741] conn(0x7fbd4803bd70 0x7fbd4803e190 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:47.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.714+0000 7fbd6e941640 1 -- 172.21.15.63:0/4093175473 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fbd48035e80 msgr2=0x7fbd48038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:47.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.714+0000 7fbd6e941640 1 --2- 172.21.15.63:0/4093175473 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fbd48035e80 0x7fbd48038340 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fbd680a24d0 tx=0x7fbd50007e40 comp rx=0 tx=0).stop 2024-04-26T04:18:47.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.714+0000 7fbd6e941640 10 monclient: shutdown 2024-04-26T04:18:47.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.714+0000 7fbd6e941640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:47.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.714+0000 7fbd6e941640 1 -- 172.21.15.63:0/4093175473 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbd680a2cd0 msgr2=0x7fbd6808fb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:47.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.714+0000 7fbd6e941640 1 --2- 172.21.15.63:0/4093175473 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbd680a2cd0 0x7fbd6808fb10 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fbd5801b040 tx=0x7fbd58019730 comp rx=0 tx=0).stop 2024-04-26T04:18:47.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.714+0000 7fbd6cede640 1 -- 172.21.15.63:0/4093175473 reap_dead start 2024-04-26T04:18:47.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.714+0000 7fbd6e941640 1 -- 172.21.15.63:0/4093175473 shutdown_connections 2024-04-26T04:18:47.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.714+0000 7fbd6e941640 1 -- 172.21.15.63:0/4093175473 >> 172.21.15.63:0/4093175473 conn(0x7fbd680a5f40 msgr2=0x7fbd680a8280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:47.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.714+0000 7fbd6e941640 1 -- 172.21.15.63:0/4093175473 shutdown_connections 2024-04-26T04:18:47.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.714+0000 7fbd6e941640 1 -- 172.21.15.63:0/4093175473 wait complete. 2024-04-26T04:18:47.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.722+0000 7f75af84b640 10 monclient: get_monmap_and_config 2024-04-26T04:18:47.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.722+0000 7f75af84b640 10 monclient: build_initial_monmap 2024-04-26T04:18:47.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.722+0000 7f75af84b640 10 monclient: monmap: 2024-04-26T04:18:47.743 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:47.743 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:47.744 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:47.726916+0000 2024-04-26T04:18:47.744 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:47.726916+0000 2024-04-26T04:18:47.744 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:47.744 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:47.744 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:18:47.744 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:18:47.744 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:18:47.744 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:18:47.744 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:18:47.744 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:47.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.722+0000 7f75af84b640 1 Processor -- start 2024-04-26T04:18:47.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.722+0000 7f75af84b640 1 -- start start 2024-04-26T04:18:47.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.722+0000 7f75af84b640 10 monclient: init 2024-04-26T04:18:47.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.722+0000 7f75af84b640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:47.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.722+0000 7f75af84b640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-04-26T04:18:47.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.722+0000 7f75af84b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f75a80a6a30 addr v1:172.21.15.186:6789/0 2024-04-26T04:18:47.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.722+0000 7f75af84b640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f75a80a1f70 0x7f75a80a2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:47.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.722+0000 7f75af84b640 10 monclient(hunting): picked mon.noname-b con 0x7f75a80a1f70 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:47.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.722+0000 7f75af84b640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f75a8140610 0x7f75a8142a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:47.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.722+0000 7f75af84b640 10 monclient(hunting): picked mon.noname-c con 0x7f75a8140610 addr v2:172.21.15.186:3300/0 2024-04-26T04:18:47.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.722+0000 7f75af84b640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75a80a4f50 con 0x7f75a80a6a30 2024-04-26T04:18:47.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.722+0000 7f75af84b640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:47.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.722+0000 7f75af84b640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f75a80a5da0 con 0x7f75a8140610 2024-04-26T04:18:47.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.722+0000 7f75af84b640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:47.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.722+0000 7f75af84b640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f75a80a6e10 con 0x7f75a80a1f70 2024-04-26T04:18:47.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.722+0000 7f75af84b640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:47.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.722+0000 7f75af84b640 10 monclient(hunting): authenticate will time out at 959.434632s 2024-04-26T04:18:47.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.722+0000 7f75acde6640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f75a80a1f70 0x7f75a80a2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:47.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.722+0000 7f75acde6640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f75a80a1f70 0x7f75a80a2bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:57800/0 (socket says 172.21.15.63:57800) 2024-04-26T04:18:47.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.722+0000 7f75acde6640 1 -- 172.21.15.63:0/211246226 learned_addr learned my addr 172.21.15.63:0/211246226 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:18:47.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.722+0000 7f75acde6640 10 monclient(hunting): get_auth_request con 0x7f75a80a1f70 auth_method 0 2024-04-26T04:18:47.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.722+0000 7f75acde6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:47.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.722+0000 7f75acde6640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:47.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.722+0000 7f75acde6640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:47.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.722+0000 7f75acde6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:47.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.722+0000 7f75acde6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:47.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.722+0000 7f75adde8640 1 --2- 172.21.15.63:0/211246226 >> v2:172.21.15.186:3300/0 conn(0x7f75a8140610 0x7f75a8142a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:47.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.722+0000 7f75acde6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:47.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.722+0000 7f75adde8640 10 monclient(hunting): get_auth_request con 0x7f75a8140610 auth_method 0 2024-04-26T04:18:47.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.722+0000 7f75adde8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:47.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.722+0000 7f75adde8640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:47.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.722+0000 7f75adde8640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:47.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.722+0000 7f75acde6640 10 monclient(hunting): handle_auth_done global_id 4205 payload 274 2024-04-26T04:18:47.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.722+0000 7f75acde6640 1 -- 172.21.15.63:0/211246226 >> v2:172.21.15.186:3300/0 conn(0x7f75a8140610 msgr2=0x7f75a8142a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:47.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.722+0000 7f75acde6640 1 --2- 172.21.15.63:0/211246226 >> v2:172.21.15.186:3300/0 conn(0x7f75a8140610 0x7f75a8142a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:47.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.722+0000 7f75acde6640 1 -- 172.21.15.63:0/211246226 >> v1:172.21.15.186:6789/0 conn(0x7f75a80a6a30 legacy=0x7f75a80a24e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:47.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.722+0000 7f75acde6640 10 monclient: _finish_hunting 0 2024-04-26T04:18:47.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.722+0000 7f75acde6640 1 monclient: found mon.noname-b 2024-04-26T04:18:47.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.722+0000 7f75acde6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:47.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.722+0000 7f75acde6640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.63:3301/0 2024-04-26T04:18:47.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.722+0000 7f75acde6640 1 -- 172.21.15.63:0/211246226 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75a80a0cd0 con 0x7f75a80a1f70 2024-04-26T04:18:47.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.726+0000 7f84da408640 1 -- 172.21.15.63:0/1623322705 --> [v2:172.21.15.63:6808/413829082,v1:172.21.15.63:6809/413829082] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f84d4062e10 con 0x7f84c003bda0 2024-04-26T04:18:47.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.726+0000 7f84d0ff9640 1 -- 172.21.15.63:0/1623322705 <== osd.2 v2:172.21.15.63:6808/413829082 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f84d4062e10 con 0x7f84c003bda0 2024-04-26T04:18:47.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.726+0000 7f84da408640 1 -- 172.21.15.63:0/1623322705 >> [v2:172.21.15.63:6808/413829082,v1:172.21.15.63:6809/413829082] conn(0x7f84c003bda0 msgr2=0x7f84c003e1c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:47.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.726+0000 7f84da408640 1 --2- 172.21.15.63:0/1623322705 >> [v2:172.21.15.63:6808/413829082,v1:172.21.15.63:6809/413829082] conn(0x7f84c003bda0 0x7f84c003e1c0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:47.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.726+0000 7f84da408640 1 -- 172.21.15.63:0/1623322705 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f84c0035eb0 msgr2=0x7f84c0038370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:47.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.726+0000 7f84da408640 1 --2- 172.21.15.63:0/1623322705 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f84c0035eb0 0x7f84c0038370 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f84d413ffe0 tx=0x7f84c80079c0 comp rx=0 tx=0).stop 2024-04-26T04:18:47.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.726+0000 7f84da408640 10 monclient: shutdown 2024-04-26T04:18:47.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.726+0000 7f84da408640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:47.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.726+0000 7f84da408640 1 -- 172.21.15.63:0/1623322705 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f84d40a2cd0 msgr2=0x7f84d413f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:47.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.726+0000 7f84da408640 1 --2- 172.21.15.63:0/1623322705 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f84d40a2cd0 0x7f84d413f290 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f84c401ab70 tx=0x7f84c4019260 comp rx=0 tx=0).stop 2024-04-26T04:18:47.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.726+0000 7f84d89a5640 1 -- 172.21.15.63:0/1623322705 reap_dead start 2024-04-26T04:18:47.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.726+0000 7f84da408640 1 -- 172.21.15.63:0/1623322705 shutdown_connections 2024-04-26T04:18:47.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.726+0000 7f84da408640 1 -- 172.21.15.63:0/1623322705 >> 172.21.15.63:0/1623322705 conn(0x7f84d40a5f40 msgr2=0x7f84d408f860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:47.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.726+0000 7f84da408640 1 -- 172.21.15.63:0/1623322705 shutdown_connections 2024-04-26T04:18:47.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.726+0000 7f84da408640 1 -- 172.21.15.63:0/1623322705 wait complete. 2024-04-26T04:18:47.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.730+0000 7f75acde6640 1 -- 172.21.15.63:0/211246226 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f75a80a1f70 msgr2=0x7f75a80a2bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:47.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.730+0000 7f75acde6640 1 --2- 172.21.15.63:0/211246226 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f75a80a1f70 0x7f75a80a2bf0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f75a80a09d0 tx=0x7f759801c250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=968754e84393eaa8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:47.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.730+0000 7f759ffff640 1 -- 172.21.15.63:0/211246226 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75980b0070 con 0x7f75a80a1f70 2024-04-26T04:18:47.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.730+0000 7f759ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:47.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.730+0000 7f759ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T04:18:47.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.730+0000 7f759ffff640 10 monclient: dump: 2024-04-26T04:18:47.752 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:47.752 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:47.752 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:47.752 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:47.753 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:47.753 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:47.753 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:47.753 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:47.753 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:47.753 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:47.753 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.730+0000 7f759ffff640 10 monclient: _finish_auth 0 2024-04-26T04:18:47.753 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.730+0000 7f759ffff640 10 monclient: _check_auth_tickets 2024-04-26T04:18:47.753 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.730+0000 7f759ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:47.753 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.730+0000 7f759ffff640 1 -- 172.21.15.63:0/211246226 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75980ab600 con 0x7f75a80a1f70 2024-04-26T04:18:47.753 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.730+0000 7f75af84b640 5 monclient: authenticate success, global_id 4205 2024-04-26T04:18:47.753 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.730+0000 7f75af84b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T04:18:47.753 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.730+0000 7f759ffff640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:47.753 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.730+0000 7f75af84b640 10 monclient: get_monmap_and_config success 2024-04-26T04:18:47.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.730+0000 7f75af84b640 4 set_mon_vals no callback set 2024-04-26T04:18:47.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.730+0000 7f759ffff640 1 -- 172.21.15.63:0/211246226 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75980ab9d0 con 0x7f75a80a1f70 2024-04-26T04:18:47.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.730+0000 7f759ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:47.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.730+0000 7f759ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:18:47.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.730+0000 7f759ffff640 10 monclient: dump: 2024-04-26T04:18:47.754 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:47.754 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:47.754 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:47.754 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:47.754 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:47.754 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:47.754 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:47.754 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:47.754 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:47.755 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:47.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.730+0000 7f75af84b640 10 monclient: shutdown 2024-04-26T04:18:47.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.730+0000 7f75af84b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:47.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.730+0000 7f75af84b640 1 -- 172.21.15.63:0/211246226 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f75a80a1f70 msgr2=0x7f75a80a2bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:47.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.730+0000 7f75af84b640 1 --2- 172.21.15.63:0/211246226 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f75a80a1f70 0x7f75a80a2bf0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f75a80a09d0 tx=0x7f759801c250 comp rx=0 tx=0).stop 2024-04-26T04:18:47.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.730+0000 7f75af84b640 1 -- 172.21.15.63:0/211246226 shutdown_connections 2024-04-26T04:18:47.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.730+0000 7f75af84b640 1 --2- 172.21.15.63:0/211246226 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f75a80a1f70 0x7f75a80a2bf0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:47.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.730+0000 7f75af84b640 1 --2- 172.21.15.63:0/211246226 >> v2:172.21.15.186:3300/0 conn(0x7f75a8140610 0x7f75a8142a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:47.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.730+0000 7f75af84b640 1 -- 172.21.15.63:0/211246226 >> 172.21.15.63:0/211246226 conn(0x7f75a80a5f40 msgr2=0x7f75a80a6320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:47.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.730+0000 7f75af84b640 1 -- 172.21.15.63:0/211246226 shutdown_connections 2024-04-26T04:18:47.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.730+0000 7f75af84b640 1 -- 172.21.15.63:0/211246226 wait complete. 2024-04-26T04:18:47.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.730+0000 7f75af84b640 10 monclient: build_initial_monmap 2024-04-26T04:18:47.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.730+0000 7f75af84b640 10 monclient: monmap: 2024-04-26T04:18:47.755 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:47.755 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:18:47.756 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:18:47.756 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:18:47.756 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:47.756 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:47.756 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:18:47.756 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:18:47.756 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:18:47.756 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:47.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.730+0000 7f75af84b640 1 Processor -- start 2024-04-26T04:18:47.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.730+0000 7f75af84b640 1 -- start start 2024-04-26T04:18:47.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.730+0000 7f75af84b640 10 monclient: init 2024-04-26T04:18:47.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.734+0000 7f75af84b640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:47.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.734+0000 7f75af84b640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T04:18:47.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.734+0000 7f75af84b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f75a80a6a30 0x7f75a81d3200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:47.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.734+0000 7f75af84b640 10 monclient(hunting): picked mon.noname-a con 0x7f75a80a6a30 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:18:47.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.734+0000 7f75af84b640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f75a8140610 0x7f75a81d3740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:47.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.734+0000 7f75af84b640 10 monclient(hunting): picked mon.noname-c con 0x7f75a8140610 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:47.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.734+0000 7f75af84b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75a81d3c80 0x7f75a813c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:47.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.734+0000 7f75af84b640 10 monclient(hunting): picked mon.noname-b con 0x7f75a81d3c80 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:18:47.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.734+0000 7f75af84b640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:47.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.734+0000 7f75af84b640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f75a8145af0 con 0x7f75a80a6a30 2024-04-26T04:18:47.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.734+0000 7f75af84b640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:47.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.734+0000 7f75af84b640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f75a8145970 con 0x7f75a81d3c80 2024-04-26T04:18:47.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.734+0000 7f75af84b640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:47.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.734+0000 7f75af84b640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f75a8145c70 con 0x7f75a8140610 2024-04-26T04:18:47.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.734+0000 7f75af84b640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:47.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.734+0000 7f75af84b640 10 monclient(hunting): authenticate will time out at 959.444671s 2024-04-26T04:18:47.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.734+0000 7f75acde6640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f75a8140610 0x7f75a81d3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:47.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.734+0000 7f75acde6640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f75a8140610 0x7f75a81d3740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:57806/0 (socket says 172.21.15.63:57806) 2024-04-26T04:18:47.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.734+0000 7f75acde6640 1 -- 172.21.15.63:0/842670099 learned_addr learned my addr 172.21.15.63:0/842670099 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:18:47.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.734+0000 7f75acde6640 10 monclient(hunting): get_auth_request con 0x7f75a8140610 auth_method 0 2024-04-26T04:18:47.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.734+0000 7f75acde6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:47.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.734+0000 7f75acde6640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:47.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.734+0000 7f75acde6640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:47.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.734+0000 7f75acde6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:47.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.734+0000 7f75acde6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:47.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.734+0000 7f75acde6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:47.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.734+0000 7f75ad5e7640 1 --2- 172.21.15.63:0/842670099 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f75a80a6a30 0x7f75a81d3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:47.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.734+0000 7f75ad5e7640 10 monclient(hunting): get_auth_request con 0x7f75a80a6a30 auth_method 0 2024-04-26T04:18:47.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.734+0000 7f75ad5e7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:47.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.734+0000 7f75ad5e7640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:47.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.734+0000 7f75ad5e7640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:47.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.734+0000 7f75adde8640 1 --2- 172.21.15.63:0/842670099 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75a81d3c80 0x7f75a813c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:47.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.734+0000 7f75adde8640 10 monclient(hunting): get_auth_request con 0x7f75a81d3c80 auth_method 0 2024-04-26T04:18:47.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.734+0000 7f75adde8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:47.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.734+0000 7f75adde8640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:47.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.734+0000 7f75adde8640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:47.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.734+0000 7f75ad5e7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:47.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.734+0000 7f75ad5e7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:47.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.734+0000 7f75ad5e7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:47.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.734+0000 7f75adde8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:47.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.734+0000 7f75adde8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:47.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.734+0000 7f75adde8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:47.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.734+0000 7f75acde6640 10 monclient(hunting): handle_auth_done global_id 4208 payload 824 2024-04-26T04:18:47.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.734+0000 7f75acde6640 1 -- 172.21.15.63:0/842670099 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75a81d3c80 msgr2=0x7f75a813c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:47.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.734+0000 7f75acde6640 1 --2- 172.21.15.63:0/842670099 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75a81d3c80 0x7f75a813c3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:47.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.734+0000 7f75acde6640 1 -- 172.21.15.63:0/842670099 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f75a80a6a30 msgr2=0x7f75a81d3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:47.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.734+0000 7f75acde6640 1 --2- 172.21.15.63:0/842670099 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f75a80a6a30 0x7f75a81d3200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:47.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.734+0000 7f75acde6640 10 monclient: _finish_hunting 0 2024-04-26T04:18:47.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.734+0000 7f75acde6640 1 monclient: found mon.noname-c 2024-04-26T04:18:47.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.734+0000 7f75acde6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:47.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.734+0000 7f75acde6640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.63:3301/0 2024-04-26T04:18:47.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.734+0000 7f75acde6640 1 -- 172.21.15.63:0/842670099 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75a8070b00 con 0x7f75a8140610 2024-04-26T04:18:47.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.734+0000 7f75ad5e7640 1 --2- 172.21.15.63:0/842670099 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f75a80a6a30 0x7f75a81d3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T04:18:47.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.734+0000 7f695ed41640 10 monclient: get_monmap_and_config 2024-04-26T04:18:47.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.734+0000 7f695ed41640 10 monclient: build_initial_monmap 2024-04-26T04:18:47.761 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.734+0000 7f695ed41640 10 monclient: monmap: 2024-04-26T04:18:47.761 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:47.761 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:47.761 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:47.741459+0000 2024-04-26T04:18:47.761 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:47.741459+0000 2024-04-26T04:18:47.761 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:47.761 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:47.761 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:18:47.761 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:18:47.761 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:18:47.762 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:18:47.762 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:18:47.762 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:47.762 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.734+0000 7f75acde6640 1 -- 172.21.15.63:0/842670099 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f75a8140610 msgr2=0x7f75a81d3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:47.762 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.734+0000 7f75acde6640 1 --2- 172.21.15.63:0/842670099 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f75a8140610 0x7f75a81d3740 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f759801a900 tx=0x7f7598018ff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:47.762 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7f759dffb640 1 -- 172.21.15.63:0/842670099 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75980b0070 con 0x7f75a8140610 2024-04-26T04:18:47.762 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7f759dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:47.763 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7f759dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:18:47.763 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7f759dffb640 10 monclient: dump: 2024-04-26T04:18:47.763 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:47.763 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:47.763 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:47.763 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:47.763 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:47.763 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:47.763 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:47.763 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:47.764 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:47.764 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:47.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7f759dffb640 10 monclient: _finish_auth 0 2024-04-26T04:18:47.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7f759dffb640 10 monclient: _check_auth_tickets 2024-04-26T04:18:47.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7f759dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:47.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7f759dffb640 1 -- 172.21.15.63:0/842670099 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75980b1040 con 0x7f75a8140610 2024-04-26T04:18:47.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7f75af84b640 5 monclient: authenticate success, global_id 4208 2024-04-26T04:18:47.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7f75af84b640 10 monclient: _renew_subs 2024-04-26T04:18:47.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7f75af84b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:47.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7f75af84b640 1 -- 172.21.15.63:0/842670099 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75a8071cd0 con 0x7f75a8140610 2024-04-26T04:18:47.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7f75af84b640 10 monclient: _renew_subs 2024-04-26T04:18:47.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7f75af84b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:47.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7f75af84b640 1 -- 172.21.15.63:0/842670099 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f75a80a3410 con 0x7f75a8140610 2024-04-26T04:18:47.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7f759dffb640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:47.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7f695ed41640 1 Processor -- start 2024-04-26T04:18:47.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7f759dffb640 1 -- 172.21.15.63:0/842670099 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75980aa9e0 con 0x7f75a8140610 2024-04-26T04:18:47.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7f759dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:47.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7f759dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:18:47.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7f759dffb640 10 monclient: dump: 2024-04-26T04:18:47.766 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:47.766 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:47.766 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:47.766 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:47.766 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:47.766 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:47.766 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:47.766 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:47.766 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:47.766 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:47.766 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7f759f7fe640 4 set_mon_vals no callback set 2024-04-26T04:18:47.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7f695ed41640 1 -- start start 2024-04-26T04:18:47.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7f695ed41640 10 monclient: init 2024-04-26T04:18:47.768 INFO:teuthology.orchestra.run.smithi063.stdout:51539607556 2024-04-26T04:18:47.769 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-26T04:18:47.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7f695ed41640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:47.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7f695ed41640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-04-26T04:18:47.770 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7fd514a53640 10 monclient: get_monmap_and_config 2024-04-26T04:18:47.770 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7fd514a53640 10 monclient: build_initial_monmap 2024-04-26T04:18:47.770 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7fd514a53640 10 monclient: monmap: 2024-04-26T04:18:47.770 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:47.770 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:47.770 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:47.743283+0000 2024-04-26T04:18:47.770 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:47.743283+0000 2024-04-26T04:18:47.771 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:47.771 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:47.771 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:18:47.771 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:18:47.771 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:18:47.771 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:18:47.771 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:18:47.771 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:47.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7f695ed41640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f69580a2cd0 addr v1:172.21.15.186:6789/0 2024-04-26T04:18:47.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7f695ed41640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f69580a30b0 0x7f695813c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:47.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7f695ed41640 10 monclient(hunting): picked mon.noname-a con 0x7f69580a30b0 addr v2:172.21.15.63:3300/0 2024-04-26T04:18:47.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7f695ed41640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f695813c540 addr v1:172.21.15.63:6789/0 2024-04-26T04:18:47.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7f695ed41640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69580a4ee0 con 0x7f695813c540 2024-04-26T04:18:47.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7f695ed41640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69580a0fb0 con 0x7f69580a2cd0 2024-04-26T04:18:47.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7f695ed41640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:47.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7f695ed41640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7f69580a5380 con 0x7f69580a30b0 2024-04-26T04:18:47.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7f695ed41640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:47.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7f695ed41640 10 monclient(hunting): authenticate will time out at 959.449548s 2024-04-26T04:18:47.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7f6957fff640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f69580a30b0 0x7f695813c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:47.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7f6957fff640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f69580a30b0 0x7f695813c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:46420/0 (socket says 172.21.15.63:46420) 2024-04-26T04:18:47.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7f6957fff640 1 -- 172.21.15.63:0/4034190141 learned_addr learned my addr 172.21.15.63:0/4034190141 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:18:47.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7f6957fff640 10 monclient(hunting): get_auth_request con 0x7f69580a30b0 auth_method 0 2024-04-26T04:18:47.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7f6957fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:47.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7f6957fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:47.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7f6957fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:47.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7f6957fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:47.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7f6957fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:47.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7f6957fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:47.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7f6957fff640 10 monclient(hunting): handle_auth_done global_id 4221 payload 274 2024-04-26T04:18:47.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7f6957fff640 1 -- 172.21.15.63:0/4034190141 >> v1:172.21.15.186:6789/0 conn(0x7f69580a2cd0 legacy=0x7f69580a47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:47.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7f6957fff640 1 -- 172.21.15.63:0/4034190141 >> v1:172.21.15.63:6789/0 conn(0x7f695813c540 legacy=0x7f695813e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:47.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7f6957fff640 10 monclient: _finish_hunting 0 2024-04-26T04:18:47.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7f6957fff640 1 monclient: found mon.noname-a 2024-04-26T04:18:47.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7f6957fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:47.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7f6957fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:18:47.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7f6957fff640 1 -- 172.21.15.63:0/4034190141 --> v2:172.21.15.63:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69580a1290 con 0x7f69580a30b0 2024-04-26T04:18:47.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7f759dffb640 1 -- 172.21.15.63:0/842670099 <== mon.2 v2:172.21.15.63:3301/0 4 ==== mgrmap(e 4) ==== 43997+0+0 (secure 0 0 0) 0x7f75980aab80 con 0x7f75a8140610 2024-04-26T04:18:47.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7f75af84b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:47.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7f75af84b640 1 -- 172.21.15.63:0/842670099 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7588000f90 con 0x7f75a8140610 2024-04-26T04:18:47.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7fd514a53640 1 Processor -- start 2024-04-26T04:18:47.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7fd514a53640 1 -- start start 2024-04-26T04:18:47.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7fd514a53640 10 monclient: init 2024-04-26T04:18:47.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7f759dffb640 1 --2- 172.21.15.63:0/842670099 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f758c035e80 0x7f758c038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:47.777 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7f759dffb640 1 -- 172.21.15.63:0/842670099 <== mon.2 v2:172.21.15.63:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7f75980e2890 con 0x7f75a8140610 2024-04-26T04:18:47.777 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7f759dffb640 1 --2- 172.21.15.63:0/842670099 >> [v2:172.21.15.186:6800/3746743567,v1:172.21.15.186:6801/3746743567] conn(0x7f758c03bd70 0x7f758c03e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:47.777 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7f759dffb640 1 -- 172.21.15.63:0/842670099 --> [v2:172.21.15.186:6800/3746743567,v1:172.21.15.186:6801/3746743567] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f75980ab600 con 0x7f758c03bd70 2024-04-26T04:18:47.777 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7f759dffb640 1 -- 172.21.15.63:0/842670099 <== mon.2 v2:172.21.15.63:3301/0 6 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (secure 0 0 0) 0x7f75980e7200 con 0x7f75a8140610 2024-04-26T04:18:47.777 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7f759dffb640 10 monclient: handle_get_version_reply finishing 1 version 15 2024-04-26T04:18:47.777 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7f75ad5e7640 1 --2- 172.21.15.63:0/842670099 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f758c035e80 0x7f758c038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:47.777 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7f75adde8640 1 --2- 172.21.15.63:0/842670099 >> [v2:172.21.15.186:6800/3746743567,v1:172.21.15.186:6801/3746743567] conn(0x7f758c03bd70 0x7f758c03e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:47.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7fd514a53640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:47.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7fd514a53640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-04-26T04:18:47.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7fd514a53640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7fd51006e490 0x7fd51006fe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:47.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7fd514a53640 10 monclient(hunting): picked mon.noname-a con 0x7fd51006e490 addr v2:172.21.15.63:3300/0 2024-04-26T04:18:47.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7f75ad5e7640 10 monclient: get_auth_request con 0x7f758c035e80 auth_method 0 2024-04-26T04:18:47.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7fd514a53640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fd510137260 0x7fd510137690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:47.779 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7f75adde8640 10 monclient: get_auth_request con 0x7f758c03bd70 auth_method 0 2024-04-26T04:18:47.779 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7fd514a53640 10 monclient(hunting): picked mon.noname-b con 0x7fd510137260 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:47.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7fd50e59c640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7fd51006e490 0x7fd51006fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:47.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7fd514a53640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd5100a1d60 0x7fd5100a2170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:47.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7fd514a53640 10 monclient(hunting): picked mon.noname-c con 0x7fd5100a1d60 addr v2:172.21.15.186:3300/0 2024-04-26T04:18:47.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7fd514a53640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:47.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7fd50e59c640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7fd51006e490 0x7fd51006fe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:46430/0 (socket says 172.21.15.63:46430) 2024-04-26T04:18:47.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7fd50e59c640 1 -- 172.21.15.63:0/2412874717 learned_addr learned my addr 172.21.15.63:0/2412874717 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:18:47.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7fd514a53640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7fd51006f490 con 0x7fd51006e490 2024-04-26T04:18:47.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7fd514a53640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:47.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7fd514a53640 1 -- 172.21.15.63:0/2412874717 --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fd5100a3840 con 0x7fd5100a1d60 2024-04-26T04:18:47.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7fd514a53640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:47.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7fd514a53640 1 -- 172.21.15.63:0/2412874717 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7fd51006f1b0 con 0x7fd510137260 2024-04-26T04:18:47.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7fd514a53640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:47.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7fd514a53640 10 monclient(hunting): authenticate will time out at 959.451446s 2024-04-26T04:18:47.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7fd50e59c640 10 monclient(hunting): get_auth_request con 0x7fd51006e490 auth_method 0 2024-04-26T04:18:47.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7fd50e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:47.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7fd50e59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:47.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7fd50e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:47.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7fd50dd9b640 1 --2- 172.21.15.63:0/2412874717 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fd510137260 0x7fd510137690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:47.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7fd50dd9b640 10 monclient(hunting): get_auth_request con 0x7fd510137260 auth_method 0 2024-04-26T04:18:47.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7fd50dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:47.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7fd50dd9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:47.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7fd50dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:47.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7fd50e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:47.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7fd50e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:47.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7fd50e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:47.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7fd50dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:47.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7fd50dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:47.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7fd50dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:47.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7fd50e59c640 10 monclient(hunting): handle_auth_done global_id 4224 payload 274 2024-04-26T04:18:47.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7fd50e59c640 1 -- 172.21.15.63:0/2412874717 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fd510137260 msgr2=0x7fd510137690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:47.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7fd50e59c640 1 --2- 172.21.15.63:0/2412874717 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fd510137260 0x7fd510137690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:47.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7fd50e59c640 1 -- 172.21.15.63:0/2412874717 >> v2:172.21.15.186:3300/0 conn(0x7fd5100a1d60 msgr2=0x7fd5100a2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:47.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7fd50e59c640 1 --2- 172.21.15.63:0/2412874717 >> v2:172.21.15.186:3300/0 conn(0x7fd5100a1d60 0x7fd5100a2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:47.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7fd50e59c640 10 monclient: _finish_hunting 0 2024-04-26T04:18:47.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7fd50e59c640 1 monclient: found mon.noname-a 2024-04-26T04:18:47.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7fd50e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:47.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7fd50e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:18:47.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.738+0000 7fd50e59c640 1 -- 172.21.15.63:0/2412874717 --> v2:172.21.15.63:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd51006f770 con 0x7fd51006e490 2024-04-26T04:18:47.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.742+0000 7f6f5b1c6640 10 monclient: get_monmap_and_config 2024-04-26T04:18:47.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.742+0000 7f6f5b1c6640 10 monclient: build_initial_monmap 2024-04-26T04:18:47.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.742+0000 7f6f5b1c6640 10 monclient: monmap: 2024-04-26T04:18:47.783 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:47.783 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:47.783 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:47.747516+0000 2024-04-26T04:18:47.783 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:47.747516+0000 2024-04-26T04:18:47.783 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:47.783 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:47.783 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:18:47.783 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:18:47.783 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:18:47.783 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:18:47.783 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:18:47.783 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:47.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.742+0000 7f75ad5e7640 1 --2- 172.21.15.63:0/842670099 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f758c035e80 0x7f758c038340 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f75a4007fd0 tx=0x7f75a4007e40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:47.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.742+0000 7f6f5b1c6640 1 Processor -- start 2024-04-26T04:18:47.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.742+0000 7f6f5b1c6640 1 -- start start 2024-04-26T04:18:47.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.742+0000 7f6f5b1c6640 10 monclient: init 2024-04-26T04:18:47.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.742+0000 7f6f5b1c6640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:47.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.742+0000 7f6f5b1c6640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-04-26T04:18:47.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.742+0000 7f6f5b1c6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6f54134b90 addr v1:172.21.15.63:6789/0 2024-04-26T04:18:47.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.742+0000 7f6f5b1c6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6f54135680 addr v1:172.21.15.186:6789/0 2024-04-26T04:18:47.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.742+0000 7f6f5b1c6640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f6f5409d5d0 0x7f6f5413d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:47.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.742+0000 7f6f5b1c6640 10 monclient(hunting): picked mon.noname-b con 0x7f6f5409d5d0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:47.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.742+0000 7f6f5b1c6640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f540a3630 con 0x7f6f54134b90 2024-04-26T04:18:47.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.742+0000 7f6f5b1c6640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f5409d9d0 con 0x7f6f54135680 2024-04-26T04:18:47.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.742+0000 7f6f5b1c6640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:47.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.742+0000 7f6f5b1c6640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f6f540a3840 con 0x7f6f5409d5d0 2024-04-26T04:18:47.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.742+0000 7f6f5b1c6640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:47.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.742+0000 7f6f5b1c6640 10 monclient(hunting): authenticate will time out at 959.454853s 2024-04-26T04:18:47.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.742+0000 7f6957fff640 1 -- 172.21.15.63:0/4034190141 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f69580a30b0 msgr2=0x7f695813c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:47.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.742+0000 7f6957fff640 1 --2- 172.21.15.63:0/4034190141 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f69580a30b0 0x7f695813c000 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f694c009870 tx=0x7f694c01c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c55a22292b8c35c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:47.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.742+0000 7f6f59763640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f6f5409d5d0 0x7f6f5413d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:47.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.742+0000 7f6f59763640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f6f5409d5d0 0x7f6f5413d7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:57818/0 (socket says 172.21.15.63:57818) 2024-04-26T04:18:47.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.742+0000 7f6f59763640 1 -- 172.21.15.63:0/2954213581 learned_addr learned my addr 172.21.15.63:0/2954213581 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:18:47.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.742+0000 7f6f59763640 10 monclient(hunting): get_auth_request con 0x7f6f5409d5d0 auth_method 0 2024-04-26T04:18:47.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.742+0000 7f6f59763640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:47.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.742+0000 7f6f59763640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:47.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.742+0000 7f6f59763640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:47.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.742+0000 7f6f59763640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:47.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.742+0000 7f6f59763640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:47.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.742+0000 7f69577fe640 1 -- 172.21.15.63:0/4034190141 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f694c0b0070 con 0x7f69580a30b0 2024-04-26T04:18:47.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.742+0000 7f69577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:47.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.742+0000 7f69577fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:18:47.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.742+0000 7f69577fe640 10 monclient: dump: 2024-04-26T04:18:47.788 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:47.788 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:47.788 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:47.788 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:47.788 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:47.788 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:47.788 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:47.788 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:47.788 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:47.788 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:47.788 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.742+0000 7f69577fe640 10 monclient: _finish_auth 0 2024-04-26T04:18:47.788 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.742+0000 7f69577fe640 10 monclient: _check_auth_tickets 2024-04-26T04:18:47.788 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.742+0000 7f69577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:47.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.742+0000 7f6f59763640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:47.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.742+0000 7f69577fe640 1 -- 172.21.15.63:0/4034190141 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f694c0ab600 con 0x7f69580a30b0 2024-04-26T04:18:47.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.742+0000 7f69577fe640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:47.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.742+0000 7f69577fe640 1 -- 172.21.15.63:0/4034190141 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f694c0ab9d0 con 0x7f69580a30b0 2024-04-26T04:18:47.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.742+0000 7f69577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:47.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.742+0000 7f69577fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:18:47.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.742+0000 7f69577fe640 10 monclient: dump: 2024-04-26T04:18:47.789 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:47.789 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:47.789 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:47.789 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:47.790 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:47.790 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:47.790 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:47.790 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:47.790 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:47.790 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:47.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.742+0000 7f695ed41640 5 monclient: authenticate success, global_id 4221 2024-04-26T04:18:47.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.742+0000 7f695ed41640 10 monclient: get_monmap_and_config success 2024-04-26T04:18:47.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.742+0000 7f695ed41640 4 set_mon_vals no callback set 2024-04-26T04:18:47.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.742+0000 7f6f59763640 10 monclient(hunting): handle_auth_done global_id 4214 payload 274 2024-04-26T04:18:47.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.742+0000 7f6f59763640 1 -- 172.21.15.63:0/2954213581 >> v1:172.21.15.186:6789/0 conn(0x7f6f54135680 legacy=0x7f6f5413a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:47.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.742+0000 7f6f59763640 1 -- 172.21.15.63:0/2954213581 >> v1:172.21.15.63:6789/0 conn(0x7f6f54134b90 legacy=0x7f6f54134f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:47.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.742+0000 7f6f59763640 10 monclient: _finish_hunting 0 2024-04-26T04:18:47.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.742+0000 7f6f59763640 1 monclient: found mon.noname-b 2024-04-26T04:18:47.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.742+0000 7f6f59763640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:47.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.742+0000 7f6f59763640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.63:3301/0 2024-04-26T04:18:47.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.742+0000 7f6f59763640 1 -- 172.21.15.63:0/2954213581 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f5409c350 con 0x7f6f5409d5d0 2024-04-26T04:18:47.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7fd50e59c640 1 -- 172.21.15.63:0/2412874717 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd51006e490 msgr2=0x7fd51006fe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:47.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7fd50e59c640 1 --2- 172.21.15.63:0/2412874717 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd51006e490 0x7fd51006fe70 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd5100a3c30 tx=0x7fd50001c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da97c652da5f3dc server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:47.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7f75adde8640 1 --2- 172.21.15.63:0/842670099 >> [v2:172.21.15.186:6800/3746743567,v1:172.21.15.186:6801/3746743567] conn(0x7f758c03bd70 0x7f758c03e190 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:47.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7fd50d59a640 1 -- 172.21.15.63:0/2412874717 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5000b1070 con 0x7fd51006e490 2024-04-26T04:18:47.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7fd50d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:47.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7fd50d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:18:47.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7fd50d59a640 10 monclient: dump: 2024-04-26T04:18:47.793 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:47.793 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:47.793 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:47.793 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:47.793 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:47.793 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:47.793 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:47.793 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:47.793 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:47.793 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:47.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7fd50d59a640 10 monclient: _finish_auth 0 2024-04-26T04:18:47.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7fd50d59a640 10 monclient: _check_auth_tickets 2024-04-26T04:18:47.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7fd50d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:47.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7fd514a53640 5 monclient: authenticate success, global_id 4224 2024-04-26T04:18:47.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7fd514a53640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T04:18:47.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7fd50d59a640 1 -- 172.21.15.63:0/2412874717 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5000ac620 con 0x7fd51006e490 2024-04-26T04:18:47.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7fd50d59a640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:47.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7fd514a53640 10 monclient: get_monmap_and_config success 2024-04-26T04:18:47.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7fd514a53640 4 set_mon_vals no callback set 2024-04-26T04:18:47.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7fd514a53640 10 monclient: shutdown 2024-04-26T04:18:47.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7fd514a53640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:47.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7fd514a53640 1 -- 172.21.15.63:0/2412874717 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd51006e490 msgr2=0x7fd51006fe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:47.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7fd514a53640 1 --2- 172.21.15.63:0/2412874717 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd51006e490 0x7fd51006fe70 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd5100a3c30 tx=0x7fd50001c3e0 comp rx=0 tx=0).stop 2024-04-26T04:18:47.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7f695ed41640 10 monclient: shutdown 2024-04-26T04:18:47.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7f695ed41640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:47.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7f695ed41640 1 -- 172.21.15.63:0/4034190141 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f69580a30b0 msgr2=0x7f695813c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:47.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7f695ed41640 1 --2- 172.21.15.63:0/4034190141 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f69580a30b0 0x7f695813c000 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f694c009870 tx=0x7f694c01c430 comp rx=0 tx=0).stop 2024-04-26T04:18:47.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7f695ed41640 1 -- 172.21.15.63:0/4034190141 shutdown_connections 2024-04-26T04:18:47.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7f695ed41640 1 --2- 172.21.15.63:0/4034190141 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f69580a30b0 0x7f695813c000 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:47.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7f695ed41640 1 -- 172.21.15.63:0/4034190141 >> 172.21.15.63:0/4034190141 conn(0x7f69580a5f40 msgr2=0x7f69580a6320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:47.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7f695ed41640 1 -- 172.21.15.63:0/4034190141 shutdown_connections 2024-04-26T04:18:47.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7f695ed41640 1 -- 172.21.15.63:0/4034190141 wait complete. 2024-04-26T04:18:47.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7f695ed41640 10 monclient: build_initial_monmap 2024-04-26T04:18:47.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7f695ed41640 10 monclient: monmap: 2024-04-26T04:18:47.796 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:47.796 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:18:47.796 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:18:47.796 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:18:47.796 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:47.796 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:47.796 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:18:47.796 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:18:47.796 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:18:47.796 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:47.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7fd514a53640 1 -- 172.21.15.63:0/2412874717 shutdown_connections 2024-04-26T04:18:47.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7fd514a53640 1 --2- 172.21.15.63:0/2412874717 >> v2:172.21.15.186:3300/0 conn(0x7fd5100a1d60 0x7fd5100a2170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:47.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7fd514a53640 1 --2- 172.21.15.63:0/2412874717 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fd510137260 0x7fd510137690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:47.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7fd514a53640 1 --2- 172.21.15.63:0/2412874717 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd51006e490 0x7fd51006fe70 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:47.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7fd514a53640 1 -- 172.21.15.63:0/2412874717 >> 172.21.15.63:0/2412874717 conn(0x7fd5100a7360 msgr2=0x7fd5100a7770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:47.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7f695ed41640 1 Processor -- start 2024-04-26T04:18:47.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7fd514a53640 1 -- 172.21.15.63:0/2412874717 shutdown_connections 2024-04-26T04:18:47.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7fd514a53640 1 -- 172.21.15.63:0/2412874717 wait complete. 2024-04-26T04:18:47.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7f695ed41640 1 -- start start 2024-04-26T04:18:47.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7f695ed41640 10 monclient: init 2024-04-26T04:18:47.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7fd514a53640 10 monclient: build_initial_monmap 2024-04-26T04:18:47.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7fd514a53640 10 monclient: monmap: 2024-04-26T04:18:47.798 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:47.798 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:18:47.799 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:18:47.799 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:18:47.799 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:47.799 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:47.799 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:18:47.799 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:18:47.799 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:18:47.799 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:47.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7f695ed41640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:47.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7f695ed41640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T04:18:47.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7f695ed41640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f69580a2cd0 0x7f6958139220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:47.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7f695ed41640 10 monclient(hunting): picked mon.noname-c con 0x7f69580a2cd0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:47.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7f695ed41640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f69580a30b0 0x7f6958139760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:47.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7f695ed41640 10 monclient(hunting): picked mon.noname-b con 0x7f69580a30b0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:18:47.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7f695ed41640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f695813c540 0x7f6958139ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:47.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7f695ed41640 10 monclient(hunting): picked mon.noname-a con 0x7f695813c540 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:18:47.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7f695ed41640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:47.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7f695ed41640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f6958142c00 con 0x7f695813c540 2024-04-26T04:18:47.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7f695ed41640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:47.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7f695ed41640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6958142a80 con 0x7f69580a30b0 2024-04-26T04:18:47.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7f695ed41640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:47.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7f695ed41640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f6958142d80 con 0x7f69580a2cd0 2024-04-26T04:18:47.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7f695ed41640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:47.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7f695ed41640 10 monclient(hunting): authenticate will time out at 959.458808s 2024-04-26T04:18:47.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7f695cadd640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f69580a2cd0 0x7f6958139220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:47.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7f695cadd640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f69580a2cd0 0x7f6958139220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:57824/0 (socket says 172.21.15.63:57824) 2024-04-26T04:18:47.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7f695cadd640 1 -- 172.21.15.63:0/2617927876 learned_addr learned my addr 172.21.15.63:0/2617927876 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:18:47.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7f695cadd640 10 monclient(hunting): get_auth_request con 0x7f69580a2cd0 auth_method 0 2024-04-26T04:18:47.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7f695cadd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:47.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7f695cadd640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:47.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7f695cadd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:47.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7f695d2de640 1 --2- 172.21.15.63:0/2617927876 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f695813c540 0x7f6958139ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:47.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7f759dffb640 1 -- 172.21.15.63:0/842670099 <== osd.4 v2:172.21.15.186:6800/3746743567 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f75980ab600 con 0x7f758c03bd70 2024-04-26T04:18:47.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7f695d2de640 10 monclient(hunting): get_auth_request con 0x7f695813c540 auth_method 0 2024-04-26T04:18:47.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7f695d2de640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:47.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7f695d2de640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:47.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7f695d2de640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:47.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7f695cadd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:47.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7f695cadd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:47.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7f695cadd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:47.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7fd514a53640 1 Processor -- start 2024-04-26T04:18:47.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7fd514a53640 1 -- start start 2024-04-26T04:18:47.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7fd514a53640 10 monclient: init 2024-04-26T04:18:47.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7f695d2de640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:47.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7f695d2de640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:47.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7f695d2de640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:47.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7f6957fff640 1 --2- 172.21.15.63:0/2617927876 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f69580a30b0 0x7f6958139760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:47.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7f6957fff640 10 monclient(hunting): get_auth_request con 0x7f69580a30b0 auth_method 0 2024-04-26T04:18:47.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7f6957fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:47.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7f6957fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:47.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7f6957fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:47.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7f695cadd640 10 monclient(hunting): handle_auth_done global_id 4217 payload 824 2024-04-26T04:18:47.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7f695cadd640 1 -- 172.21.15.63:0/2617927876 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f69580a30b0 msgr2=0x7f6958139760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:47.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7f695cadd640 1 --2- 172.21.15.63:0/2617927876 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f69580a30b0 0x7f6958139760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:47.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7f695cadd640 1 -- 172.21.15.63:0/2617927876 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f695813c540 msgr2=0x7f6958139ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:47.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7f695cadd640 1 --2- 172.21.15.63:0/2617927876 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f695813c540 0x7f6958139ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:47.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7f695cadd640 10 monclient: _finish_hunting 0 2024-04-26T04:18:47.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7f695cadd640 1 monclient: found mon.noname-c 2024-04-26T04:18:47.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7f695cadd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:47.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7f695cadd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.63:3301/0 2024-04-26T04:18:47.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7f695cadd640 1 -- 172.21.15.63:0/2617927876 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f695808c8c0 con 0x7f69580a2cd0 2024-04-26T04:18:47.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7f6957fff640 1 --2- 172.21.15.63:0/2617927876 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f69580a30b0 0x7f6958139760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T04:18:47.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7fd514a53640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:47.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7fd514a53640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T04:18:47.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7fd514a53640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd51006e490 0x7fd5101cee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:47.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7fd514a53640 10 monclient(hunting): picked mon.noname-b con 0x7fd51006e490 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:18:47.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7fd514a53640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fd5100a1d60 0x7fd510135f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:47.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7fd514a53640 10 monclient(hunting): picked mon.noname-c con 0x7fd5100a1d60 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:47.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7fd514a53640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd510137260 0x7fd5101364b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:47.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7fd514a53640 10 monclient(hunting): picked mon.noname-a con 0x7fd510137260 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:18:47.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7fd514a53640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:47.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7fd514a53640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fd510140b00 con 0x7fd510137260 2024-04-26T04:18:47.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7fd514a53640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:47.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7fd514a53640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd510140980 con 0x7fd51006e490 2024-04-26T04:18:47.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7fd514a53640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:47.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7fd514a53640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7fd510140c80 con 0x7fd5100a1d60 2024-04-26T04:18:47.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7fd514a53640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:47.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7fd514a53640 10 monclient(hunting): authenticate will time out at 959.459718s 2024-04-26T04:18:47.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7f6f59763640 1 -- 172.21.15.63:0/2954213581 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f6f5409d5d0 msgr2=0x7f6f5413d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:47.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7f6f59763640 1 --2- 172.21.15.63:0/2954213581 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f6f5409d5d0 0x7f6f5413d7f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c009e80 tx=0x7f6f4c01c890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=372e509826a2117c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:47.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7fd50ed9d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd510137260 0x7fd5101364b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:47.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7fd50ed9d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd510137260 0x7fd5101364b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:46444/0 (socket says 172.21.15.63:46444) 2024-04-26T04:18:47.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7fd50ed9d640 1 -- 172.21.15.63:0/3529123244 learned_addr learned my addr 172.21.15.63:0/3529123244 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:18:47.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7fd50ed9d640 10 monclient(hunting): get_auth_request con 0x7fd510137260 auth_method 0 2024-04-26T04:18:47.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7fd50ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:47.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7fd50ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:47.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.746+0000 7fd50ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:47.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7fd50ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:47.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7fd50ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:47.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7fd50ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:47.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f4b7fe640 1 -- 172.21.15.63:0/2954213581 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f4c0a5620 con 0x7f6f5409d5d0 2024-04-26T04:18:47.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f4b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:47.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f4b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T04:18:47.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f4b7fe640 10 monclient: dump: 2024-04-26T04:18:47.812 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:47.812 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:47.812 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:47.812 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:47.812 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:47.812 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:47.812 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:47.812 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:47.812 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:47.812 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:47.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f4b7fe640 10 monclient: _finish_auth 0 2024-04-26T04:18:47.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f4b7fe640 10 monclient: _check_auth_tickets 2024-04-26T04:18:47.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f4b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:47.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f4b7fe640 1 -- 172.21.15.63:0/2954213581 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f4c0a57c0 con 0x7f6f5409d5d0 2024-04-26T04:18:47.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f5b1c6640 5 monclient: authenticate success, global_id 4214 2024-04-26T04:18:47.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f5b1c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T04:18:47.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f4b7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:47.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f5b1c6640 10 monclient: get_monmap_and_config success 2024-04-26T04:18:47.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f4b7fe640 1 -- 172.21.15.63:0/2954213581 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f4c0a5b90 con 0x7f6f5409d5d0 2024-04-26T04:18:47.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f5b1c6640 4 set_mon_vals no callback set 2024-04-26T04:18:47.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f4b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:47.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f4b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:18:47.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f4b7fe640 10 monclient: dump: 2024-04-26T04:18:47.813 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:47.813 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:47.813 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:47.813 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:47.814 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:47.814 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:47.814 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:47.814 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:47.814 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:47.814 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:47.814 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7fd50ed9d640 10 monclient(hunting): handle_auth_done global_id 4230 payload 824 2024-04-26T04:18:47.814 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7fd50ed9d640 1 -- 172.21.15.63:0/3529123244 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fd5100a1d60 msgr2=0x7fd510135f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:47.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7fd50ed9d640 1 --2- 172.21.15.63:0/3529123244 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fd5100a1d60 0x7fd510135f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:47.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7fd50ed9d640 1 -- 172.21.15.63:0/3529123244 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd51006e490 msgr2=0x7fd5101cee30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-26T04:18:47.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7fd50ed9d640 1 --2- 172.21.15.63:0/3529123244 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd51006e490 0x7fd5101cee30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:47.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7fd50ed9d640 10 monclient: _finish_hunting 0 2024-04-26T04:18:47.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7fd50ed9d640 1 monclient: found mon.noname-a 2024-04-26T04:18:47.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7fd50ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:47.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7fd50ed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:18:47.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7fd50ed9d640 1 -- 172.21.15.63:0/3529123244 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd51013dc80 con 0x7fd510137260 2024-04-26T04:18:47.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f5b1c6640 10 monclient: shutdown 2024-04-26T04:18:47.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f5b1c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:47.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f5b1c6640 1 -- 172.21.15.63:0/2954213581 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f6f5409d5d0 msgr2=0x7f6f5413d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:47.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f5b1c6640 1 --2- 172.21.15.63:0/2954213581 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f6f5409d5d0 0x7f6f5413d7f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c009e80 tx=0x7f6f4c01c890 comp rx=0 tx=0).stop 2024-04-26T04:18:47.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f5b1c6640 1 -- 172.21.15.63:0/2954213581 shutdown_connections 2024-04-26T04:18:47.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f5b1c6640 1 --2- 172.21.15.63:0/2954213581 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f6f5409d5d0 0x7f6f5413d7f0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:47.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f5b1c6640 1 -- 172.21.15.63:0/2954213581 >> 172.21.15.63:0/2954213581 conn(0x7f6f540a7360 msgr2=0x7f6f540a7770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:47.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f5b1c6640 1 -- 172.21.15.63:0/2954213581 shutdown_connections 2024-04-26T04:18:47.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f5b1c6640 1 -- 172.21.15.63:0/2954213581 wait complete. 2024-04-26T04:18:47.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f695cadd640 1 -- 172.21.15.63:0/2617927876 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f69580a2cd0 msgr2=0x7f6958139220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:47.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f695cadd640 1 --2- 172.21.15.63:0/2617927876 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f69580a2cd0 0x7f6958139220 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f694800c450 tx=0x7f694800c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:47.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f5b1c6640 10 monclient: build_initial_monmap 2024-04-26T04:18:47.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f5b1c6640 10 monclient: monmap: 2024-04-26T04:18:47.817 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:47.817 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:18:47.817 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:18:47.817 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:18:47.817 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:47.817 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:47.818 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:18:47.818 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:18:47.818 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:18:47.818 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:47.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f69557fa640 1 -- 172.21.15.63:0/2617927876 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69480119a0 con 0x7f69580a2cd0 2024-04-26T04:18:47.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f69557fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:47.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f69557fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:18:47.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f69557fa640 10 monclient: dump: 2024-04-26T04:18:47.819 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:47.819 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:47.819 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:47.819 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:47.819 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:47.819 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:47.819 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:47.819 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:47.819 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:47.819 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:47.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f69557fa640 10 monclient: _finish_auth 0 2024-04-26T04:18:47.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f69557fa640 10 monclient: _check_auth_tickets 2024-04-26T04:18:47.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f69557fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:47.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f69557fa640 1 -- 172.21.15.63:0/2617927876 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6948017070 con 0x7f69580a2cd0 2024-04-26T04:18:47.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f69557fa640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:47.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f69557fa640 1 -- 172.21.15.63:0/2617927876 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6948011da0 con 0x7f69580a2cd0 2024-04-26T04:18:47.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f69557fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:47.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f69557fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:18:47.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f69557fa640 10 monclient: dump: 2024-04-26T04:18:47.820 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:47.820 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:47.820 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:47.820 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:47.820 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:47.820 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:47.820 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:47.820 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:47.821 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:47.821 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:47.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6956ffd640 4 set_mon_vals no callback set 2024-04-26T04:18:47.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f5b1c6640 1 Processor -- start 2024-04-26T04:18:47.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f5b1c6640 1 -- start start 2024-04-26T04:18:47.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f5b1c6640 10 monclient: init 2024-04-26T04:18:47.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f5b1c6640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:47.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f5b1c6640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T04:18:47.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f5b1c6640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f6f5409d5d0 0x7f6f54072800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:47.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f5b1c6640 10 monclient(hunting): picked mon.noname-c con 0x7f6f5409d5d0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:47.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f5b1c6640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f54134b90 0x7f6f54133fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:47.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f5b1c6640 10 monclient(hunting): picked mon.noname-b con 0x7f6f54134b90 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:18:47.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f5b1c6640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6f54135680 0x7f6f541344f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:47.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f5b1c6640 10 monclient(hunting): picked mon.noname-a con 0x7f6f54135680 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:18:47.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f5b1c6640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:47.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f5b1c6640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f6f5406e600 con 0x7f6f54135680 2024-04-26T04:18:47.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f5b1c6640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:47.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f5b1c6640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6f5406e480 con 0x7f6f54134b90 2024-04-26T04:18:47.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f5b1c6640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:47.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f5b1c6640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f6f5406e780 con 0x7f6f5409d5d0 2024-04-26T04:18:47.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f5b1c6640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:47.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f5b1c6640 10 monclient(hunting): authenticate will time out at 959.462839s 2024-04-26T04:18:47.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7fd50ed9d640 1 -- 172.21.15.63:0/3529123244 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd510137260 msgr2=0x7fd5101364b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:47.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7fd50ed9d640 1 --2- 172.21.15.63:0/3529123244 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd510137260 0x7fd5101364b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd50800b9a0 tx=0x7fd50800be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:47.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f59763640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6f54135680 0x7f6f541344f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:47.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f59763640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6f54135680 0x7f6f541344f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:46454/0 (socket says 172.21.15.63:46454) 2024-04-26T04:18:47.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f59763640 1 -- 172.21.15.63:0/891658063 learned_addr learned my addr 172.21.15.63:0/891658063 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:18:47.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f59763640 10 monclient(hunting): get_auth_request con 0x7f6f54135680 auth_method 0 2024-04-26T04:18:47.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f59763640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:47.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f59763640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:47.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f59763640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:47.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f58f62640 1 --2- 172.21.15.63:0/891658063 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f6f5409d5d0 0x7f6f54072800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:47.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f58f62640 10 monclient(hunting): get_auth_request con 0x7f6f5409d5d0 auth_method 0 2024-04-26T04:18:47.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f58f62640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:47.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f58f62640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:47.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f58f62640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:47.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f59763640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:47.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f59763640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:47.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f59763640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:47.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f4bfff640 1 --2- 172.21.15.63:0/891658063 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f54134b90 0x7f6f54133fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:47.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7fd50ed9d640 1 --2- 172.21.15.63:0/3529123244 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd510137260 0x7fd5101364b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd50800b9a0 tx=0x7fd50800be60 comp rx=0 tx=0).queue_received will delay after 1.71411e+09 on 0x7fd508014440 config(0 keys) 2024-04-26T04:18:47.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f4bfff640 10 monclient(hunting): get_auth_request con 0x7f6f54134b90 auth_method 0 2024-04-26T04:18:47.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f4bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:47.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f4bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:47.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:47.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f59763640 10 monclient(hunting): handle_auth_done global_id 4233 payload 824 2024-04-26T04:18:47.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f59763640 1 -- 172.21.15.63:0/891658063 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f6f5409d5d0 msgr2=0x7f6f54072800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:47.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f59763640 1 --2- 172.21.15.63:0/891658063 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f6f5409d5d0 0x7f6f54072800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:47.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f59763640 1 -- 172.21.15.63:0/891658063 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f54134b90 msgr2=0x7f6f54133fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:47.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f59763640 1 --2- 172.21.15.63:0/891658063 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f54134b90 0x7f6f54133fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:47.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f59763640 10 monclient: _finish_hunting 0 2024-04-26T04:18:47.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f59763640 1 monclient: found mon.noname-a 2024-04-26T04:18:47.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f59763640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:47.826 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f59763640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:18:47.827 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7f6f59763640 1 -- 172.21.15.63:0/891658063 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f54143f40 con 0x7f6f54135680 2024-04-26T04:18:47.828 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7fd4feffd640 1 -- 172.21.15.63:0/3529123244 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd508019070 con 0x7fd510137260 2024-04-26T04:18:47.828 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7fd4feffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:47.828 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7fd4feffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:18:47.828 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7fd4feffd640 10 monclient: dump: 2024-04-26T04:18:47.828 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:47.828 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:47.828 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:47.828 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:47.828 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:47.828 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:47.828 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:47.828 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:47.829 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:47.829 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:47.829 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7fd4feffd640 10 monclient: _finish_auth 0 2024-04-26T04:18:47.829 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7fd4feffd640 10 monclient: _check_auth_tickets 2024-04-26T04:18:47.829 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7fd4feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:47.829 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7fd4feffd640 1 -- 172.21.15.63:0/3529123244 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd508014440 con 0x7fd510137260 2024-04-26T04:18:47.829 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7fd4feffd640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:47.829 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7fd50cd99640 4 set_mon_vals no callback set 2024-04-26T04:18:47.829 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7fd514a53640 5 monclient: authenticate success, global_id 4230 2024-04-26T04:18:47.829 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7fd514a53640 10 monclient: _renew_subs 2024-04-26T04:18:47.829 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7fd514a53640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:47.829 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7fd514a53640 1 -- 172.21.15.63:0/3529123244 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd51013c270 con 0x7fd510137260 2024-04-26T04:18:47.829 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7fd514a53640 10 monclient: _renew_subs 2024-04-26T04:18:47.829 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7fd514a53640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:47.829 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.750+0000 7fd514a53640 1 -- 172.21.15.63:0/3529123244 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd51013b990 con 0x7fd510137260 2024-04-26T04:18:47.830 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.754+0000 7fd514a53640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:47.830 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.754+0000 7fd514a53640 1 -- 172.21.15.63:0/3529123244 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd510062e10 con 0x7fd510137260 2024-04-26T04:18:47.830 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.754+0000 7fd4feffd640 1 -- 172.21.15.63:0/3529123244 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd508014830 con 0x7fd510137260 2024-04-26T04:18:47.830 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.754+0000 7fd4feffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:47.830 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.754+0000 7fd4feffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:18:47.830 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.754+0000 7fd4feffd640 10 monclient: dump: 2024-04-26T04:18:47.830 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:47.830 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:47.830 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:47.830 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:47.830 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:47.830 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:47.830 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:47.830 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:47.830 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:47.831 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:47.831 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.754+0000 7fd4feffd640 1 -- 172.21.15.63:0/3529123244 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 4) ==== 43997+0+0 (secure 0 0 0) 0x7fd50801c070 con 0x7fd510137260 2024-04-26T04:18:47.831 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.754+0000 7fd4feffd640 1 --2- 172.21.15.63:0/3529123244 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fd4f0035e30 0x7fd4f00382f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:47.831 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.754+0000 7fd4feffd640 1 -- 172.21.15.63:0/3529123244 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7fd508048b10 con 0x7fd510137260 2024-04-26T04:18:47.831 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.754+0000 7fd4feffd640 1 --2- 172.21.15.63:0/3529123244 >> [v2:172.21.15.186:6807/4227496981,v1:172.21.15.186:6808/4227496981] conn(0x7fd4f003bca0 0x7fd4f003e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:47.831 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.754+0000 7fd4feffd640 1 -- 172.21.15.63:0/3529123244 --> [v2:172.21.15.186:6807/4227496981,v1:172.21.15.186:6808/4227496981] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd508015e50 con 0x7fd4f003bca0 2024-04-26T04:18:47.831 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.754+0000 7fd50dd9b640 1 --2- 172.21.15.63:0/3529123244 >> [v2:172.21.15.186:6807/4227496981,v1:172.21.15.186:6808/4227496981] conn(0x7fd4f003bca0 0x7fd4f003e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:47.831 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.754+0000 7fd50dd9b640 10 monclient: get_auth_request con 0x7fd4f003bca0 auth_method 0 2024-04-26T04:18:47.833 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.754+0000 7f6f59763640 1 -- 172.21.15.63:0/891658063 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6f54135680 msgr2=0x7f6f541344f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:47.833 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.754+0000 7f6f59763640 1 --2- 172.21.15.63:0/891658063 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6f54135680 0x7f6f541344f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c01af40 tx=0x7f6f4c019630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:47.833 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.754+0000 7f6f497fa640 1 -- 172.21.15.63:0/891658063 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f4c0bc020 con 0x7f6f54135680 2024-04-26T04:18:47.833 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.754+0000 7f6f497fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:47.833 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.754+0000 7f6f497fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:18:47.833 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.754+0000 7f6f497fa640 10 monclient: dump: 2024-04-26T04:18:47.833 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:47.833 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:47.833 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:47.833 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:47.834 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:47.834 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:47.834 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:47.834 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:47.834 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:47.834 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:47.834 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.754+0000 7f6f497fa640 10 monclient: _finish_auth 0 2024-04-26T04:18:47.834 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.754+0000 7f6f497fa640 10 monclient: _check_auth_tickets 2024-04-26T04:18:47.834 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.754+0000 7f6f497fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:47.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.754+0000 7f6f497fa640 1 -- 172.21.15.63:0/891658063 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f4c0b8880 con 0x7f6f54135680 2024-04-26T04:18:47.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.754+0000 7f6f497fa640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:47.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.754+0000 7f6f497fa640 1 -- 172.21.15.63:0/891658063 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f4c0a5620 con 0x7f6f54135680 2024-04-26T04:18:47.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.754+0000 7f6f497fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:47.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.754+0000 7f6f497fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:18:47.835 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.754+0000 7f6f497fa640 10 monclient: dump: 2024-04-26T04:18:47.835 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:47.835 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:47.835 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:47.835 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:47.835 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:47.835 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:47.835 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:47.835 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:47.836 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:47.836 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:47.836 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.754+0000 7f6f5b1c6640 5 monclient: authenticate success, global_id 4233 2024-04-26T04:18:47.836 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.754+0000 7f6f5b1c6640 10 monclient: _renew_subs 2024-04-26T04:18:47.836 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.754+0000 7f6f5b1c6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:47.836 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.754+0000 7f6f5b1c6640 1 -- 172.21.15.63:0/891658063 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f541436b0 con 0x7f6f54135680 2024-04-26T04:18:47.836 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.754+0000 7f6f5b1c6640 10 monclient: _renew_subs 2024-04-26T04:18:47.836 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.754+0000 7f6f5b1c6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:47.836 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.754+0000 7f6f5b1c6640 1 -- 172.21.15.63:0/891658063 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f54142cd0 con 0x7f6f54135680 2024-04-26T04:18:47.836 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.754+0000 7f6f4affd640 4 set_mon_vals no callback set 2024-04-26T04:18:47.837 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.754+0000 7f6f5b1c6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:47.837 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.754+0000 7f6f5b1c6640 1 -- 172.21.15.63:0/891658063 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6f541431f0 con 0x7f6f54135680 2024-04-26T04:18:47.837 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.754+0000 7f6f497fa640 1 -- 172.21.15.63:0/891658063 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 4) ==== 43997+0+0 (secure 0 0 0) 0x7f6f4c0bf020 con 0x7f6f54135680 2024-04-26T04:18:47.837 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.754+0000 7f6f497fa640 1 --2- 172.21.15.63:0/891658063 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f6f40035e30 0x7f6f400382f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:47.837 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.754+0000 7f6f497fa640 1 -- 172.21.15.63:0/891658063 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7f6f4c0e3cc0 con 0x7f6f54135680 2024-04-26T04:18:47.837 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.754+0000 7f6f497fa640 1 --2- 172.21.15.63:0/891658063 >> [v2:172.21.15.186:6816/3109931567,v1:172.21.15.186:6817/3109931567] conn(0x7f6f4003bc30 0x7f6f4003e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:47.837 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.754+0000 7f6f497fa640 1 -- 172.21.15.63:0/891658063 --> [v2:172.21.15.186:6816/3109931567,v1:172.21.15.186:6817/3109931567] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6f4c0c0e70 con 0x7f6f4003bc30 2024-04-26T04:18:47.837 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.754+0000 7f6f497fa640 1 -- 172.21.15.63:0/891658063 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (secure 0 0 0) 0x7f6f4c0e82a0 con 0x7f6f54135680 2024-04-26T04:18:47.837 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.754+0000 7f6f497fa640 10 monclient: handle_get_version_reply finishing 1 version 15 2024-04-26T04:18:47.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.758+0000 7f695ed41640 5 monclient: authenticate success, global_id 4217 2024-04-26T04:18:47.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.758+0000 7f695ed41640 10 monclient: _renew_subs 2024-04-26T04:18:47.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.758+0000 7f695ed41640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:47.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.758+0000 7f695ed41640 1 -- 172.21.15.63:0/2617927876 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f695808cba0 con 0x7f69580a2cd0 2024-04-26T04:18:47.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.758+0000 7f695ed41640 10 monclient: _renew_subs 2024-04-26T04:18:47.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.758+0000 7f695ed41640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:47.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.758+0000 7f695ed41640 1 -- 172.21.15.63:0/2617927876 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f695813ba00 con 0x7f69580a2cd0 2024-04-26T04:18:47.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.758+0000 7f6f58f62640 1 --2- 172.21.15.63:0/891658063 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f6f40035e30 0x7f6f400382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:47.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.758+0000 7f6f58f62640 10 monclient: get_auth_request con 0x7f6f40035e30 auth_method 0 2024-04-26T04:18:47.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.758+0000 7f6f4bfff640 1 --2- 172.21.15.63:0/891658063 >> [v2:172.21.15.186:6816/3109931567,v1:172.21.15.186:6817/3109931567] conn(0x7f6f4003bc30 0x7f6f4003e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:47.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.758+0000 7f6f4bfff640 10 monclient: get_auth_request con 0x7f6f4003bc30 auth_method 0 2024-04-26T04:18:47.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.758+0000 7fd50e59c640 1 --2- 172.21.15.63:0/3529123244 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fd4f0035e30 0x7fd4f00382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:47.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.758+0000 7fd50e59c640 10 monclient: get_auth_request con 0x7fd4f0035e30 auth_method 0 2024-04-26T04:18:47.841 INFO:teuthology.orchestra.run.smithi063.stdout:51539607556 2024-04-26T04:18:47.841 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-26T04:18:47.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.758+0000 7f695ed41640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:47.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.758+0000 7f695ed41640 1 -- 172.21.15.63:0/2617927876 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f69580a37d0 con 0x7f69580a2cd0 2024-04-26T04:18:47.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.758+0000 7f69557fa640 1 -- 172.21.15.63:0/2617927876 <== mon.2 v2:172.21.15.63:3301/0 4 ==== mgrmap(e 4) ==== 43997+0+0 (secure 0 0 0) 0x7f694801c460 con 0x7f69580a2cd0 2024-04-26T04:18:47.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.758+0000 7f69557fa640 1 --2- 172.21.15.63:0/2617927876 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f6944035e80 0x7f6944038340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:47.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.758+0000 7f69557fa640 1 -- 172.21.15.63:0/2617927876 <== mon.2 v2:172.21.15.63:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7f69480489f0 con 0x7f69580a2cd0 2024-04-26T04:18:47.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.758+0000 7f69557fa640 1 --2- 172.21.15.63:0/2617927876 >> [v2:172.21.15.63:6800/2758152700,v1:172.21.15.63:6801/2758152700] conn(0x7f694403bd70 0x7f694403e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:47.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.758+0000 7f69557fa640 1 -- 172.21.15.63:0/2617927876 --> [v2:172.21.15.63:6800/2758152700,v1:172.21.15.63:6801/2758152700] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f694403e840 con 0x7f694403bd70 2024-04-26T04:18:47.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.758+0000 7f69557fa640 1 -- 172.21.15.63:0/2617927876 <== mon.2 v2:172.21.15.63:3301/0 6 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (secure 0 0 0) 0x7f694804d2a0 con 0x7f69580a2cd0 2024-04-26T04:18:47.844 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.758+0000 7f69557fa640 10 monclient: handle_get_version_reply finishing 1 version 15 2024-04-26T04:18:47.844 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.758+0000 7f695d2de640 1 --2- 172.21.15.63:0/2617927876 >> [v2:172.21.15.63:6800/2758152700,v1:172.21.15.63:6801/2758152700] conn(0x7f694403bd70 0x7f694403e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:47.844 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.758+0000 7f695d2de640 10 monclient: get_auth_request con 0x7f694403bd70 auth_method 0 2024-04-26T04:18:47.844 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.758+0000 7fd50dd9b640 1 --2- 172.21.15.63:0/3529123244 >> [v2:172.21.15.186:6807/4227496981,v1:172.21.15.186:6808/4227496981] conn(0x7fd4f003bca0 0x7fd4f003e0c0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:47.844 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.758+0000 7fd50e59c640 1 --2- 172.21.15.63:0/3529123244 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fd4f0035e30 0x7fd4f00382f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd5000ac000 tx=0x7fd5000a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:47.844 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.762+0000 7fd4feffd640 1 -- 172.21.15.63:0/3529123244 <== osd.3 v2:172.21.15.186:6807/4227496981 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd508015e50 con 0x7fd4f003bca0 2024-04-26T04:18:47.845 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.762+0000 7f6f4bfff640 1 --2- 172.21.15.63:0/891658063 >> [v2:172.21.15.186:6816/3109931567,v1:172.21.15.186:6817/3109931567] conn(0x7f6f4003bc30 0x7f6f4003e050 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:47.845 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.762+0000 7f695d2de640 1 --2- 172.21.15.63:0/2617927876 >> [v2:172.21.15.63:6800/2758152700,v1:172.21.15.63:6801/2758152700] conn(0x7f694403bd70 0x7f694403e190 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:47.845 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.762+0000 7f6f497fa640 1 -- 172.21.15.63:0/891658063 <== osd.5 v2:172.21.15.186:6816/3109931567 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6f4c0c0e70 con 0x7f6f4003bc30 2024-04-26T04:18:47.846 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.762+0000 7f69557fa640 1 -- 172.21.15.63:0/2617927876 <== osd.0 v2:172.21.15.63:6800/2758152700 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f694403e840 con 0x7f694403bd70 2024-04-26T04:18:47.846 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.766+0000 7f75af84b640 1 -- 172.21.15.63:0/842670099 --> [v2:172.21.15.186:6800/3746743567,v1:172.21.15.186:6801/3746743567] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7588002c80 con 0x7f758c03bd70 2024-04-26T04:18:47.846 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.766+0000 7f759dffb640 1 -- 172.21.15.63:0/842670099 <== osd.4 v2:172.21.15.186:6800/3746743567 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f7588002c80 con 0x7f758c03bd70 2024-04-26T04:18:47.847 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.770+0000 7f75af84b640 1 -- 172.21.15.63:0/842670099 >> [v2:172.21.15.186:6800/3746743567,v1:172.21.15.186:6801/3746743567] conn(0x7f758c03bd70 msgr2=0x7f758c03e190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:47.847 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.770+0000 7f75af84b640 1 --2- 172.21.15.63:0/842670099 >> [v2:172.21.15.186:6800/3746743567,v1:172.21.15.186:6801/3746743567] conn(0x7f758c03bd70 0x7f758c03e190 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:47.847 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.770+0000 7f75af84b640 1 -- 172.21.15.63:0/842670099 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f758c035e80 msgr2=0x7f758c038340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:47.847 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.770+0000 7f75af84b640 1 --2- 172.21.15.63:0/842670099 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f758c035e80 0x7f758c038340 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f75a4007fd0 tx=0x7f75a4007e40 comp rx=0 tx=0).stop 2024-04-26T04:18:47.847 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.770+0000 7f75af84b640 10 monclient: shutdown 2024-04-26T04:18:47.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.770+0000 7f75af84b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:47.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.770+0000 7f75af84b640 1 -- 172.21.15.63:0/842670099 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f75a8140610 msgr2=0x7f75a81d3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:47.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.770+0000 7f75af84b640 1 --2- 172.21.15.63:0/842670099 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f75a8140610 0x7f75a81d3740 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f759801a900 tx=0x7f7598018ff0 comp rx=0 tx=0).stop 2024-04-26T04:18:47.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.770+0000 7f75adde8640 1 -- 172.21.15.63:0/842670099 reap_dead start 2024-04-26T04:18:47.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.770+0000 7f75af84b640 1 -- 172.21.15.63:0/842670099 shutdown_connections 2024-04-26T04:18:47.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.770+0000 7f75af84b640 1 -- 172.21.15.63:0/842670099 >> 172.21.15.63:0/842670099 conn(0x7f75a80a5f40 msgr2=0x7f75a80a8680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:47.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.770+0000 7f75af84b640 1 -- 172.21.15.63:0/842670099 shutdown_connections 2024-04-26T04:18:47.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.770+0000 7f75af84b640 1 -- 172.21.15.63:0/842670099 wait complete. 2024-04-26T04:18:47.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.774+0000 7fd514a53640 1 -- 172.21.15.63:0/3529123244 --> [v2:172.21.15.186:6807/4227496981,v1:172.21.15.186:6808/4227496981] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd51013c710 con 0x7fd4f003bca0 2024-04-26T04:18:47.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.778+0000 7fd4feffd640 1 -- 172.21.15.63:0/3529123244 <== osd.3 v2:172.21.15.186:6807/4227496981 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fd51013c710 con 0x7fd4f003bca0 2024-04-26T04:18:47.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.778+0000 7fd514a53640 1 -- 172.21.15.63:0/3529123244 >> [v2:172.21.15.186:6807/4227496981,v1:172.21.15.186:6808/4227496981] conn(0x7fd4f003bca0 msgr2=0x7fd4f003e0c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:47.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.778+0000 7fd514a53640 1 --2- 172.21.15.63:0/3529123244 >> [v2:172.21.15.186:6807/4227496981,v1:172.21.15.186:6808/4227496981] conn(0x7fd4f003bca0 0x7fd4f003e0c0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:47.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.778+0000 7fd514a53640 1 -- 172.21.15.63:0/3529123244 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fd4f0035e30 msgr2=0x7fd4f00382f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:47.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.778+0000 7fd514a53640 1 --2- 172.21.15.63:0/3529123244 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fd4f0035e30 0x7fd4f00382f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd5000ac000 tx=0x7fd5000a7000 comp rx=0 tx=0).stop 2024-04-26T04:18:47.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.778+0000 7fd514a53640 10 monclient: shutdown 2024-04-26T04:18:47.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.778+0000 7fd514a53640 20 monclient: shutdown canceling and discarding version request 1 2024-04-26T04:18:47.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.778+0000 7fd514a53640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:47.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.778+0000 7fd514a53640 1 -- 172.21.15.63:0/3529123244 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd510137260 msgr2=0x7fd5101364b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:47.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.778+0000 7fd514a53640 1 --2- 172.21.15.63:0/3529123244 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd510137260 0x7fd5101364b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd50800b9a0 tx=0x7fd50800be60 comp rx=0 tx=0).stop 2024-04-26T04:18:47.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.778+0000 7fd50ed9d640 1 -- 172.21.15.63:0/3529123244 reap_dead start 2024-04-26T04:18:47.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.778+0000 7fd4feffd640 1 -- 172.21.15.63:0/3529123244 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (unknown 0 0 0) 0x7fd50804d200 con 0x7fd510137260 2024-04-26T04:18:47.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.778+0000 7fd4feffd640 10 monclient: discarding stray monitor message mon_get_version_reply(handle=1 version=15) 2024-04-26T04:18:47.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.778+0000 7fd514a53640 1 -- 172.21.15.63:0/3529123244 shutdown_connections 2024-04-26T04:18:47.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.778+0000 7fd514a53640 1 -- 172.21.15.63:0/3529123244 >> 172.21.15.63:0/3529123244 conn(0x7fd5100a7360 msgr2=0x7fd5100a5ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:47.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.778+0000 7fd514a53640 1 -- 172.21.15.63:0/3529123244 shutdown_connections 2024-04-26T04:18:47.851 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.778+0000 7fd514a53640 1 -- 172.21.15.63:0/3529123244 wait complete. 2024-04-26T04:18:47.851 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.778+0000 7f6f5b1c6640 1 -- 172.21.15.63:0/891658063 --> [v2:172.21.15.186:6816/3109931567,v1:172.21.15.186:6817/3109931567] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6f5412b9e0 con 0x7f6f4003bc30 2024-04-26T04:18:47.851 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.778+0000 7f6f497fa640 1 -- 172.21.15.63:0/891658063 <== osd.5 v2:172.21.15.186:6816/3109931567 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f6f5412b9e0 con 0x7f6f4003bc30 2024-04-26T04:18:47.851 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.778+0000 7f695ed41640 1 -- 172.21.15.63:0/2617927876 --> [v2:172.21.15.63:6800/2758152700,v1:172.21.15.63:6801/2758152700] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f695808cfd0 con 0x7f694403bd70 2024-04-26T04:18:47.852 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.778+0000 7f6f5b1c6640 1 -- 172.21.15.63:0/891658063 >> [v2:172.21.15.186:6816/3109931567,v1:172.21.15.186:6817/3109931567] conn(0x7f6f4003bc30 msgr2=0x7f6f4003e050 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:47.852 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.778+0000 7f6f5b1c6640 1 --2- 172.21.15.63:0/891658063 >> [v2:172.21.15.186:6816/3109931567,v1:172.21.15.186:6817/3109931567] conn(0x7f6f4003bc30 0x7f6f4003e050 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:47.852 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.778+0000 7f6f5b1c6640 1 -- 172.21.15.63:0/891658063 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f6f40035e30 msgr2=0x7f6f400382f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:47.852 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.778+0000 7f6f5b1c6640 1 --2- 172.21.15.63:0/891658063 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f6f40035e30 0x7f6f400382f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:47.852 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.782+0000 7f6f5b1c6640 10 monclient: shutdown 2024-04-26T04:18:47.852 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.782+0000 7f6f5b1c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:47.852 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.782+0000 7f6f5b1c6640 1 -- 172.21.15.63:0/891658063 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6f54135680 msgr2=0x7f6f541344f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:47.852 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.782+0000 7f6f5b1c6640 1 --2- 172.21.15.63:0/891658063 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6f54135680 0x7f6f541344f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c01af40 tx=0x7f6f4c019630 comp rx=0 tx=0).stop 2024-04-26T04:18:47.852 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.782+0000 7f6f59763640 1 -- 172.21.15.63:0/891658063 reap_dead start 2024-04-26T04:18:47.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.782+0000 7f69557fa640 1 -- 172.21.15.63:0/2617927876 <== osd.0 v2:172.21.15.63:6800/2758152700 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f695808cfd0 con 0x7f694403bd70 2024-04-26T04:18:47.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.782+0000 7f6f5b1c6640 1 -- 172.21.15.63:0/891658063 shutdown_connections 2024-04-26T04:18:47.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.782+0000 7f6f5b1c6640 1 -- 172.21.15.63:0/891658063 >> 172.21.15.63:0/891658063 conn(0x7f6f540a7360 msgr2=0x7f6f5412dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:47.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.782+0000 7f6f5b1c6640 1 -- 172.21.15.63:0/891658063 shutdown_connections 2024-04-26T04:18:47.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.782+0000 7f6f5b1c6640 1 -- 172.21.15.63:0/891658063 wait complete. 2024-04-26T04:18:47.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.782+0000 7f695ed41640 1 -- 172.21.15.63:0/2617927876 >> [v2:172.21.15.63:6800/2758152700,v1:172.21.15.63:6801/2758152700] conn(0x7f694403bd70 msgr2=0x7f694403e190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:47.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.782+0000 7f695ed41640 1 --2- 172.21.15.63:0/2617927876 >> [v2:172.21.15.63:6800/2758152700,v1:172.21.15.63:6801/2758152700] conn(0x7f694403bd70 0x7f694403e190 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:47.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.782+0000 7f695ed41640 1 -- 172.21.15.63:0/2617927876 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f6944035e80 msgr2=0x7f6944038340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:47.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.782+0000 7f695ed41640 1 --2- 172.21.15.63:0/2617927876 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f6944035e80 0x7f6944038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:47.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.782+0000 7f695ed41640 10 monclient: shutdown 2024-04-26T04:18:47.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.782+0000 7f695ed41640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:47.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.782+0000 7f695ed41640 1 -- 172.21.15.63:0/2617927876 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f69580a2cd0 msgr2=0x7f6958139220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:47.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.782+0000 7f695ed41640 1 --2- 172.21.15.63:0/2617927876 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f69580a2cd0 0x7f6958139220 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f694800c450 tx=0x7f694800c910 comp rx=0 tx=0).stop 2024-04-26T04:18:47.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.782+0000 7f695d2de640 1 -- 172.21.15.63:0/2617927876 reap_dead start 2024-04-26T04:18:47.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.782+0000 7f695ed41640 1 -- 172.21.15.63:0/2617927876 shutdown_connections 2024-04-26T04:18:47.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.782+0000 7f695ed41640 1 -- 172.21.15.63:0/2617927876 >> 172.21.15.63:0/2617927876 conn(0x7f69580a5f40 msgr2=0x7f69580a1a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:47.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.782+0000 7f695ed41640 1 -- 172.21.15.63:0/2617927876 shutdown_connections 2024-04-26T04:18:47.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.782+0000 7f695ed41640 1 -- 172.21.15.63:0/2617927876 wait complete. 2024-04-26T04:18:47.856 INFO:teuthology.orchestra.run.smithi063.stdout:47244640259 2024-04-26T04:18:47.856 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-26T04:18:47.857 INFO:teuthology.orchestra.run.smithi063.stdout:47244640259 2024-04-26T04:18:47.857 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-26T04:18:47.858 INFO:teuthology.orchestra.run.smithi063.stdout:47244640259 2024-04-26T04:18:47.858 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-26T04:18:47.859 INFO:teuthology.orchestra.run.smithi063.stdout:51539607557 2024-04-26T04:18:47.859 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-26T04:18:47.995 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.990+0000 7fbebb487640 10 monclient: get_monmap_and_config 2024-04-26T04:18:47.996 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.990+0000 7fbebb487640 10 monclient: build_initial_monmap 2024-04-26T04:18:47.996 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.990+0000 7fbebb487640 10 monclient: monmap: 2024-04-26T04:18:47.996 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:47.996 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:47.996 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:47.995230+0000 2024-04-26T04:18:47.996 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:47.995230+0000 2024-04-26T04:18:47.996 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:47.996 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:47.996 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:18:47.996 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:18:47.997 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:18:47.997 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:18:47.997 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:18:47.997 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:47.997 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.990+0000 7fbebb487640 1 Processor -- start 2024-04-26T04:18:47.997 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.990+0000 7fbebb487640 1 -- start start 2024-04-26T04:18:47.997 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.990+0000 7fbebb487640 10 monclient: init 2024-04-26T04:18:47.997 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.990+0000 7f743a483640 10 monclient: get_monmap_and_config 2024-04-26T04:18:47.998 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.990+0000 7f743a483640 10 monclient: build_initial_monmap 2024-04-26T04:18:47.998 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.990+0000 7f743a483640 10 monclient: monmap: 2024-04-26T04:18:47.998 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:47.998 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:47.998 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:47.996066+0000 2024-04-26T04:18:47.998 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:47.996066+0000 2024-04-26T04:18:47.998 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:47.998 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:47.998 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:18:47.998 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:18:47.998 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:18:47.998 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:18:47.999 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:18:47.999 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:48.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.990+0000 7fbebb487640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:48.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.990+0000 7fbebb487640 10 monclient: _add_conns ranks=[4,3,0,1,2] 2024-04-26T04:18:48.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.990+0000 7fbebb487640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fbeb40a2cd0 0x7fbeb40a30b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:48.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.990+0000 7fbebb487640 10 monclient(hunting): picked mon.noname-c con 0x7fbeb40a2cd0 addr v2:172.21.15.186:3300/0 2024-04-26T04:18:48.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.990+0000 7fbebb487640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7fbeb40a47d0 0x7fbeb4136ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:48.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.990+0000 7fbebb487640 10 monclient(hunting): picked mon.noname-a con 0x7fbeb40a47d0 addr v2:172.21.15.63:3300/0 2024-04-26T04:18:48.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.990+0000 7fbebb487640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbeb40a4c00 addr v1:172.21.15.63:6789/0 2024-04-26T04:18:48.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.990+0000 7fbeb9a24640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7fbeb40a4c00 0x7fbeb4137220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:44176/0 (socket says 172.21.15.63:44176) 2024-04-26T04:18:48.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.990+0000 7fbeb9a24640 1 -- 172.21.15.63:0/773491537 learned_addr learned my addr 172.21.15.63:0/773491537 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-04-26T04:18:48.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.990+0000 7fbeb8a22640 1 --2- 172.21.15.63:0/773491537 >> v2:172.21.15.63:3300/0 conn(0x7fbeb40a47d0 0x7fbeb4136ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:48.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.990+0000 7fbebb487640 1 -- 172.21.15.63:0/773491537 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbeb40a0fb0 con 0x7fbeb40a4c00 2024-04-26T04:18:48.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.990+0000 7fbebb487640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:48.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.990+0000 7fbebb487640 1 -- 172.21.15.63:0/773491537 --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7fbeb40a35f0 con 0x7fbeb40a47d0 2024-04-26T04:18:48.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.990+0000 7fbebb487640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:48.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.990+0000 7fbebb487640 1 -- 172.21.15.63:0/773491537 --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fbeb40a5380 con 0x7fbeb40a2cd0 2024-04-26T04:18:48.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.990+0000 7fbebb487640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:48.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.990+0000 7fbebb487640 10 monclient(hunting): authenticate will time out at 959.703119s 2024-04-26T04:18:48.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.990+0000 7fbeb8a22640 10 monclient(hunting): get_auth_request con 0x7fbeb40a47d0 auth_method 0 2024-04-26T04:18:48.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.990+0000 7fbeb8a22640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:48.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.990+0000 7fbeb8a22640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:48.002 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.990+0000 7fbeb8a22640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:48.002 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.990+0000 7fbeb9223640 1 --2- 172.21.15.63:0/773491537 >> v2:172.21.15.186:3300/0 conn(0x7fbeb40a2cd0 0x7fbeb40a30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:48.002 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.990+0000 7fbeb9223640 10 monclient(hunting): get_auth_request con 0x7fbeb40a2cd0 auth_method 0 2024-04-26T04:18:48.002 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.990+0000 7fbeb9223640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:48.002 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.990+0000 7fbeb9223640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:48.002 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.990+0000 7fbeb9223640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:48.002 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.990+0000 7fbeb8a22640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:48.002 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.990+0000 7fbeb8a22640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:48.002 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.990+0000 7fbeb8a22640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:48.002 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.990+0000 7fbeb9223640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:48.002 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.990+0000 7fbeb9223640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:48.002 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.990+0000 7fbeb9223640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:48.003 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.990+0000 7f743a483640 1 Processor -- start 2024-04-26T04:18:48.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.990+0000 7fbeb8a22640 10 monclient(hunting): handle_auth_done global_id 4236 payload 274 2024-04-26T04:18:48.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.990+0000 7fbeb8a22640 1 -- 172.21.15.63:0/773491537 >> v2:172.21.15.186:3300/0 conn(0x7fbeb40a2cd0 msgr2=0x7fbeb40a30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:48.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.990+0000 7fbeb8a22640 1 --2- 172.21.15.63:0/773491537 >> v2:172.21.15.186:3300/0 conn(0x7fbeb40a2cd0 0x7fbeb40a30b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:48.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.990+0000 7fbeb8a22640 1 -- 172.21.15.63:0/773491537 >> v1:172.21.15.63:6789/0 conn(0x7fbeb40a4c00 legacy=0x7fbeb4137220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:48.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.990+0000 7fbeb8a22640 10 monclient: _finish_hunting 0 2024-04-26T04:18:48.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.990+0000 7fbeb8a22640 1 monclient: found mon.noname-a 2024-04-26T04:18:48.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.990+0000 7fbeb8a22640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:48.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.990+0000 7fbeb8a22640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:18:48.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.990+0000 7fbeb8a22640 1 -- 172.21.15.63:0/773491537 --> v2:172.21.15.63:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbeb40a1290 con 0x7fbeb40a47d0 2024-04-26T04:18:48.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.990+0000 7f743a483640 1 -- start start 2024-04-26T04:18:48.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.990+0000 7f743a483640 10 monclient: init 2024-04-26T04:18:48.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.994+0000 7f743a483640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:48.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.994+0000 7f743a483640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-04-26T04:18:48.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.994+0000 7f743a483640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7434071db0 addr v1:172.21.15.63:6789/0 2024-04-26T04:18:48.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.994+0000 7f743a483640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f7434071250 0x7f7434138500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:48.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.994+0000 7f743a483640 10 monclient(hunting): picked mon.noname-b con 0x7f7434071250 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:48.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.994+0000 7f743a483640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7434071660 addr v1:172.21.15.186:6789/0 2024-04-26T04:18:48.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.994+0000 7f743a483640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f743406e8b0 con 0x7f7434071db0 2024-04-26T04:18:48.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.994+0000 7f743a483640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f743406ed20 con 0x7f7434071660 2024-04-26T04:18:48.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.994+0000 7f743a483640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:48.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.994+0000 7f743a483640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f74340728a0 con 0x7f7434071250 2024-04-26T04:18:48.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.994+0000 7f743a483640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:48.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.994+0000 7f743a483640 10 monclient(hunting): authenticate will time out at 959.704275s 2024-04-26T04:18:48.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.994+0000 7f74337fe640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f7434071250 0x7f7434138500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:48.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.994+0000 7f74337fe640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f7434071250 0x7f7434138500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:57844/0 (socket says 172.21.15.63:57844) 2024-04-26T04:18:48.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.994+0000 7f74337fe640 1 -- 172.21.15.63:0/632535251 learned_addr learned my addr 172.21.15.63:0/632535251 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:18:48.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.994+0000 7f74337fe640 10 monclient(hunting): get_auth_request con 0x7f7434071250 auth_method 0 2024-04-26T04:18:48.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.994+0000 7f74337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:48.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.994+0000 7f74337fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:48.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.994+0000 7f74337fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:48.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.994+0000 7f74337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:48.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.994+0000 7f74337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:48.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.994+0000 7f74337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:48.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.994+0000 7f74337fe640 10 monclient(hunting): handle_auth_done global_id 4223 payload 274 2024-04-26T04:18:48.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.994+0000 7f74337fe640 1 -- 172.21.15.63:0/632535251 >> v1:172.21.15.186:6789/0 conn(0x7f7434071660 legacy=0x7f743413aa50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T04:18:48.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.994+0000 7f74337fe640 1 -- 172.21.15.63:0/632535251 >> v1:172.21.15.63:6789/0 conn(0x7f7434071db0 legacy=0x7f7434072190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T04:18:48.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.994+0000 7f74337fe640 10 monclient: _finish_hunting 0 2024-04-26T04:18:48.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.994+0000 7f74337fe640 1 monclient: found mon.noname-b 2024-04-26T04:18:48.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.994+0000 7f74337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:48.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.994+0000 7f74337fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.63:3301/0 2024-04-26T04:18:48.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.994+0000 7f74337fe640 1 -- 172.21.15.63:0/632535251 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f743406ffd0 con 0x7f7434071250 2024-04-26T04:18:48.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.998+0000 7f74337fe640 1 -- 172.21.15.63:0/632535251 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f7434071250 msgr2=0x7f7434138500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:48.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.998+0000 7f74337fe640 1 --2- 172.21.15.63:0/632535251 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f7434071250 0x7f7434138500 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f74280098a0 tx=0x7f742801c3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bcd33134273ce58e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:48.010 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.998+0000 7fbeb8a22640 1 -- 172.21.15.63:0/773491537 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbeb40a47d0 msgr2=0x7fbeb4136ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:48.010 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.998+0000 7fbeb8a22640 1 --2- 172.21.15.63:0/773491537 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbeb40a47d0 0x7fbeb4136ce0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fbeb406c760 tx=0x7fbea401c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=716912ce0d00961f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:48.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.998+0000 7f7432ffd640 1 -- 172.21.15.63:0/632535251 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74280b0070 con 0x7f7434071250 2024-04-26T04:18:48.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.998+0000 7f7432ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:48.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.998+0000 7f7432ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T04:18:48.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.998+0000 7f7432ffd640 10 monclient: dump: 2024-04-26T04:18:48.011 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:48.011 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:48.011 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:48.011 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:48.011 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:48.012 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:48.012 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:48.012 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:48.012 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:48.012 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:48.012 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.998+0000 7f7432ffd640 10 monclient: _finish_auth 0 2024-04-26T04:18:48.012 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.998+0000 7f7432ffd640 10 monclient: _check_auth_tickets 2024-04-26T04:18:48.012 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.998+0000 7f7432ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:48.012 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.998+0000 7f743a483640 5 monclient: authenticate success, global_id 4223 2024-04-26T04:18:48.013 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.998+0000 7f743a483640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T04:18:48.013 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.998+0000 7f7432ffd640 1 -- 172.21.15.63:0/632535251 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74280ab600 con 0x7f7434071250 2024-04-26T04:18:48.013 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.998+0000 7f7432ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:48.013 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.998+0000 7f7432ffd640 1 -- 172.21.15.63:0/632535251 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74280ab9d0 con 0x7f7434071250 2024-04-26T04:18:48.013 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.998+0000 7f7432ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:48.013 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.998+0000 7f7432ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:18:48.013 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.998+0000 7f7432ffd640 10 monclient: dump: 2024-04-26T04:18:48.013 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:48.013 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:48.013 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:48.013 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:48.014 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:48.014 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:48.014 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:48.014 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:48.014 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:48.014 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:48.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.998+0000 7f743a483640 10 monclient: get_monmap_and_config success 2024-04-26T04:18:48.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.998+0000 7f743a483640 4 set_mon_vals no callback set 2024-04-26T04:18:48.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.998+0000 7fbeabfff640 1 -- 172.21.15.63:0/773491537 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbea40b0070 con 0x7fbeb40a47d0 2024-04-26T04:18:48.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.998+0000 7fbeabfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:48.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.998+0000 7fbeabfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:18:48.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.998+0000 7fbeabfff640 10 monclient: dump: 2024-04-26T04:18:48.015 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:48.015 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:48.015 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:48.015 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:48.016 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:48.016 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:48.016 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:48.016 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:48.016 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:48.016 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:48.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.998+0000 7fbeabfff640 10 monclient: _finish_auth 0 2024-04-26T04:18:48.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.998+0000 7fbeabfff640 10 monclient: _check_auth_tickets 2024-04-26T04:18:48.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.998+0000 7fbeabfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:48.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.998+0000 7fbeabfff640 1 -- 172.21.15.63:0/773491537 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbea40ab600 con 0x7fbeb40a47d0 2024-04-26T04:18:48.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.998+0000 7fbeabfff640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:48.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.998+0000 7fbeabfff640 1 -- 172.21.15.63:0/773491537 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbea40ab9d0 con 0x7fbeb40a47d0 2024-04-26T04:18:48.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.998+0000 7fbeabfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:48.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.998+0000 7fbeabfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:18:48.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.998+0000 7fbeabfff640 10 monclient: dump: 2024-04-26T04:18:48.017 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:48.017 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:48.017 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:48.018 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:48.018 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:48.018 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:48.018 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:48.018 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:48.018 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:48.018 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:48.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.998+0000 7fbebb487640 5 monclient: authenticate success, global_id 4236 2024-04-26T04:18:48.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.998+0000 7fbebb487640 10 monclient: get_monmap_and_config success 2024-04-26T04:18:48.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.998+0000 7f743a483640 10 monclient: shutdown 2024-04-26T04:18:48.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.998+0000 7f743a483640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:48.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.998+0000 7f743a483640 1 -- 172.21.15.63:0/632535251 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f7434071250 msgr2=0x7f7434138500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:48.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.998+0000 7f743a483640 1 --2- 172.21.15.63:0/632535251 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f7434071250 0x7f7434138500 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f74280098a0 tx=0x7f742801c3c0 comp rx=0 tx=0).stop 2024-04-26T04:18:48.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.998+0000 7f743a483640 1 -- 172.21.15.63:0/632535251 shutdown_connections 2024-04-26T04:18:48.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.998+0000 7f743a483640 1 --2- 172.21.15.63:0/632535251 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f7434071250 0x7f7434138500 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:48.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.998+0000 7f743a483640 1 -- 172.21.15.63:0/632535251 >> 172.21.15.63:0/632535251 conn(0x7f74340a6170 msgr2=0x7f74340a2590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:48.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.998+0000 7f743a483640 1 -- 172.21.15.63:0/632535251 shutdown_connections 2024-04-26T04:18:48.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.998+0000 7f743a483640 1 -- 172.21.15.63:0/632535251 wait complete. 2024-04-26T04:18:48.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.998+0000 7fbebb487640 4 set_mon_vals no callback set 2024-04-26T04:18:48.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.998+0000 7fbebb487640 10 monclient: shutdown 2024-04-26T04:18:48.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.998+0000 7fbebb487640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:48.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.998+0000 7fbebb487640 1 -- 172.21.15.63:0/773491537 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbeb40a47d0 msgr2=0x7fbeb4136ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:48.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.998+0000 7fbebb487640 1 --2- 172.21.15.63:0/773491537 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbeb40a47d0 0x7fbeb4136ce0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fbeb406c760 tx=0x7fbea401c420 comp rx=0 tx=0).stop 2024-04-26T04:18:48.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.998+0000 7fbebb487640 1 -- 172.21.15.63:0/773491537 shutdown_connections 2024-04-26T04:18:48.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.998+0000 7fbebb487640 1 --2- 172.21.15.63:0/773491537 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbeb40a47d0 0x7fbeb4136ce0 secure :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fbeb406c760 tx=0x7fbea401c420 comp rx=0 tx=0).stop 2024-04-26T04:18:48.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.998+0000 7fbebb487640 1 --2- 172.21.15.63:0/773491537 >> v2:172.21.15.186:3300/0 conn(0x7fbeb40a2cd0 0x7fbeb40a30b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:48.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.998+0000 7fbebb487640 1 -- 172.21.15.63:0/773491537 >> 172.21.15.63:0/773491537 conn(0x7fbeb40a5f40 msgr2=0x7fbeb40a6320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:48.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.998+0000 7fbebb487640 1 -- 172.21.15.63:0/773491537 shutdown_connections 2024-04-26T04:18:48.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.998+0000 7f743a483640 10 monclient: build_initial_monmap 2024-04-26T04:18:48.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.998+0000 7f743a483640 10 monclient: monmap: 2024-04-26T04:18:48.021 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:48.021 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:18:48.021 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:18:48.022 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:18:48.022 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:48.022 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:48.022 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:18:48.022 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:18:48.022 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:18:48.022 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:48.022 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7f743a483640 1 Processor -- start 2024-04-26T04:18:48.022 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7f743a483640 1 -- start start 2024-04-26T04:18:48.022 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7f743a483640 10 monclient: init 2024-04-26T04:18:48.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:47.998+0000 7fbebb487640 1 -- 172.21.15.63:0/773491537 wait complete. 2024-04-26T04:18:48.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7fbebb487640 10 monclient: build_initial_monmap 2024-04-26T04:18:48.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7fbebb487640 10 monclient: monmap: 2024-04-26T04:18:48.023 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:48.023 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:18:48.023 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:18:48.023 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:18:48.023 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:48.023 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:48.023 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:18:48.024 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:18:48.024 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:18:48.024 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:48.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7fbebb487640 1 Processor -- start 2024-04-26T04:18:48.027 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7f743a483640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:48.027 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7f743a483640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T04:18:48.027 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7f743a483640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7434071250 0x7f74341cca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:48.027 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7f743a483640 10 monclient(hunting): picked mon.noname-b con 0x7f7434071250 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:18:48.027 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7f743a483640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f7434071660 0x7f74341cd130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:48.027 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7f743a483640 10 monclient(hunting): picked mon.noname-c con 0x7f7434071660 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:48.027 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7f743a483640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7434071db0 0x7f74341373a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:48.027 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7f743a483640 10 monclient(hunting): picked mon.noname-a con 0x7f7434071db0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:18:48.027 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7f743a483640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:48.027 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7f743a483640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f743406c1b0 con 0x7f7434071db0 2024-04-26T04:18:48.027 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7f743a483640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:48.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7f743a483640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f743406c030 con 0x7f7434071250 2024-04-26T04:18:48.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7f743a483640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:48.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7f743a483640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f743406c330 con 0x7f7434071660 2024-04-26T04:18:48.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7f743a483640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:48.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7f74337fe640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f7434071660 0x7f74341cd130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:48.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7f743a483640 10 monclient(hunting): authenticate will time out at 959.713393s 2024-04-26T04:18:48.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7f74337fe640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f7434071660 0x7f74341cd130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:57856/0 (socket says 172.21.15.63:57856) 2024-04-26T04:18:48.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7f74337fe640 1 -- 172.21.15.63:0/1808898411 learned_addr learned my addr 172.21.15.63:0/1808898411 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:18:48.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7f74337fe640 10 monclient(hunting): get_auth_request con 0x7f7434071660 auth_method 0 2024-04-26T04:18:48.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7f74337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:48.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7f74337fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:48.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7f74337fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:48.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7f7438a20640 1 --2- 172.21.15.63:0/1808898411 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7434071db0 0x7f74341373a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:48.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7f7438a20640 10 monclient(hunting): get_auth_request con 0x7f7434071db0 auth_method 0 2024-04-26T04:18:48.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7f7438a20640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:48.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7f7438a20640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:48.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7f7438a20640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:48.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7f74337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:48.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7f74337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:48.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7f74337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:48.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7f7433fff640 1 --2- 172.21.15.63:0/1808898411 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7434071250 0x7f74341cca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:48.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7f7433fff640 10 monclient(hunting): get_auth_request con 0x7f7434071250 auth_method 0 2024-04-26T04:18:48.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7f7433fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:48.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7f7433fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:48.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7f7433fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:48.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7f7438a20640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:48.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7f7438a20640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:48.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7f7438a20640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:48.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7fbebb487640 1 -- start start 2024-04-26T04:18:48.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7fbebb487640 10 monclient: init 2024-04-26T04:18:48.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7fbebb487640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:48.031 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7fbebb487640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T04:18:48.031 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7fbebb487640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbeb40a2cd0 0x7fbeb41dbd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:48.031 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7fbebb487640 10 monclient(hunting): picked mon.noname-b con 0x7fbeb40a2cd0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:18:48.031 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7fbebb487640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbeb40a4c00 0x7fbeb41dc2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:48.031 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7fbebb487640 10 monclient(hunting): picked mon.noname-a con 0x7fbeb40a4c00 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:18:48.031 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7fbebb487640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fbeb41dc7e0 0x7fbeb4130aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:48.031 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7fbebb487640 10 monclient(hunting): picked mon.noname-c con 0x7fbeb41dc7e0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:48.031 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7fbebb487640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:48.031 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7fbebb487640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fbeb4141a50 con 0x7fbeb40a4c00 2024-04-26T04:18:48.031 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7fbebb487640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:48.031 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7fbebb487640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbeb4137b40 con 0x7fbeb40a2cd0 2024-04-26T04:18:48.031 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7fbebb487640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:48.031 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7fbebb487640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7fbeb4141bd0 con 0x7fbeb41dc7e0 2024-04-26T04:18:48.032 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7fbebb487640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:48.032 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7fbebb487640 10 monclient(hunting): authenticate will time out at 959.713665s 2024-04-26T04:18:48.032 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7f7433fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:48.032 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7f7433fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:48.032 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7f7433fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:48.032 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7f7438a20640 10 monclient(hunting): handle_auth_done global_id 4239 payload 824 2024-04-26T04:18:48.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7fbeb9a24640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fbeb41dc7e0 0x7fbeb4130aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:48.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7fbeb9a24640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fbeb41dc7e0 0x7fbeb4130aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:57862/0 (socket says 172.21.15.63:57862) 2024-04-26T04:18:48.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7fbeb9a24640 1 -- 172.21.15.63:0/1638960592 learned_addr learned my addr 172.21.15.63:0/1638960592 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:18:48.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7fbeb9a24640 10 monclient(hunting): get_auth_request con 0x7fbeb41dc7e0 auth_method 0 2024-04-26T04:18:48.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7fbeb9a24640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:48.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7fbeb9a24640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:48.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7fbeb9a24640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:48.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7fbeb9a24640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:48.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7fbeb9a24640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:48.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7f7438a20640 1 -- 172.21.15.63:0/1808898411 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f7434071660 msgr2=0x7f74341cd130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:48.035 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7fbeb9a24640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:48.035 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7fbeb8a22640 1 --2- 172.21.15.63:0/1638960592 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbeb40a4c00 0x7fbeb41dc2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:48.035 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7fbeb8a22640 10 monclient(hunting): get_auth_request con 0x7fbeb40a4c00 auth_method 0 2024-04-26T04:18:48.035 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7fbeb8a22640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:48.035 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7fbeb8a22640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:48.035 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7fbeb8a22640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:48.035 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7fbeb9223640 1 --2- 172.21.15.63:0/1638960592 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbeb40a2cd0 0x7fbeb41dbd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:48.035 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7fbeb9223640 10 monclient(hunting): get_auth_request con 0x7fbeb40a2cd0 auth_method 0 2024-04-26T04:18:48.035 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7fbeb9223640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:48.035 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7fbeb9223640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:48.035 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7fbeb9223640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:48.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7fbeb8a22640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:48.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7fbeb8a22640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:48.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7fbeb8a22640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:48.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7fbeb9a24640 10 monclient(hunting): handle_auth_done global_id 4229 payload 824 2024-04-26T04:18:48.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7fbeb9a24640 1 -- 172.21.15.63:0/1638960592 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbeb40a2cd0 msgr2=0x7fbeb41dbd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:48.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7fbeb9a24640 1 --2- 172.21.15.63:0/1638960592 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbeb40a2cd0 0x7fbeb41dbd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:48.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7fbeb9a24640 1 -- 172.21.15.63:0/1638960592 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbeb40a4c00 msgr2=0x7fbeb41dc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:48.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7fbeb9a24640 1 --2- 172.21.15.63:0/1638960592 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbeb40a4c00 0x7fbeb41dc2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:48.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7fbeb9a24640 10 monclient: _finish_hunting 0 2024-04-26T04:18:48.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7fbeb9a24640 1 monclient: found mon.noname-c 2024-04-26T04:18:48.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7fbeb9a24640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:48.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7fbeb9a24640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.63:3301/0 2024-04-26T04:18:48.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7fbeb9a24640 1 -- 172.21.15.63:0/1638960592 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbeb4134e90 con 0x7fbeb41dc7e0 2024-04-26T04:18:48.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7f7438a20640 1 --2- 172.21.15.63:0/1808898411 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f7434071660 0x7f74341cd130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:48.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7f7438a20640 1 -- 172.21.15.63:0/1808898411 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7434071250 msgr2=0x7f74341cca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:48.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7f7438a20640 1 --2- 172.21.15.63:0/1808898411 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7434071250 0x7f74341cca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:48.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7f7438a20640 10 monclient: _finish_hunting 0 2024-04-26T04:18:48.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7f7438a20640 1 monclient: found mon.noname-a 2024-04-26T04:18:48.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7f7438a20640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:48.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7f7438a20640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:18:48.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7f7438a20640 1 -- 172.21.15.63:0/1808898411 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7434072a40 con 0x7f7434071db0 2024-04-26T04:18:48.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7f7433fff640 1 --2- 172.21.15.63:0/1808898411 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7434071250 0x7f74341cca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T04:18:48.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.002+0000 7f74337fe640 1 --2- 172.21.15.63:0/1808898411 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f7434071660 0x7f74341cd130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T04:18:48.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.006+0000 7fbeb9a24640 1 -- 172.21.15.63:0/1638960592 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fbeb41dc7e0 msgr2=0x7fbeb4130aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:48.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.006+0000 7fbeb9a24640 1 --2- 172.21.15.63:0/1638960592 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fbeb41dc7e0 0x7fbeb4130aa0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fbeac00caf0 tx=0x7fbeac00a060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:48.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.006+0000 7fbea9ffb640 1 -- 172.21.15.63:0/1638960592 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeac012400 con 0x7fbeb41dc7e0 2024-04-26T04:18:48.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.006+0000 7fbea9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:48.039 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.006+0000 7fbea9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:18:48.039 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.006+0000 7fbea9ffb640 10 monclient: dump: 2024-04-26T04:18:48.039 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:48.039 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:48.039 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:48.039 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:48.039 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:48.039 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:48.039 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:48.039 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:48.039 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:48.039 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:48.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.006+0000 7f7438a20640 1 -- 172.21.15.63:0/1808898411 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7434071db0 msgr2=0x7f74341373a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:48.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.006+0000 7f7438a20640 1 --2- 172.21.15.63:0/1808898411 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7434071db0 0x7f74341373a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f742c00be00 tx=0x7f742c00bef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:48.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.006+0000 7f7430ff9640 1 -- 172.21.15.63:0/1808898411 DispatchQueue::entry inject delay of 0.002000 2024-04-26T04:18:48.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.006+0000 7fbea9ffb640 10 monclient: _finish_auth 0 2024-04-26T04:18:48.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.006+0000 7fbea9ffb640 10 monclient: _check_auth_tickets 2024-04-26T04:18:48.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.006+0000 7fbea9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:48.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.006+0000 7fbea9ffb640 1 -- 172.21.15.63:0/1638960592 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbeac0125c0 con 0x7fbeb41dc7e0 2024-04-26T04:18:48.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.006+0000 7fbebb487640 5 monclient: authenticate success, global_id 4229 2024-04-26T04:18:48.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.006+0000 7fbea9ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:48.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.006+0000 7fbea9ffb640 1 -- 172.21.15.63:0/1638960592 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeac0129b0 con 0x7fbeb41dc7e0 2024-04-26T04:18:48.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.006+0000 7fbea9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:48.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.006+0000 7fbea9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:18:48.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.006+0000 7fbea9ffb640 10 monclient: dump: 2024-04-26T04:18:48.043 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:48.043 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:48.043 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:48.043 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:48.043 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:48.043 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:48.043 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:48.043 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:48.043 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:48.044 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:48.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.006+0000 7fbeab7fe640 4 set_mon_vals no callback set 2024-04-26T04:18:48.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.006+0000 7fbebb487640 10 monclient: _renew_subs 2024-04-26T04:18:48.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.006+0000 7fbebb487640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:48.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.006+0000 7fbebb487640 1 -- 172.21.15.63:0/1638960592 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbeb4072b40 con 0x7fbeb41dc7e0 2024-04-26T04:18:48.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.006+0000 7fbebb487640 10 monclient: _renew_subs 2024-04-26T04:18:48.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.006+0000 7fbebb487640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:48.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.006+0000 7fbebb487640 1 -- 172.21.15.63:0/1638960592 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbeb40a37d0 con 0x7fbeb41dc7e0 2024-04-26T04:18:48.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.006+0000 7fbebb487640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:48.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.006+0000 7fbebb487640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:48.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.006+0000 7fbebb487640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:48.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.006+0000 7fbebb487640 1 -- 172.21.15.63:0/1638960592 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbeb4062e10 con 0x7fbeb41dc7e0 2024-04-26T04:18:48.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.006+0000 7fbea9ffb640 1 -- 172.21.15.63:0/1638960592 <== mon.2 v2:172.21.15.63:3301/0 4 ==== mgrmap(e 4) ==== 43997+0+0 (secure 0 0 0) 0x7fbeac012bd0 con 0x7fbeb41dc7e0 2024-04-26T04:18:48.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.010+0000 7fbea9ffb640 1 --2- 172.21.15.63:0/1638960592 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fbe98035e30 0x7fbe980382f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:48.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.010+0000 7fbea9ffb640 1 -- 172.21.15.63:0/1638960592 <== mon.2 v2:172.21.15.63:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7fbeac049d90 con 0x7fbeb41dc7e0 2024-04-26T04:18:48.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.010+0000 7fbeb9223640 1 --2- 172.21.15.63:0/1638960592 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fbe98035e30 0x7fbe980382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:48.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.010+0000 7fbeb9223640 10 monclient: get_auth_request con 0x7fbe98035e30 auth_method 0 2024-04-26T04:18:48.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.010+0000 7fbeb9223640 1 --2- 172.21.15.63:0/1638960592 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fbe98035e30 0x7fbe980382f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fbeb0009af0 tx=0x7fbeb0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:48.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.006+0000 7f7430ff9640 1 -- 172.21.15.63:0/1808898411 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f742c019070 con 0x7f7434071db0 2024-04-26T04:18:48.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.006+0000 7f7430ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:48.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.006+0000 7f7430ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:18:48.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.006+0000 7f7430ff9640 10 monclient: dump: 2024-04-26T04:18:48.047 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:48.047 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:48.047 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:48.047 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:48.047 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:48.048 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:48.048 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:48.048 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:48.048 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:48.048 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:48.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.006+0000 7f7430ff9640 10 monclient: _finish_auth 0 2024-04-26T04:18:48.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.006+0000 7f7430ff9640 10 monclient: _check_auth_tickets 2024-04-26T04:18:48.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.006+0000 7f7430ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:48.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.006+0000 7f7430ff9640 1 -- 172.21.15.63:0/1808898411 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f742c014640 con 0x7f7434071db0 2024-04-26T04:18:48.049 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.006+0000 7f7430ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:48.049 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.006+0000 7f7430ff9640 1 -- 172.21.15.63:0/1808898411 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f742c014a30 con 0x7f7434071db0 2024-04-26T04:18:48.049 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.006+0000 7f7430ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:48.049 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.006+0000 7f7430ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:18:48.049 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.006+0000 7f7430ff9640 10 monclient: dump: 2024-04-26T04:18:48.049 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:48.049 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:48.049 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:48.049 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:48.049 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:48.049 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:48.049 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:48.050 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:48.050 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:48.050 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:48.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.006+0000 7f74327fc640 4 set_mon_vals no callback set 2024-04-26T04:18:48.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.010+0000 7f743a483640 5 monclient: authenticate success, global_id 4239 2024-04-26T04:18:48.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.010+0000 7f743a483640 10 monclient: _renew_subs 2024-04-26T04:18:48.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.010+0000 7f743a483640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:48.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.010+0000 7f743a483640 1 -- 172.21.15.63:0/1808898411 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7434136660 con 0x7f7434071db0 2024-04-26T04:18:48.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.010+0000 7f743a483640 10 monclient: _renew_subs 2024-04-26T04:18:48.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.010+0000 7f743a483640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:48.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.010+0000 7f743a483640 1 -- 172.21.15.63:0/1808898411 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f743413af80 con 0x7f7434071db0 2024-04-26T04:18:48.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.014+0000 7f743a483640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:48.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.014+0000 7f743a483640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:48.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.014+0000 7f743a483640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:48.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.014+0000 7f743a483640 1 -- 172.21.15.63:0/1808898411 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7434062e10 con 0x7f7434071db0 2024-04-26T04:18:48.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.014+0000 7f7430ff9640 1 -- 172.21.15.63:0/1808898411 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 4) ==== 43997+0+0 (secure 0 0 0) 0x7f742c01c070 con 0x7f7434071db0 2024-04-26T04:18:48.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.014+0000 7f7430ff9640 1 --2- 172.21.15.63:0/1808898411 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f742003a290 0x7f742003c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:48.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.014+0000 7f7430ff9640 1 -- 172.21.15.63:0/1808898411 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7f742c048e80 con 0x7f7434071db0 2024-04-26T04:18:48.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.014+0000 7f7433fff640 1 --2- 172.21.15.63:0/1808898411 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f742003a290 0x7f742003c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:48.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.014+0000 7f7433fff640 10 monclient: get_auth_request con 0x7f742003a290 auth_method 0 2024-04-26T04:18:48.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.018+0000 7f7433fff640 1 --2- 172.21.15.63:0/1808898411 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f742003a290 0x7f742003c750 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7424009d50 tx=0x7f74240090e0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:48.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.018+0000 7fbea9ffb640 1 -- 172.21.15.63:0/1638960592 <== mon.2 v2:172.21.15.63:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fbeb4072b40 con 0x7fbeb41dc7e0 2024-04-26T04:18:48.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.018+0000 7fbea9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:48.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.018+0000 7fbea9ffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:18:48.053 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.022+0000 7f7430ff9640 1 -- 172.21.15.63:0/1808898411 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f7434136660 con 0x7f7434071db0 2024-04-26T04:18:48.053 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.022+0000 7f7430ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:48.053 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.022+0000 7f7430ff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:18:48.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.038+0000 7f79bb0a4640 10 monclient: get_monmap_and_config 2024-04-26T04:18:48.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.038+0000 7f79bb0a4640 10 monclient: build_initial_monmap 2024-04-26T04:18:48.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.038+0000 7f79bb0a4640 10 monclient: monmap: 2024-04-26T04:18:48.055 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:48.055 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:48.055 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:48.043819+0000 2024-04-26T04:18:48.055 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:48.043819+0000 2024-04-26T04:18:48.055 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:48.055 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:48.055 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:18:48.055 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:18:48.055 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:18:48.055 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:18:48.055 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:18:48.055 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:48.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.038+0000 7f79bb0a4640 1 Processor -- start 2024-04-26T04:18:48.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.038+0000 7f79bb0a4640 1 -- start start 2024-04-26T04:18:48.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.038+0000 7f79bb0a4640 10 monclient: init 2024-04-26T04:18:48.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.038+0000 7f79bb0a4640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:48.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.038+0000 7f79bb0a4640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-04-26T04:18:48.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.038+0000 7f79bb0a4640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f79b40a2cd0 0x7f79b40a30b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:48.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.038+0000 7f79bb0a4640 10 monclient(hunting): picked mon.noname-a con 0x7f79b40a2cd0 addr v2:172.21.15.63:3300/0 2024-04-26T04:18:48.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.038+0000 7f79bb0a4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f79b40a47d0 addr v1:172.21.15.186:6789/0 2024-04-26T04:18:48.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.038+0000 7f79bb0a4640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f79b40a4c00 0x7f79b41373f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:48.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.038+0000 7f79bb0a4640 10 monclient(hunting): picked mon.noname-b con 0x7f79b40a4c00 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:48.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.038+0000 7f79bb0a4640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79b40a0fb0 con 0x7f79b40a47d0 2024-04-26T04:18:48.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.038+0000 7f79bb0a4640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:48.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.038+0000 7f79bb0a4640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7f79b40a35f0 con 0x7f79b40a2cd0 2024-04-26T04:18:48.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.038+0000 7f79bb0a4640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:48.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.038+0000 7f79bb0a4640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f79b40a5380 con 0x7f79b40a4c00 2024-04-26T04:18:48.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.038+0000 7f79bb0a4640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:48.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.038+0000 7f79bb0a4640 10 monclient(hunting): authenticate will time out at 959.751678s 2024-04-26T04:18:48.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.038+0000 7f79b8e40640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f79b40a2cd0 0x7f79b40a30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:48.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.038+0000 7f79b9641640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f79b40a4c00 0x7f79b41373f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:48.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.038+0000 7f79b8e40640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f79b40a2cd0 0x7f79b40a30b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:46492/0 (socket says 172.21.15.63:46492) 2024-04-26T04:18:48.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.038+0000 7f79b8e40640 1 -- 172.21.15.63:0/4253396935 learned_addr learned my addr 172.21.15.63:0/4253396935 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:18:48.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.038+0000 7f79b8e40640 10 monclient(hunting): get_auth_request con 0x7f79b40a2cd0 auth_method 0 2024-04-26T04:18:48.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.038+0000 7f79b8e40640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:48.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.038+0000 7f79b8e40640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:48.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.038+0000 7f79b8e40640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:48.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.038+0000 7f79b9641640 10 monclient(hunting): get_auth_request con 0x7f79b40a4c00 auth_method 0 2024-04-26T04:18:48.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.038+0000 7f79b9641640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:48.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.038+0000 7f79b9641640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:48.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.038+0000 7f79b9641640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:48.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.042+0000 7f79b8e40640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:48.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.042+0000 7f79b8e40640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:48.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.042+0000 7f79b8e40640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:48.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.042+0000 7f79b9641640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:48.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.042+0000 7f79b9641640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:48.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.042+0000 7f79b9641640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:48.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.042+0000 7f79b9641640 10 monclient(hunting): handle_auth_done global_id 4232 payload 274 2024-04-26T04:18:48.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.042+0000 7f79b9641640 1 -- 172.21.15.63:0/4253396935 >> v2:172.21.15.63:3300/0 conn(0x7f79b40a2cd0 msgr2=0x7f79b40a30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:48.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.042+0000 7f79b9641640 1 --2- 172.21.15.63:0/4253396935 >> v2:172.21.15.63:3300/0 conn(0x7f79b40a2cd0 0x7f79b40a30b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:48.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.042+0000 7f79b9641640 1 -- 172.21.15.63:0/4253396935 >> v1:172.21.15.186:6789/0 conn(0x7f79b40a47d0 legacy=0x7f79b4136ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:48.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.042+0000 7f79b9641640 10 monclient: _finish_hunting 0 2024-04-26T04:18:48.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.042+0000 7f79b9641640 1 monclient: found mon.noname-b 2024-04-26T04:18:48.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.042+0000 7f79b9641640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:48.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.042+0000 7f79b9641640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.63:3301/0 2024-04-26T04:18:48.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.042+0000 7f79b9641640 1 -- 172.21.15.63:0/4253396935 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79b40a1290 con 0x7f79b40a4c00 2024-04-26T04:18:48.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f79b9641640 1 -- 172.21.15.63:0/4253396935 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f79b40a4c00 msgr2=0x7f79b41373f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:48.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f79b9641640 1 --2- 172.21.15.63:0/4253396935 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f79b40a4c00 0x7f79b41373f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f79ac009d90 tx=0x7f79ac01c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=788d135989b02845 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:48.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f476d21f640 10 monclient: get_monmap_and_config 2024-04-26T04:18:48.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f476d21f640 10 monclient: build_initial_monmap 2024-04-26T04:18:48.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f476d21f640 10 monclient: monmap: 2024-04-26T04:18:48.060 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:48.060 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:48.060 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:48.050494+0000 2024-04-26T04:18:48.060 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:48.050494+0000 2024-04-26T04:18:48.060 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:48.060 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:48.060 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:18:48.060 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:18:48.060 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:18:48.060 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:18:48.061 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:18:48.061 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:48.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f79b37fe640 1 -- 172.21.15.63:0/4253396935 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79ac0a6710 con 0x7f79b40a4c00 2024-04-26T04:18:48.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f79b37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:48.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f79b37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T04:18:48.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f79b37fe640 10 monclient: dump: 2024-04-26T04:18:48.061 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:48.061 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:48.061 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:48.062 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:48.062 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:48.062 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:48.062 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:48.062 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:48.062 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:48.062 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:48.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f79b37fe640 10 monclient: _finish_auth 0 2024-04-26T04:18:48.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f79b37fe640 10 monclient: _check_auth_tickets 2024-04-26T04:18:48.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f79b37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:48.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f79b37fe640 1 -- 172.21.15.63:0/4253396935 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79ac0a68b0 con 0x7f79b40a4c00 2024-04-26T04:18:48.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f79b37fe640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:48.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f79bb0a4640 5 monclient: authenticate success, global_id 4232 2024-04-26T04:18:48.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f79b37fe640 1 -- 172.21.15.63:0/4253396935 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79ac0a6c80 con 0x7f79b40a4c00 2024-04-26T04:18:48.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f79bb0a4640 10 monclient: get_monmap_and_config success 2024-04-26T04:18:48.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f79b37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:48.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f79bb0a4640 4 set_mon_vals no callback set 2024-04-26T04:18:48.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f79b37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:18:48.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f79b37fe640 10 monclient: dump: 2024-04-26T04:18:48.063 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:48.063 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:48.063 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:48.063 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:48.063 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:48.063 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:48.063 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:48.063 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:48.064 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:48.064 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:48.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f476d21f640 1 Processor -- start 2024-04-26T04:18:48.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f79bb0a4640 10 monclient: shutdown 2024-04-26T04:18:48.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f79bb0a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:48.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f79bb0a4640 1 -- 172.21.15.63:0/4253396935 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f79b40a4c00 msgr2=0x7f79b41373f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:48.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f79bb0a4640 1 --2- 172.21.15.63:0/4253396935 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f79b40a4c00 0x7f79b41373f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f79ac009d90 tx=0x7f79ac01c880 comp rx=0 tx=0).stop 2024-04-26T04:18:48.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f476d21f640 1 -- start start 2024-04-26T04:18:48.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f476d21f640 10 monclient: init 2024-04-26T04:18:48.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f79bb0a4640 1 -- 172.21.15.63:0/4253396935 shutdown_connections 2024-04-26T04:18:48.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f79bb0a4640 1 --2- 172.21.15.63:0/4253396935 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f79b40a4c00 0x7f79b41373f0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:48.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f79bb0a4640 1 --2- 172.21.15.63:0/4253396935 >> v2:172.21.15.63:3300/0 conn(0x7f79b40a2cd0 0x7f79b40a30b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:48.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f79bb0a4640 1 -- 172.21.15.63:0/4253396935 >> 172.21.15.63:0/4253396935 conn(0x7f79b40a5f40 msgr2=0x7f79b40a6320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:48.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f79bb0a4640 1 -- 172.21.15.63:0/4253396935 shutdown_connections 2024-04-26T04:18:48.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f79bb0a4640 1 -- 172.21.15.63:0/4253396935 wait complete. 2024-04-26T04:18:48.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f476d21f640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:48.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f476d21f640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-04-26T04:18:48.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f476d21f640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f4760027800 0x7f47600ba150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:48.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f476d21f640 10 monclient(hunting): picked mon.noname-c con 0x7f4760027800 addr v2:172.21.15.186:3300/0 2024-04-26T04:18:48.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f476d21f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4760022300 addr v1:172.21.15.63:6789/0 2024-04-26T04:18:48.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f476d21f640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f47600bee10 0x7f47600c1200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:48.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f476d21f640 10 monclient(hunting): picked mon.noname-a con 0x7f47600bee10 addr v2:172.21.15.63:3300/0 2024-04-26T04:18:48.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f476d21f640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47600ba810 con 0x7f4760022300 2024-04-26T04:18:48.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f476d21f640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:48.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f476d21f640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7f4760028600 con 0x7f47600bee10 2024-04-26T04:18:48.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f476d21f640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:48.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f476d21f640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f4760027be0 con 0x7f4760027800 2024-04-26T04:18:48.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f476d21f640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:48.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f476d21f640 10 monclient(hunting): authenticate will time out at 959.758203s 2024-04-26T04:18:48.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f47677fe640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f4760022300 0x7f4760022710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:44188/0 (socket says 172.21.15.63:44188) 2024-04-26T04:18:48.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f47677fe640 1 -- 172.21.15.63:0/2751933442 learned_addr learned my addr 172.21.15.63:0/2751933442 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-04-26T04:18:48.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f476ca1e640 1 --2- 172.21.15.63:0/2751933442 >> v2:172.21.15.63:3300/0 conn(0x7f47600bee10 0x7f47600c1200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:48.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f79bb0a4640 10 monclient: build_initial_monmap 2024-04-26T04:18:48.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f79bb0a4640 10 monclient: monmap: 2024-04-26T04:18:48.069 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:48.069 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:18:48.069 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:18:48.070 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:18:48.070 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:48.070 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:48.070 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:18:48.070 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:18:48.070 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:18:48.070 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:48.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f476ca1e640 10 monclient(hunting): get_auth_request con 0x7f47600bee10 auth_method 0 2024-04-26T04:18:48.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f476ca1e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:48.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f476ca1e640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:48.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f476ca1e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:48.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f4767fff640 1 --2- 172.21.15.63:0/2751933442 >> v2:172.21.15.186:3300/0 conn(0x7f4760027800 0x7f47600ba150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:48.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f4767fff640 10 monclient(hunting): get_auth_request con 0x7f4760027800 auth_method 0 2024-04-26T04:18:48.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f4767fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:48.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f4767fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:48.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f4767fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:48.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f476ca1e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:48.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f476ca1e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:48.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f476ca1e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:48.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f4767fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:48.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f4767fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:48.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f4767fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:48.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f476ca1e640 10 monclient(hunting): handle_auth_done global_id 4248 payload 274 2024-04-26T04:18:48.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f476ca1e640 1 -- 172.21.15.63:0/2751933442 >> v2:172.21.15.186:3300/0 conn(0x7f4760027800 msgr2=0x7f47600ba150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:48.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f476ca1e640 1 --2- 172.21.15.63:0/2751933442 >> v2:172.21.15.186:3300/0 conn(0x7f4760027800 0x7f47600ba150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:48.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f476ca1e640 1 -- 172.21.15.63:0/2751933442 >> v1:172.21.15.63:6789/0 conn(0x7f4760022300 legacy=0x7f4760022710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:48.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f476ca1e640 10 monclient: _finish_hunting 0 2024-04-26T04:18:48.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f476ca1e640 1 monclient: found mon.noname-a 2024-04-26T04:18:48.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f476ca1e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:48.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f476ca1e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:18:48.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f476ca1e640 1 -- 172.21.15.63:0/2751933442 --> v2:172.21.15.63:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4760025980 con 0x7f47600bee10 2024-04-26T04:18:48.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7fcdf763c640 10 monclient: get_monmap_and_config 2024-04-26T04:18:48.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7fcdf763c640 10 monclient: build_initial_monmap 2024-04-26T04:18:48.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7fcdf763c640 10 monclient: monmap: 2024-04-26T04:18:48.074 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:48.074 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:48.074 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:48.053059+0000 2024-04-26T04:18:48.074 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:48.053059+0000 2024-04-26T04:18:48.074 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:48.074 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:48.074 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:18:48.074 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:18:48.074 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:18:48.074 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:18:48.075 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:18:48.075 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:48.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7fcdf763c640 1 Processor -- start 2024-04-26T04:18:48.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7fcdf763c640 1 -- start start 2024-04-26T04:18:48.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7fcdf763c640 10 monclient: init 2024-04-26T04:18:48.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7fcdf763c640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:48.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7fcdf763c640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-04-26T04:18:48.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7fcdf763c640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fcdf00a1d60 0x7fcdf00a2140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:48.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7fcdf763c640 10 monclient(hunting): picked mon.noname-c con 0x7fcdf00a1d60 addr v2:172.21.15.186:3300/0 2024-04-26T04:18:48.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7fcdf763c640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7fcdf006fe70 0x7fcdf00a2680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:48.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7fcdf763c640 10 monclient(hunting): picked mon.noname-a con 0x7fcdf006fe70 addr v2:172.21.15.63:3300/0 2024-04-26T04:18:48.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7fcdf763c640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcdf013f700 0x7fcdf0141af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:48.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7fcdf763c640 10 monclient(hunting): picked mon.noname-b con 0x7fcdf013f700 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:48.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7fcdf763c640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:48.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7fcdf763c640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7fcdf00a33c0 con 0x7fcdf006fe70 2024-04-26T04:18:48.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7fcdf763c640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:48.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7fcdf763c640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fcdf00a2bc0 con 0x7fcdf00a1d60 2024-04-26T04:18:48.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7fcdf763c640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:48.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7fcdf763c640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7fcdf006fa50 con 0x7fcdf013f700 2024-04-26T04:18:48.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7fcdf763c640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:48.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7fcdf763c640 10 monclient(hunting): authenticate will time out at 959.761020s 2024-04-26T04:18:48.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f79bb0a4640 1 Processor -- start 2024-04-26T04:18:48.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f79bb0a4640 1 -- start start 2024-04-26T04:18:48.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f79bb0a4640 10 monclient: init 2024-04-26T04:18:48.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f79bb0a4640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:48.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f79bb0a4640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T04:18:48.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f79bb0a4640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79b40a2cd0 0x7f79b41ceed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:48.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f79bb0a4640 10 monclient(hunting): picked mon.noname-a con 0x7f79b40a2cd0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:18:48.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f79bb0a4640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f79b40a47d0 0x7f79b41cf600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:48.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f79bb0a4640 10 monclient(hunting): picked mon.noname-c con 0x7f79b40a47d0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:48.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f79bb0a4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f79b40a4c00 0x7f79b413ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:48.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f79bb0a4640 10 monclient(hunting): picked mon.noname-b con 0x7f79b40a4c00 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:18:48.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f79bb0a4640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:48.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f79bb0a4640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f79b4141a50 con 0x7f79b40a2cd0 2024-04-26T04:18:48.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f79bb0a4640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:48.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f79bb0a4640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f79b4137b40 con 0x7f79b40a4c00 2024-04-26T04:18:48.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f79bb0a4640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:48.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f79bb0a4640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f79b4141bd0 con 0x7f79b40a47d0 2024-04-26T04:18:48.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f79bb0a4640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:48.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f79bb0a4640 10 monclient(hunting): authenticate will time out at 959.759518s 2024-04-26T04:18:48.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f79b3fff640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f79b40a47d0 0x7f79b41cf600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:48.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f79b3fff640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f79b40a47d0 0x7f79b41cf600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:57876/0 (socket says 172.21.15.63:57876) 2024-04-26T04:18:48.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f79b3fff640 1 -- 172.21.15.63:0/2635447425 learned_addr learned my addr 172.21.15.63:0/2635447425 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:18:48.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f79b3fff640 10 monclient(hunting): get_auth_request con 0x7f79b40a47d0 auth_method 0 2024-04-26T04:18:48.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f79b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:48.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f79b3fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:48.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f79b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:48.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f79b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:48.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f79b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:48.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.046+0000 7f79b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:48.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f79b3fff640 10 monclient(hunting): handle_auth_done global_id 4235 payload 824 2024-04-26T04:18:48.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f79b3fff640 1 -- 172.21.15.63:0/2635447425 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f79b40a4c00 msgr2=0x7f79b413ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:48.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f79b3fff640 1 --2- 172.21.15.63:0/2635447425 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f79b40a4c00 0x7f79b413ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:48.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f79b3fff640 1 -- 172.21.15.63:0/2635447425 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79b40a2cd0 msgr2=0x7f79b41ceed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:48.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f79b3fff640 1 --2- 172.21.15.63:0/2635447425 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79b40a2cd0 0x7f79b41ceed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:48.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f79b3fff640 10 monclient: _finish_hunting 0 2024-04-26T04:18:48.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f79b3fff640 1 monclient: found mon.noname-c 2024-04-26T04:18:48.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f79b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:48.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f79b3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.63:3301/0 2024-04-26T04:18:48.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f79b3fff640 1 -- 172.21.15.63:0/2635447425 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79b4072a90 con 0x7f79b40a47d0 2024-04-26T04:18:48.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7fcdf4bd7640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7fcdf006fe70 0x7fcdf00a2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:48.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7fcdf53d8640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fcdf00a1d60 0x7fcdf00a2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:48.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7fcdf4bd7640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7fcdf006fe70 0x7fcdf00a2680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:46526/0 (socket says 172.21.15.63:46526) 2024-04-26T04:18:48.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7fcdf4bd7640 1 -- 172.21.15.63:0/649345254 learned_addr learned my addr 172.21.15.63:0/649345254 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:18:48.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7fcdf4bd7640 10 monclient(hunting): get_auth_request con 0x7fcdf006fe70 auth_method 0 2024-04-26T04:18:48.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7fcdf4bd7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:48.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7fcdf4bd7640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:48.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7fcdf4bd7640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:48.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7fcdf53d8640 10 monclient(hunting): get_auth_request con 0x7fcdf00a1d60 auth_method 0 2024-04-26T04:18:48.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7fcdf53d8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:48.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7fcdf53d8640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:48.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7fcdf53d8640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:48.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7fcdf5bd9640 1 --2- 172.21.15.63:0/649345254 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcdf013f700 0x7fcdf0141af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:48.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7fcdf4bd7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:48.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7fcdf4bd7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:48.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7fcdf4bd7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:48.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f8d38d32640 10 monclient: get_monmap_and_config 2024-04-26T04:18:48.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f8d38d32640 10 monclient: build_initial_monmap 2024-04-26T04:18:48.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f8d38d32640 10 monclient: monmap: 2024-04-26T04:18:48.082 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:48.082 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:48.082 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:48.055017+0000 2024-04-26T04:18:48.082 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:48.055017+0000 2024-04-26T04:18:48.083 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:48.083 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:48.083 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:18:48.083 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:18:48.083 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:18:48.083 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:18:48.083 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:18:48.083 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:48.084 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7fcdf5bd9640 10 monclient(hunting): get_auth_request con 0x7fcdf013f700 auth_method 0 2024-04-26T04:18:48.084 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7fcdf5bd9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:48.084 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7fcdf5bd9640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:48.084 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7fcdf5bd9640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:48.084 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7fcdf53d8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:48.084 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7fcdf53d8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:48.084 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7fcdf53d8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:48.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7fcdf4bd7640 10 monclient(hunting): handle_auth_done global_id 4251 payload 274 2024-04-26T04:18:48.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7fcdf4bd7640 1 -- 172.21.15.63:0/649345254 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcdf013f700 msgr2=0x7fcdf0141af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:48.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7fcdf4bd7640 1 --2- 172.21.15.63:0/649345254 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcdf013f700 0x7fcdf0141af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:48.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7fcdf4bd7640 1 -- 172.21.15.63:0/649345254 >> v2:172.21.15.186:3300/0 conn(0x7fcdf00a1d60 msgr2=0x7fcdf00a2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:48.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7fcdf4bd7640 1 --2- 172.21.15.63:0/649345254 >> v2:172.21.15.186:3300/0 conn(0x7fcdf00a1d60 0x7fcdf00a2140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:48.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7fcdf4bd7640 10 monclient: _finish_hunting 0 2024-04-26T04:18:48.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7fcdf4bd7640 1 monclient: found mon.noname-a 2024-04-26T04:18:48.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7fcdf4bd7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:48.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7fcdf4bd7640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:18:48.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7fcdf4bd7640 1 -- 172.21.15.63:0/649345254 --> v2:172.21.15.63:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdf00a3840 con 0x7fcdf006fe70 2024-04-26T04:18:48.086 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f8d38d32640 1 Processor -- start 2024-04-26T04:18:48.086 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f8d38d32640 1 -- start start 2024-04-26T04:18:48.086 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f8d38d32640 10 monclient: init 2024-04-26T04:18:48.086 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f8d38d32640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:48.086 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f8d38d32640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-04-26T04:18:48.086 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f8d38d32640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8d2c0255d0 addr v1:172.21.15.63:6789/0 2024-04-26T04:18:48.087 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f8d38d32640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f8d2c023d00 0x7f8d2c0c06f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:48.087 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f8d38d32640 10 monclient(hunting): picked mon.noname-c con 0x7f8d2c023d00 addr v2:172.21.15.186:3300/0 2024-04-26T04:18:48.087 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f8d38d32640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f8d2c0c5460 0x7f8d2c0c7850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:48.087 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f8d38d32640 10 monclient(hunting): picked mon.noname-a con 0x7f8d2c0c5460 addr v2:172.21.15.63:3300/0 2024-04-26T04:18:48.087 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f79b3fff640 1 -- 172.21.15.63:0/2635447425 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f79b40a47d0 msgr2=0x7f79b41cf600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:48.087 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f79b3fff640 1 --2- 172.21.15.63:0/2635447425 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f79b40a47d0 0x7f79b41cf600 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f79a000c490 tx=0x7f79a0001e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:48.088 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f8d38d32640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d2c0c0d10 con 0x7f8d2c0255d0 2024-04-26T04:18:48.088 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f8d38d32640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:48.088 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f8d38d32640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7f8d2c0259b0 con 0x7f8d2c0c5460 2024-04-26T04:18:48.088 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f8d38d32640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:48.088 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f8d38d32640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f8d2c020550 con 0x7f8d2c023d00 2024-04-26T04:18:48.089 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f79b17fa640 1 -- 172.21.15.63:0/2635447425 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79a0017070 con 0x7f79b40a47d0 2024-04-26T04:18:48.089 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f79b17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:48.089 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f79b17fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:18:48.089 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f79b17fa640 10 monclient: dump: 2024-04-26T04:18:48.090 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:48.090 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:48.090 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:48.090 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:48.090 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:48.090 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:48.090 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:48.090 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:48.091 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:48.091 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:48.091 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f79b17fa640 10 monclient: _finish_auth 0 2024-04-26T04:18:48.091 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f79b17fa640 10 monclient: _check_auth_tickets 2024-04-26T04:18:48.091 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f79b17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:48.091 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f79b17fa640 1 -- 172.21.15.63:0/2635447425 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79a0003820 con 0x7f79b40a47d0 2024-04-26T04:18:48.091 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f79b17fa640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:48.091 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f79b17fa640 1 -- 172.21.15.63:0/2635447425 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79a0003c10 con 0x7f79b40a47d0 2024-04-26T04:18:48.091 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f79b17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:48.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f79b17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:18:48.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f79b17fa640 10 monclient: dump: 2024-04-26T04:18:48.092 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:48.092 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:48.092 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:48.092 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:48.092 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:48.092 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:48.092 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:48.092 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:48.092 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:48.093 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:48.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f79bb0a4640 5 monclient: authenticate success, global_id 4235 2024-04-26T04:18:48.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f79bb0a4640 10 monclient: _renew_subs 2024-04-26T04:18:48.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f79bb0a4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:48.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f79bb0a4640 1 -- 172.21.15.63:0/2635447425 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79b413d720 con 0x7f79b40a47d0 2024-04-26T04:18:48.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f79bb0a4640 10 monclient: _renew_subs 2024-04-26T04:18:48.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f79bb0a4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:48.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f79bb0a4640 1 -- 172.21.15.63:0/2635447425 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f79b4120580 con 0x7f79b40a47d0 2024-04-26T04:18:48.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f79b2ffd640 4 set_mon_vals no callback set 2024-04-26T04:18:48.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f8d38d32640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:48.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f8d38d32640 10 monclient(hunting): authenticate will time out at 959.762481s 2024-04-26T04:18:48.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f8d337fe640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f8d2c0255d0 0x7f8d2c0235f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:44194/0 (socket says 172.21.15.63:44194) 2024-04-26T04:18:48.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f8d337fe640 1 -- 172.21.15.63:0/2367056632 learned_addr learned my addr 172.21.15.63:0/2367056632 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-04-26T04:18:48.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f8d32ffd640 1 --2- 172.21.15.63:0/2367056632 >> v2:172.21.15.186:3300/0 conn(0x7f8d2c023d00 0x7f8d2c0c06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:48.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f8d32ffd640 10 monclient(hunting): get_auth_request con 0x7f8d2c023d00 auth_method 0 2024-04-26T04:18:48.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f8d32ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:48.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f8d32ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:48.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f8d32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:48.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f8d32ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:48.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f8d32ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:48.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f8d32ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:48.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f476ca1e640 1 -- 172.21.15.63:0/2751933442 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f47600bee10 msgr2=0x7f47600c1200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:48.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f476ca1e640 1 --2- 172.21.15.63:0/2751933442 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f47600bee10 0x7f47600c1200 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4768064cd0 tx=0x7f47680980b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c84aa5b51862e342 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:48.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f4766ffd640 1 -- 172.21.15.63:0/2751933442 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f476812d070 con 0x7f47600bee10 2024-04-26T04:18:48.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f4766ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:48.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f4766ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:18:48.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f4766ffd640 10 monclient: dump: 2024-04-26T04:18:48.095 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:48.095 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:48.095 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:48.096 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:48.096 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:48.096 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:48.096 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:48.096 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:48.096 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:48.096 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:48.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f4766ffd640 10 monclient: _finish_auth 0 2024-04-26T04:18:48.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f4766ffd640 10 monclient: _check_auth_tickets 2024-04-26T04:18:48.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f4766ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:48.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f4766ffd640 1 -- 172.21.15.63:0/2751933442 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4768128440 con 0x7f47600bee10 2024-04-26T04:18:48.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f4766ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:48.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f4766ffd640 1 -- 172.21.15.63:0/2751933442 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47681287f0 con 0x7f47600bee10 2024-04-26T04:18:48.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f4766ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:48.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f4766ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:18:48.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f4766ffd640 10 monclient: dump: 2024-04-26T04:18:48.097 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:48.097 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:48.097 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:48.097 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:48.097 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:48.097 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:48.097 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:48.097 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:48.097 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:48.097 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:48.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f476d21f640 5 monclient: authenticate success, global_id 4248 2024-04-26T04:18:48.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f476d21f640 10 monclient: get_monmap_and_config success 2024-04-26T04:18:48.098 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f476d21f640 4 set_mon_vals no callback set 2024-04-26T04:18:48.098 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f79b17fa640 1 -- 172.21.15.63:0/2635447425 <== mon.2 v2:172.21.15.63:3301/0 4 ==== mgrmap(e 4) ==== 43997+0+0 (secure 0 0 0) 0x7f79a001a360 con 0x7f79b40a47d0 2024-04-26T04:18:48.098 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f8d33fff640 1 --2- 172.21.15.63:0/2367056632 >> v2:172.21.15.63:3300/0 conn(0x7f8d2c0c5460 0x7f8d2c0c7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:48.098 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f8d33fff640 10 monclient(hunting): get_auth_request con 0x7f8d2c0c5460 auth_method 0 2024-04-26T04:18:48.098 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f8d33fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:48.098 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f8d33fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:48.098 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f8d33fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:48.098 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f8d32ffd640 10 monclient(hunting): handle_auth_done global_id 4258 payload 274 2024-04-26T04:18:48.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f8d32ffd640 1 -- 172.21.15.63:0/2367056632 >> v2:172.21.15.63:3300/0 conn(0x7f8d2c0c5460 msgr2=0x7f8d2c0c7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:48.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f8d32ffd640 1 --2- 172.21.15.63:0/2367056632 >> v2:172.21.15.63:3300/0 conn(0x7f8d2c0c5460 0x7f8d2c0c7850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:48.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f8d32ffd640 1 -- 172.21.15.63:0/2367056632 >> v1:172.21.15.63:6789/0 conn(0x7f8d2c0255d0 legacy=0x7f8d2c0235f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:48.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f8d32ffd640 10 monclient: _finish_hunting 0 2024-04-26T04:18:48.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f8d32ffd640 1 monclient: found mon.noname-c 2024-04-26T04:18:48.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f8d32ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:48.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f8d32ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.186:3300/0 2024-04-26T04:18:48.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.050+0000 7f8d32ffd640 1 -- 172.21.15.63:0/2367056632 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d2c026390 con 0x7f8d2c023d00 2024-04-26T04:18:48.100 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7f79bb0a4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:48.100 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7f79bb0a4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:48.100 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7f79bb0a4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:48.100 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7f79b17fa640 1 --2- 172.21.15.63:0/2635447425 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f7998035e30 0x7f79980382f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:48.100 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7f79bb0a4640 1 -- 172.21.15.63:0/2635447425 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7994005090 con 0x7f79b40a47d0 2024-04-26T04:18:48.100 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7f79b17fa640 1 -- 172.21.15.63:0/2635447425 <== mon.2 v2:172.21.15.63:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7f79a0047ff0 con 0x7f79b40a47d0 2024-04-26T04:18:48.100 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7f476d21f640 10 monclient: shutdown 2024-04-26T04:18:48.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7f476d21f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:48.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7f476d21f640 1 -- 172.21.15.63:0/2751933442 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f47600bee10 msgr2=0x7f47600c1200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:48.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7f476d21f640 1 --2- 172.21.15.63:0/2751933442 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f47600bee10 0x7f47600c1200 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4768064cd0 tx=0x7f47680980b0 comp rx=0 tx=0).stop 2024-04-26T04:18:48.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7f476d21f640 1 -- 172.21.15.63:0/2751933442 shutdown_connections 2024-04-26T04:18:48.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7f476d21f640 1 --2- 172.21.15.63:0/2751933442 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f47600bee10 0x7f47600c1200 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:48.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7f476d21f640 1 --2- 172.21.15.63:0/2751933442 >> v2:172.21.15.186:3300/0 conn(0x7f4760027800 0x7f47600ba150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:48.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7f476d21f640 1 -- 172.21.15.63:0/2751933442 >> 172.21.15.63:0/2751933442 conn(0x7f47600068a0 msgr2=0x7f4760006cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:48.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7f79b8e40640 1 --2- 172.21.15.63:0/2635447425 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f7998035e30 0x7f79980382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:48.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7f79b8e40640 10 monclient: get_auth_request con 0x7f7998035e30 auth_method 0 2024-04-26T04:18:48.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7f476d21f640 1 -- 172.21.15.63:0/2751933442 shutdown_connections 2024-04-26T04:18:48.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7f476d21f640 1 -- 172.21.15.63:0/2751933442 wait complete. 2024-04-26T04:18:48.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7f476d21f640 10 monclient: build_initial_monmap 2024-04-26T04:18:48.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7f476d21f640 10 monclient: monmap: 2024-04-26T04:18:48.104 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:48.104 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:18:48.104 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:18:48.104 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:18:48.104 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:48.104 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:48.104 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:18:48.104 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:18:48.104 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:18:48.104 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:48.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7f476d21f640 1 Processor -- start 2024-04-26T04:18:48.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7f476d21f640 1 -- start start 2024-04-26T04:18:48.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7f476d21f640 10 monclient: init 2024-04-26T04:18:48.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7f476d21f640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:48.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7f476d21f640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T04:18:48.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7f476d21f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4760022300 0x7f47600be250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:48.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7f476d21f640 10 monclient(hunting): picked mon.noname-a con 0x7f4760022300 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:18:48.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7f476d21f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4760027800 0x7f4760007c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:48.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7f476d21f640 10 monclient(hunting): picked mon.noname-b con 0x7f4760027800 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:18:48.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7f476d21f640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f47600bee10 0x7f4760008180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:48.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7f476d21f640 10 monclient(hunting): picked mon.noname-c con 0x7f47600bee10 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:48.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7f476d21f640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:48.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7f476d21f640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f47600c4350 con 0x7f4760022300 2024-04-26T04:18:48.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7f476d21f640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:48.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7f476d21f640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f47600c41d0 con 0x7f4760027800 2024-04-26T04:18:48.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7f476d21f640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:48.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7f476d21f640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f47600c44d0 con 0x7f47600bee10 2024-04-26T04:18:48.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7f476d21f640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:48.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7f476d21f640 10 monclient(hunting): authenticate will time out at 959.766512s 2024-04-26T04:18:48.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7f476ca1e640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f47600bee10 0x7f4760008180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:48.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7f476ca1e640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f47600bee10 0x7f4760008180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:57890/0 (socket says 172.21.15.63:57890) 2024-04-26T04:18:48.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7f476ca1e640 1 -- 172.21.15.63:0/955616177 learned_addr learned my addr 172.21.15.63:0/955616177 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:18:48.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7f476ca1e640 10 monclient(hunting): get_auth_request con 0x7f47600bee10 auth_method 0 2024-04-26T04:18:48.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7f476ca1e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:48.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7f476ca1e640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:48.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7f476ca1e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:48.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7f476ca1e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:48.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7f476ca1e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:48.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7f476ca1e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:48.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7f476ca1e640 10 monclient(hunting): handle_auth_done global_id 4241 payload 824 2024-04-26T04:18:48.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7f476ca1e640 1 -- 172.21.15.63:0/955616177 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4760027800 msgr2=0x7f4760007c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:48.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7f476ca1e640 1 --2- 172.21.15.63:0/955616177 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4760027800 0x7f4760007c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:48.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7f476ca1e640 1 -- 172.21.15.63:0/955616177 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4760022300 msgr2=0x7f47600be250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:48.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7f476ca1e640 1 --2- 172.21.15.63:0/955616177 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4760022300 0x7f47600be250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:48.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7f476ca1e640 10 monclient: _finish_hunting 0 2024-04-26T04:18:48.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7f476ca1e640 1 monclient: found mon.noname-c 2024-04-26T04:18:48.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7f476ca1e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:48.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7f476ca1e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.63:3301/0 2024-04-26T04:18:48.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7f476ca1e640 1 -- 172.21.15.63:0/955616177 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f476001f0d0 con 0x7f47600bee10 2024-04-26T04:18:48.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7fcdf4bd7640 1 -- 172.21.15.63:0/649345254 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcdf006fe70 msgr2=0x7fcdf00a2680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:48.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7fcdf4bd7640 1 --2- 172.21.15.63:0/649345254 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcdf006fe70 0x7fcdf00a2680 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fcdf00a1a20 tx=0x7fcde001c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8a51fc49f006ca9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:48.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7fcde7fff640 1 -- 172.21.15.63:0/649345254 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcde00b1070 con 0x7fcdf006fe70 2024-04-26T04:18:48.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7fcde7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:48.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7fcde7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:18:48.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7fcde7fff640 10 monclient: dump: 2024-04-26T04:18:48.108 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:48.109 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:48.109 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:48.109 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:48.109 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:48.109 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:48.109 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:48.109 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:48.109 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:48.109 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:48.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7fcde7fff640 10 monclient: _finish_auth 0 2024-04-26T04:18:48.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7fcde7fff640 10 monclient: _check_auth_tickets 2024-04-26T04:18:48.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7fcde7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:48.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7fcde7fff640 1 -- 172.21.15.63:0/649345254 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcde00ac600 con 0x7fcdf006fe70 2024-04-26T04:18:48.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7fcdf763c640 5 monclient: authenticate success, global_id 4251 2024-04-26T04:18:48.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7fcdf763c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T04:18:48.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7fcde7fff640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:48.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7fcdf763c640 10 monclient: get_monmap_and_config success 2024-04-26T04:18:48.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7fcdf763c640 4 set_mon_vals no callback set 2024-04-26T04:18:48.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7fcde7fff640 1 -- 172.21.15.63:0/649345254 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcde00ac9d0 con 0x7fcdf006fe70 2024-04-26T04:18:48.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7fcde7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:48.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7fcde7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:18:48.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7fcde7fff640 10 monclient: dump: 2024-04-26T04:18:48.110 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:48.110 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:48.110 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:48.110 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:48.110 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:48.111 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:48.111 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:48.111 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:48.111 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:48.111 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:48.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.054+0000 7f79b8e40640 1 --2- 172.21.15.63:0/2635447425 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f7998035e30 0x7f79980382f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f79a8006aa0 tx=0x7f79a8006a30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:48.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7fcdf763c640 10 monclient: shutdown 2024-04-26T04:18:48.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7fcdf763c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:48.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7fcdf763c640 1 -- 172.21.15.63:0/649345254 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcdf006fe70 msgr2=0x7fcdf00a2680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:48.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7fcdf763c640 1 --2- 172.21.15.63:0/649345254 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcdf006fe70 0x7fcdf00a2680 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fcdf00a1a20 tx=0x7fcde001c3e0 comp rx=0 tx=0).stop 2024-04-26T04:18:48.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7fcdf763c640 1 -- 172.21.15.63:0/649345254 shutdown_connections 2024-04-26T04:18:48.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7fcdf763c640 1 --2- 172.21.15.63:0/649345254 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcdf013f700 0x7fcdf0141af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:48.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7fcdf763c640 1 --2- 172.21.15.63:0/649345254 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcdf006fe70 0x7fcdf00a2680 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:48.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7fcdf763c640 1 --2- 172.21.15.63:0/649345254 >> v2:172.21.15.186:3300/0 conn(0x7fcdf00a1d60 0x7fcdf00a2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:48.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7fcdf763c640 1 -- 172.21.15.63:0/649345254 >> 172.21.15.63:0/649345254 conn(0x7fcdf00a7360 msgr2=0x7fcdf00a7770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:48.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7fcdf763c640 1 -- 172.21.15.63:0/649345254 shutdown_connections 2024-04-26T04:18:48.114 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7f8d32ffd640 1 -- 172.21.15.63:0/2367056632 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d2c023d00 msgr2=0x7f8d2c0c06f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:48.114 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7f8d32ffd640 1 --2- 172.21.15.63:0/2367056632 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d2c023d00 0x7f8d2c0c06f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c027a80 tx=0x7f8d2001c320 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b249700d9d941078 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:48.114 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7f8d327fc640 1 -- 172.21.15.63:0/2367056632 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d200b0070 con 0x7f8d2c023d00 2024-04-26T04:18:48.114 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7f8d327fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:48.114 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7f8d327fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:18:48.114 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7f8d327fc640 10 monclient: dump: 2024-04-26T04:18:48.114 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:48.114 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:48.114 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:48.114 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:48.114 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:48.114 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:48.115 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:48.115 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:48.115 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:48.115 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:48.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7f8d327fc640 10 monclient: _finish_auth 0 2024-04-26T04:18:48.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7f8d327fc640 10 monclient: _check_auth_tickets 2024-04-26T04:18:48.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7f8d327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:48.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7f8d327fc640 1 -- 172.21.15.63:0/2367056632 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d200ab600 con 0x7f8d2c023d00 2024-04-26T04:18:48.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7f8d327fc640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:48.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7f8d327fc640 1 -- 172.21.15.63:0/2367056632 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d200ab9d0 con 0x7f8d2c023d00 2024-04-26T04:18:48.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7f8d327fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:48.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7f8d327fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T04:18:48.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7f8d327fc640 10 monclient: dump: 2024-04-26T04:18:48.116 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:48.116 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:48.116 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:48.116 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:48.116 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:48.116 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:48.116 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:48.116 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:48.116 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:48.116 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:48.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7f8d38d32640 5 monclient: authenticate success, global_id 4258 2024-04-26T04:18:48.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7f8d38d32640 10 monclient: get_monmap_and_config success 2024-04-26T04:18:48.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7f8d38d32640 4 set_mon_vals no callback set 2024-04-26T04:18:48.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7f476ca1e640 1 -- 172.21.15.63:0/955616177 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f47600bee10 msgr2=0x7f4760008180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:48.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7f476ca1e640 1 --2- 172.21.15.63:0/955616177 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f47600bee10 0x7f4760008180 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4768074710 tx=0x7f4768074130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:48.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7f8d38d32640 10 monclient: shutdown 2024-04-26T04:18:48.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7f8d38d32640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:48.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7f8d38d32640 1 -- 172.21.15.63:0/2367056632 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d2c023d00 msgr2=0x7f8d2c0c06f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:48.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7f8d38d32640 1 --2- 172.21.15.63:0/2367056632 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d2c023d00 0x7f8d2c0c06f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c027a80 tx=0x7f8d2001c320 comp rx=0 tx=0).stop 2024-04-26T04:18:48.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7f8d38d32640 1 -- 172.21.15.63:0/2367056632 shutdown_connections 2024-04-26T04:18:48.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7f8d38d32640 1 --2- 172.21.15.63:0/2367056632 >> v2:172.21.15.63:3300/0 conn(0x7f8d2c0c5460 0x7f8d2c0c7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:48.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7f8d38d32640 1 --2- 172.21.15.63:0/2367056632 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d2c023d00 0x7f8d2c0c06f0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:48.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7f8d38d32640 1 -- 172.21.15.63:0/2367056632 >> 172.21.15.63:0/2367056632 conn(0x7f8d2c0068d0 msgr2=0x7f8d2c006ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:48.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7fcdf763c640 1 -- 172.21.15.63:0/649345254 wait complete. 2024-04-26T04:18:48.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7fcdf763c640 10 monclient: build_initial_monmap 2024-04-26T04:18:48.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7fcdf763c640 10 monclient: monmap: 2024-04-26T04:18:48.119 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:48.119 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:18:48.120 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:18:48.120 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:18:48.120 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:48.120 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:48.120 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:18:48.120 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:18:48.120 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:18:48.120 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:48.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7fcdf763c640 1 Processor -- start 2024-04-26T04:18:48.121 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7f8d38d32640 1 -- 172.21.15.63:0/2367056632 shutdown_connections 2024-04-26T04:18:48.121 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7f8d38d32640 1 -- 172.21.15.63:0/2367056632 wait complete. 2024-04-26T04:18:48.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7fcdf763c640 1 -- start start 2024-04-26T04:18:48.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7f4764ff9640 1 -- 172.21.15.63:0/955616177 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4768137020 con 0x7f47600bee10 2024-04-26T04:18:48.123 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7f4764ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:48.123 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7f4764ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:18:48.123 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7f4764ff9640 10 monclient: dump: 2024-04-26T04:18:48.123 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:48.123 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:48.123 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:48.123 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:48.124 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:48.124 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:48.124 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:48.124 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:48.124 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:48.124 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:48.124 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7f4764ff9640 10 monclient: _finish_auth 0 2024-04-26T04:18:48.125 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7f4764ff9640 10 monclient: _check_auth_tickets 2024-04-26T04:18:48.125 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7f4764ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:48.125 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7f79b17fa640 1 -- 172.21.15.63:0/2635447425 <== mon.2 v2:172.21.15.63:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f79a0003df0 con 0x7f79b40a47d0 2024-04-26T04:18:48.126 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7f79b17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:48.126 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7f79b17fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:18:48.127 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7f4764ff9640 1 -- 172.21.15.63:0/955616177 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47681318c0 con 0x7f47600bee10 2024-04-26T04:18:48.127 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7f4764ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:48.127 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7f47667fc640 4 set_mon_vals no callback set 2024-04-26T04:18:48.127 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7f4764ff9640 1 -- 172.21.15.63:0/955616177 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f476812d040 con 0x7f47600bee10 2024-04-26T04:18:48.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7f4764ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:48.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7f4764ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:18:48.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7f4764ff9640 10 monclient: dump: 2024-04-26T04:18:48.128 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:48.128 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:48.128 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:48.128 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:48.128 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:48.129 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:48.129 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:48.129 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:48.129 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:48.129 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:48.129 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7f476d21f640 5 monclient: authenticate success, global_id 4241 2024-04-26T04:18:48.129 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7f476d21f640 10 monclient: _renew_subs 2024-04-26T04:18:48.129 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7f476d21f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:48.129 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7f476d21f640 1 -- 172.21.15.63:0/955616177 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47600bb590 con 0x7f47600bee10 2024-04-26T04:18:48.129 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7f476d21f640 10 monclient: _renew_subs 2024-04-26T04:18:48.130 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7f476d21f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:48.130 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7f476d21f640 1 -- 172.21.15.63:0/955616177 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f47600bafd0 con 0x7f47600bee10 2024-04-26T04:18:48.130 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7fcdf763c640 10 monclient: init 2024-04-26T04:18:48.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.062+0000 7f476d21f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:48.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.062+0000 7f476d21f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:48.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.062+0000 7f476d21f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:48.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.062+0000 7f476d21f640 1 -- 172.21.15.63:0/955616177 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4760005720 con 0x7f47600bee10 2024-04-26T04:18:48.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.062+0000 7f4764ff9640 1 -- 172.21.15.63:0/955616177 <== mon.2 v2:172.21.15.63:3301/0 4 ==== mgrmap(e 4) ==== 43997+0+0 (secure 0 0 0) 0x7f476813a070 con 0x7f47600bee10 2024-04-26T04:18:48.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.062+0000 7f4764ff9640 1 --2- 172.21.15.63:0/955616177 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f474c035e30 0x7f474c0382f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:48.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.062+0000 7f4764ff9640 1 -- 172.21.15.63:0/955616177 <== mon.2 v2:172.21.15.63:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7f476815e630 con 0x7f47600bee10 2024-04-26T04:18:48.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.062+0000 7f4767fff640 1 --2- 172.21.15.63:0/955616177 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f474c035e30 0x7f474c0382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:48.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.062+0000 7f4767fff640 10 monclient: get_auth_request con 0x7f474c035e30 auth_method 0 2024-04-26T04:18:48.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7f4767fff640 1 --2- 172.21.15.63:0/955616177 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f474c035e30 0x7f474c0382f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f475c009a60 tx=0x7f475c009520 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:48.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7f4764ff9640 1 -- 172.21.15.63:0/955616177 <== mon.2 v2:172.21.15.63:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f4768091fa0 con 0x7f47600bee10 2024-04-26T04:18:48.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7f4764ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:48.134 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7f4764ff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:18:48.134 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7fcdf763c640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:48.134 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7fcdf763c640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T04:18:48.134 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7fcdf763c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcdf006fe70 0x7fcdf009f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:48.134 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7fcdf763c640 10 monclient(hunting): picked mon.noname-b con 0x7fcdf006fe70 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:18:48.135 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7fcdf763c640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcdf00a1d60 0x7fcdf009fe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:48.135 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7fcdf763c640 10 monclient(hunting): picked mon.noname-c con 0x7fcdf00a1d60 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:48.135 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7fcdf763c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcdf013f700 0x7fcdf01d2d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:48.135 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7fcdf763c640 10 monclient(hunting): picked mon.noname-a con 0x7fcdf013f700 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:18:48.135 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7fcdf763c640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:48.135 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7fcdf763c640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fcdf0144850 con 0x7fcdf013f700 2024-04-26T04:18:48.135 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7fcdf763c640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:48.135 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7fcdf763c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcdf009e360 con 0x7fcdf006fe70 2024-04-26T04:18:48.135 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7fcdf763c640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:48.135 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7fcdf763c640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7fcdf01449d0 con 0x7fcdf00a1d60 2024-04-26T04:18:48.135 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7fcdf763c640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:48.136 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7fcdf763c640 10 monclient(hunting): authenticate will time out at 959.778924s 2024-04-26T04:18:48.136 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7fcdf4bd7640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcdf00a1d60 0x7fcdf009fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:48.136 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7fcdf4bd7640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcdf00a1d60 0x7fcdf009fe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:57910/0 (socket says 172.21.15.63:57910) 2024-04-26T04:18:48.136 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7fcdf4bd7640 1 -- 172.21.15.63:0/1139037904 learned_addr learned my addr 172.21.15.63:0/1139037904 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:18:48.136 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7fcdf4bd7640 10 monclient(hunting): get_auth_request con 0x7fcdf00a1d60 auth_method 0 2024-04-26T04:18:48.136 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7fcdf4bd7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:48.136 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7fcdf4bd7640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:48.136 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7fcdf4bd7640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:48.136 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7fcdf4bd7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:48.136 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7fcdf4bd7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:48.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7fcdf4bd7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:48.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7fcdf5bd9640 1 --2- 172.21.15.63:0/1139037904 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcdf013f700 0x7fcdf01d2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:48.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7fcdf53d8640 1 --2- 172.21.15.63:0/1139037904 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcdf006fe70 0x7fcdf009f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:48.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7fcdf53d8640 10 monclient(hunting): get_auth_request con 0x7fcdf006fe70 auth_method 0 2024-04-26T04:18:48.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7fcdf53d8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:48.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7fcdf53d8640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:48.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7fcdf53d8640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:48.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7fcdf5bd9640 10 monclient(hunting): get_auth_request con 0x7fcdf013f700 auth_method 0 2024-04-26T04:18:48.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7fcdf5bd9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:48.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7fcdf5bd9640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:48.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7fcdf5bd9640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:48.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7fcdf4bd7640 10 monclient(hunting): handle_auth_done global_id 4244 payload 824 2024-04-26T04:18:48.138 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7fcdf4bd7640 1 -- 172.21.15.63:0/1139037904 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcdf006fe70 msgr2=0x7fcdf009f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:48.138 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7fcdf4bd7640 1 --2- 172.21.15.63:0/1139037904 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcdf006fe70 0x7fcdf009f930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:48.138 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7fcdf4bd7640 1 -- 172.21.15.63:0/1139037904 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcdf013f700 msgr2=0x7fcdf01d2d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:48.138 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7fcdf4bd7640 1 --2- 172.21.15.63:0/1139037904 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcdf013f700 0x7fcdf01d2d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:48.138 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7fcdf4bd7640 10 monclient: _finish_hunting 0 2024-04-26T04:18:48.138 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7fcdf4bd7640 1 monclient: found mon.noname-c 2024-04-26T04:18:48.138 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7fcdf4bd7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:48.138 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7fcdf4bd7640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.63:3301/0 2024-04-26T04:18:48.138 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7fcdf4bd7640 1 -- 172.21.15.63:0/1139037904 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdf006e820 con 0x7fcdf00a1d60 2024-04-26T04:18:48.139 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7f8d38d32640 10 monclient: build_initial_monmap 2024-04-26T04:18:48.139 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7f8d38d32640 10 monclient: monmap: 2024-04-26T04:18:48.140 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:48.140 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:18:48.140 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:18:48.140 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:18:48.140 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:48.140 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:48.140 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:18:48.140 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:18:48.140 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:18:48.140 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:48.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.058+0000 7f8d38d32640 1 Processor -- start 2024-04-26T04:18:48.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.062+0000 7f8d38d32640 1 -- start start 2024-04-26T04:18:48.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.062+0000 7f8d38d32640 10 monclient: init 2024-04-26T04:18:48.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.062+0000 7f8d38d32640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:48.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.062+0000 7f8d38d32640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T04:18:48.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.062+0000 7f8d38d32640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d2c023d00 0x7f8d2c0c4520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:48.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.062+0000 7f8d38d32640 10 monclient(hunting): picked mon.noname-b con 0x7f8d2c023d00 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:18:48.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.062+0000 7f8d38d32640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d2c0255d0 0x7f8d2c160d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:48.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.062+0000 7f8d38d32640 10 monclient(hunting): picked mon.noname-a con 0x7f8d2c0255d0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:18:48.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.062+0000 7f8d38d32640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f8d2c0c5460 0x7f8d2c161270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:48.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.062+0000 7f8d38d32640 10 monclient(hunting): picked mon.noname-c con 0x7f8d2c0c5460 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:48.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.062+0000 7f8d38d32640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:48.142 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.062+0000 7f8d38d32640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f8d2c0caa00 con 0x7f8d2c0255d0 2024-04-26T04:18:48.142 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.062+0000 7f8d38d32640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:48.142 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.062+0000 7f8d38d32640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8d2c0ca880 con 0x7f8d2c023d00 2024-04-26T04:18:48.142 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.062+0000 7f8d38d32640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:48.142 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.062+0000 7f8d38d32640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f8d2c0cab80 con 0x7f8d2c0c5460 2024-04-26T04:18:48.142 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.062+0000 7f8d38d32640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:48.142 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.062+0000 7f8d38d32640 10 monclient(hunting): authenticate will time out at 959.772869s 2024-04-26T04:18:48.142 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.062+0000 7f8d32ffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d2c0255d0 0x7f8d2c160d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:48.142 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.062+0000 7f8d32ffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d2c0255d0 0x7f8d2c160d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:46550/0 (socket says 172.21.15.63:46550) 2024-04-26T04:18:48.142 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.062+0000 7f8d32ffd640 1 -- 172.21.15.63:0/3837581086 learned_addr learned my addr 172.21.15.63:0/3837581086 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:18:48.142 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.062+0000 7f8d32ffd640 10 monclient(hunting): get_auth_request con 0x7f8d2c0255d0 auth_method 0 2024-04-26T04:18:48.142 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.062+0000 7f8d32ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:48.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.062+0000 7f8d32ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:48.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.062+0000 7f8d32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:48.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.062+0000 7f8d32ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:48.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.062+0000 7f8d32ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:48.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.062+0000 7f8d32ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:48.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.062+0000 7f8d32ffd640 10 monclient(hunting): handle_auth_done global_id 4257 payload 824 2024-04-26T04:18:48.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.062+0000 7f8d32ffd640 1 -- 172.21.15.63:0/3837581086 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f8d2c0c5460 msgr2=0x7f8d2c161270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:48.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.062+0000 7f8d32ffd640 1 --2- 172.21.15.63:0/3837581086 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f8d2c0c5460 0x7f8d2c161270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:48.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.062+0000 7f8d32ffd640 1 -- 172.21.15.63:0/3837581086 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d2c023d00 msgr2=0x7f8d2c0c4520 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T04:18:48.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.062+0000 7f8d32ffd640 1 --2- 172.21.15.63:0/3837581086 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d2c023d00 0x7f8d2c0c4520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:48.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.062+0000 7f8d32ffd640 10 monclient: _finish_hunting 0 2024-04-26T04:18:48.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.062+0000 7f8d32ffd640 1 monclient: found mon.noname-a 2024-04-26T04:18:48.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.062+0000 7f8d32ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:48.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.062+0000 7f8d32ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:18:48.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.062+0000 7f8d32ffd640 1 -- 172.21.15.63:0/3837581086 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d2c01f660 con 0x7f8d2c0255d0 2024-04-26T04:18:48.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.062+0000 7f8d32ffd640 1 -- 172.21.15.63:0/3837581086 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d2c0255d0 msgr2=0x7f8d2c160d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:48.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.062+0000 7f8d32ffd640 1 --2- 172.21.15.63:0/3837581086 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d2c0255d0 0x7f8d2c160d30 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8d20018f30 tx=0x7f8d200190c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:48.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.062+0000 7f8d1ffff640 1 -- 172.21.15.63:0/3837581086 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d200b9020 con 0x7f8d2c0255d0 2024-04-26T04:18:48.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7f8d1ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:48.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7f8d1ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:18:48.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7f8d1ffff640 10 monclient: dump: 2024-04-26T04:18:48.144 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:48.144 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:48.144 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:48.144 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:48.144 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:48.145 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:48.145 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:48.145 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:48.145 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:48.145 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:48.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7f8d1ffff640 10 monclient: _finish_auth 0 2024-04-26T04:18:48.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7f8d1ffff640 10 monclient: _check_auth_tickets 2024-04-26T04:18:48.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7f8d1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:48.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7f8d1ffff640 1 -- 172.21.15.63:0/3837581086 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d200b5aa0 con 0x7f8d2c0255d0 2024-04-26T04:18:48.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7f8d38d32640 5 monclient: authenticate success, global_id 4257 2024-04-26T04:18:48.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7f8d38d32640 10 monclient: _renew_subs 2024-04-26T04:18:48.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7f8d38d32640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:48.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7f8d38d32640 1 -- 172.21.15.63:0/3837581086 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d2c029760 con 0x7f8d2c0255d0 2024-04-26T04:18:48.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7f8d1ffff640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:48.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7f8d1ffff640 1 -- 172.21.15.63:0/3837581086 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d200b0040 con 0x7f8d2c0255d0 2024-04-26T04:18:48.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7f8d1ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:48.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7f8d1ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:18:48.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7f8d1ffff640 10 monclient: dump: 2024-04-26T04:18:48.146 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:48.146 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:48.146 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:48.146 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:48.146 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:48.146 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:48.147 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:48.147 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:48.147 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:48.147 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:48.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7f8d31ffb640 4 set_mon_vals no callback set 2024-04-26T04:18:48.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7f8d38d32640 10 monclient: _renew_subs 2024-04-26T04:18:48.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7f8d38d32640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:48.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7f8d38d32640 1 -- 172.21.15.63:0/3837581086 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d2c01d530 con 0x7f8d2c0255d0 2024-04-26T04:18:48.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7f8d38d32640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:48.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7f8d38d32640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:48.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7f8d38d32640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:48.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7f8d38d32640 1 -- 172.21.15.63:0/3837581086 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d2c0052d0 con 0x7f8d2c0255d0 2024-04-26T04:18:48.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7f8d1ffff640 1 -- 172.21.15.63:0/3837581086 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 4) ==== 43997+0+0 (secure 0 0 0) 0x7f8d200bc020 con 0x7f8d2c0255d0 2024-04-26T04:18:48.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7f8d1ffff640 1 --2- 172.21.15.63:0/3837581086 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f8d18035e60 0x7f8d18038320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:48.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.066+0000 7f8d1ffff640 1 -- 172.21.15.63:0/3837581086 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7f8d200e2fa0 con 0x7f8d2c0255d0 2024-04-26T04:18:48.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.070+0000 7f8d337fe640 1 --2- 172.21.15.63:0/3837581086 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f8d18035e60 0x7f8d18038320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:48.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.070+0000 7f8d337fe640 10 monclient: get_auth_request con 0x7f8d18035e60 auth_method 0 2024-04-26T04:18:48.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.070+0000 7fcdf4bd7640 1 -- 172.21.15.63:0/1139037904 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcdf00a1d60 msgr2=0x7fcdf009fe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:48.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.070+0000 7fcdf4bd7640 1 --2- 172.21.15.63:0/1139037904 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcdf00a1d60 0x7fcdf009fe70 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fcde001cc80 tx=0x7fcde001ac20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:48.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.070+0000 7fcde5ffb640 1 -- 172.21.15.63:0/1139037904 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcde00b1040 con 0x7fcdf00a1d60 2024-04-26T04:18:48.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.070+0000 7fcde5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:48.150 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.070+0000 7fcde5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:18:48.150 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.070+0000 7fcde5ffb640 10 monclient: dump: 2024-04-26T04:18:48.150 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:48.150 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:48.150 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:48.150 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:48.150 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:48.150 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:48.150 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:48.150 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:48.150 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:48.150 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:48.151 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.070+0000 7fcde5ffb640 10 monclient: _finish_auth 0 2024-04-26T04:18:48.151 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.070+0000 7fcde5ffb640 10 monclient: _check_auth_tickets 2024-04-26T04:18:48.151 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.070+0000 7fcde5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:48.151 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.070+0000 7fcdf763c640 5 monclient: authenticate success, global_id 4244 2024-04-26T04:18:48.151 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.070+0000 7fcdf763c640 10 monclient: _renew_subs 2024-04-26T04:18:48.151 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.070+0000 7fcdf763c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:48.151 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.070+0000 7fcdf763c640 1 -- 172.21.15.63:0/1139037904 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdf00a6660 con 0x7fcdf00a1d60 2024-04-26T04:18:48.151 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.070+0000 7fcdf763c640 10 monclient: _renew_subs 2024-04-26T04:18:48.151 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.070+0000 7fcdf763c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:48.151 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.070+0000 7fcdf763c640 1 -- 172.21.15.63:0/1139037904 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdf011efa0 con 0x7fcdf00a1d60 2024-04-26T04:18:48.151 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.070+0000 7fcde5ffb640 1 -- 172.21.15.63:0/1139037904 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcde00ac400 con 0x7fcdf00a1d60 2024-04-26T04:18:48.151 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.070+0000 7fcde5ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:48.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.070+0000 7fcde77fe640 4 set_mon_vals no callback set 2024-04-26T04:18:48.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.070+0000 7fcde5ffb640 1 -- 172.21.15.63:0/1139037904 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcde00b65c0 con 0x7fcdf00a1d60 2024-04-26T04:18:48.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.070+0000 7fcde5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:48.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.070+0000 7fcde5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:18:48.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.070+0000 7fcde5ffb640 10 monclient: dump: 2024-04-26T04:18:48.152 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:48.152 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:48.152 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:48.152 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:48.152 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:48.152 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:48.152 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:48.153 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:48.153 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:48.153 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:48.153 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.070+0000 7f8d337fe640 1 --2- 172.21.15.63:0/3837581086 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f8d18035e60 0x7f8d18038320 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8d28004f30 tx=0x7f8d28007ab0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:48.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.070+0000 7fcdf763c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:48.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.070+0000 7fcdf763c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:48.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.070+0000 7fcdf763c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:48.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.070+0000 7fcdf763c640 1 -- 172.21.15.63:0/1139037904 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcdd0005090 con 0x7fcdf00a1d60 2024-04-26T04:18:48.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.070+0000 7fcde5ffb640 1 -- 172.21.15.63:0/1139037904 <== mon.2 v2:172.21.15.63:3301/0 4 ==== mgrmap(e 4) ==== 43997+0+0 (secure 0 0 0) 0x7fcde00b6760 con 0x7fcdf00a1d60 2024-04-26T04:18:48.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.074+0000 7fcde5ffb640 1 --2- 172.21.15.63:0/1139037904 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fcdd4035e60 0x7fcdd4038320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:48.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.074+0000 7fcde5ffb640 1 -- 172.21.15.63:0/1139037904 <== mon.2 v2:172.21.15.63:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7fcde00e2400 con 0x7fcdf00a1d60 2024-04-26T04:18:48.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.074+0000 7fcdf53d8640 1 --2- 172.21.15.63:0/1139037904 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fcdd4035e60 0x7fcdd4038320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:48.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.074+0000 7fcdf53d8640 10 monclient: get_auth_request con 0x7fcdd4035e60 auth_method 0 2024-04-26T04:18:48.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.074+0000 7f8d1ffff640 1 -- 172.21.15.63:0/3837581086 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f8d2c01d530 con 0x7f8d2c0255d0 2024-04-26T04:18:48.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.074+0000 7f8d1ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:48.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.074+0000 7f8d1ffff640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:18:48.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.074+0000 7fcdf53d8640 1 --2- 172.21.15.63:0/1139037904 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fcdd4035e60 0x7fcdd4038320 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fcdf011e330 tx=0x7fcdec008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:48.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.078+0000 7fcde5ffb640 1 -- 172.21.15.63:0/1139037904 <== mon.2 v2:172.21.15.63:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fcde00160f0 con 0x7fcdf00a1d60 2024-04-26T04:18:48.157 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.078+0000 7fcde5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:48.157 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.078+0000 7fcde5ffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:18:48.199 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.194+0000 7fbebb487640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-04-26T04:18:48.199 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.194+0000 7fbebb487640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-04-26T04:18:48.199 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.194+0000 7fbebb487640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:48.199 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.194+0000 7fbebb487640 1 -- 172.21.15.63:0/1638960592 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fbeb41321c0 con 0x7fbeb41dc7e0 2024-04-26T04:18:48.199 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.194+0000 7fbea9ffb640 1 -- 172.21.15.63:0/1638960592 <== mon.2 v2:172.21.15.63:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fbeb4134e90 con 0x7fbeb41dc7e0 2024-04-26T04:18:48.199 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.194+0000 7fbea9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-04-26T04:18:48.199 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.194+0000 7fbea9ffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:18:48.199 INFO:teuthology.orchestra.run.smithi063.stdout:47244640258 2024-04-26T04:18:48.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.194+0000 7fbebb487640 1 -- 172.21.15.63:0/1638960592 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fbe98035e30 msgr2=0x7fbe980382f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:48.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.194+0000 7fbebb487640 1 --2- 172.21.15.63:0/1638960592 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fbe98035e30 0x7fbe980382f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fbeb0009af0 tx=0x7fbeb0008040 comp rx=0 tx=0).stop 2024-04-26T04:18:48.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.194+0000 7fbebb487640 10 monclient: shutdown 2024-04-26T04:18:48.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.194+0000 7fbebb487640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:48.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.194+0000 7fbebb487640 1 -- 172.21.15.63:0/1638960592 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fbeb41dc7e0 msgr2=0x7fbeb4130aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:48.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.194+0000 7fbebb487640 1 --2- 172.21.15.63:0/1638960592 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fbeb41dc7e0 0x7fbeb4130aa0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fbeac00caf0 tx=0x7fbeac00a060 comp rx=0 tx=0).stop 2024-04-26T04:18:48.202 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.194+0000 7fbebb487640 1 -- 172.21.15.63:0/1638960592 shutdown_connections 2024-04-26T04:18:48.202 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.194+0000 7fbebb487640 1 --2- 172.21.15.63:0/1638960592 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fbe98035e30 0x7fbe980382f0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:48.202 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.194+0000 7fbebb487640 1 --2- 172.21.15.63:0/1638960592 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fbeb41dc7e0 0x7fbeb4130aa0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:48.202 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.194+0000 7fbebb487640 1 --2- 172.21.15.63:0/1638960592 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbeb40a4c00 0x7fbeb41dc2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:48.202 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.194+0000 7fbebb487640 1 --2- 172.21.15.63:0/1638960592 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbeb40a2cd0 0x7fbeb41dbd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:48.202 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.194+0000 7fbebb487640 1 -- 172.21.15.63:0/1638960592 >> 172.21.15.63:0/1638960592 conn(0x7fbeb40a5f40 msgr2=0x7fbeb408dfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:48.202 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.194+0000 7fbebb487640 1 -- 172.21.15.63:0/1638960592 shutdown_connections 2024-04-26T04:18:48.202 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.194+0000 7fbebb487640 1 -- 172.21.15.63:0/1638960592 wait complete. 2024-04-26T04:18:48.218 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640258 for osd.3 2024-04-26T04:18:48.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.238+0000 7fcdf763c640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-04-26T04:18:48.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.238+0000 7fcdf763c640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-04-26T04:18:48.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.238+0000 7fcdf763c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:48.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.238+0000 7fcdf763c640 1 -- 172.21.15.63:0/1139037904 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fcdd0005380 con 0x7fcdf00a1d60 2024-04-26T04:18:48.244 INFO:teuthology.orchestra.run.smithi063.stdout:51539607555 2024-04-26T04:18:48.244 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.238+0000 7fcde5ffb640 1 -- 172.21.15.63:0/1139037904 <== mon.2 v2:172.21.15.63:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fcde00be370 con 0x7fcdf00a1d60 2024-04-26T04:18:48.245 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.238+0000 7fcde5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-04-26T04:18:48.245 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.238+0000 7fcde5ffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:18:48.245 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.238+0000 7fcdf763c640 1 -- 172.21.15.63:0/1139037904 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fcdd4035e60 msgr2=0x7fcdd4038320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:48.245 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.238+0000 7fcdf763c640 1 --2- 172.21.15.63:0/1139037904 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fcdd4035e60 0x7fcdd4038320 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fcdf011e330 tx=0x7fcdec008040 comp rx=0 tx=0).stop 2024-04-26T04:18:48.246 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.238+0000 7fcdf763c640 10 monclient: shutdown 2024-04-26T04:18:48.246 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.238+0000 7fcdf763c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:48.246 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.238+0000 7fcdf763c640 1 -- 172.21.15.63:0/1139037904 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcdf00a1d60 msgr2=0x7fcdf009fe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:48.246 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.238+0000 7fcdf763c640 1 --2- 172.21.15.63:0/1139037904 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcdf00a1d60 0x7fcdf009fe70 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fcde001cc80 tx=0x7fcde001ac20 comp rx=0 tx=0).stop 2024-04-26T04:18:48.246 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.238+0000 7fcdf763c640 1 -- 172.21.15.63:0/1139037904 shutdown_connections 2024-04-26T04:18:48.246 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.238+0000 7fcdf763c640 1 --2- 172.21.15.63:0/1139037904 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fcdd4035e60 0x7fcdd4038320 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:48.246 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.238+0000 7fcdf763c640 1 --2- 172.21.15.63:0/1139037904 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcdf013f700 0x7fcdf01d2d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:48.246 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.238+0000 7fcdf763c640 1 --2- 172.21.15.63:0/1139037904 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcdf00a1d60 0x7fcdf009fe70 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:48.246 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.238+0000 7fcdf763c640 1 --2- 172.21.15.63:0/1139037904 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcdf006fe70 0x7fcdf009f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:48.246 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.238+0000 7fcdf763c640 1 -- 172.21.15.63:0/1139037904 >> 172.21.15.63:0/1139037904 conn(0x7fcdf00a7360 msgr2=0x7fcdf0072610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:48.247 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.242+0000 7fcdf763c640 1 -- 172.21.15.63:0/1139037904 shutdown_connections 2024-04-26T04:18:48.247 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.242+0000 7fcdf763c640 1 -- 172.21.15.63:0/1139037904 wait complete. 2024-04-26T04:18:48.260 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607556 got 51539607555 for osd.1 2024-04-26T04:18:48.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.330+0000 7f476d21f640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-04-26T04:18:48.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.330+0000 7f476d21f640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-04-26T04:18:48.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.330+0000 7f476d21f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:48.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.330+0000 7f476d21f640 1 -- 172.21.15.63:0/955616177 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f476002a8f0 con 0x7f47600bee10 2024-04-26T04:18:48.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.330+0000 7f4764ff9640 1 -- 172.21.15.63:0/955616177 <== mon.2 v2:172.21.15.63:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f476813a980 con 0x7f47600bee10 2024-04-26T04:18:48.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.330+0000 7f4764ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-04-26T04:18:48.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.330+0000 7f4764ff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:18:48.336 INFO:teuthology.orchestra.run.smithi063.stdout:51539607555 2024-04-26T04:18:48.339 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.334+0000 7f476d21f640 1 -- 172.21.15.63:0/955616177 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f474c035e30 msgr2=0x7f474c0382f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:48.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.334+0000 7f476d21f640 1 --2- 172.21.15.63:0/955616177 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f474c035e30 0x7f474c0382f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f475c009a60 tx=0x7f475c009520 comp rx=0 tx=0).stop 2024-04-26T04:18:48.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.334+0000 7f476d21f640 10 monclient: shutdown 2024-04-26T04:18:48.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.334+0000 7f476d21f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:48.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.334+0000 7f476d21f640 1 -- 172.21.15.63:0/955616177 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f47600bee10 msgr2=0x7f4760008180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:48.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.334+0000 7f476d21f640 1 --2- 172.21.15.63:0/955616177 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f47600bee10 0x7f4760008180 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4768074710 tx=0x7f4768074130 comp rx=0 tx=0).stop 2024-04-26T04:18:48.341 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.334+0000 7f476d21f640 1 -- 172.21.15.63:0/955616177 shutdown_connections 2024-04-26T04:18:48.341 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.334+0000 7f476d21f640 1 --2- 172.21.15.63:0/955616177 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f474c035e30 0x7f474c0382f0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:48.341 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.334+0000 7f476d21f640 1 --2- 172.21.15.63:0/955616177 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f47600bee10 0x7f4760008180 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:48.341 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.334+0000 7f476d21f640 1 --2- 172.21.15.63:0/955616177 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4760027800 0x7f4760007c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:48.341 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.334+0000 7f476d21f640 1 --2- 172.21.15.63:0/955616177 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4760022300 0x7f47600be250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:48.341 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.334+0000 7f476d21f640 1 -- 172.21.15.63:0/955616177 >> 172.21.15.63:0/955616177 conn(0x7f47600068a0 msgr2=0x7f4760020aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:48.342 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.334+0000 7f476d21f640 1 -- 172.21.15.63:0/955616177 shutdown_connections 2024-04-26T04:18:48.342 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.334+0000 7f476d21f640 1 -- 172.21.15.63:0/955616177 wait complete. 2024-04-26T04:18:48.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.334+0000 7f8d38d32640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-04-26T04:18:48.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.334+0000 7f8d38d32640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-04-26T04:18:48.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.334+0000 7f8d38d32640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:48.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.334+0000 7f8d38d32640 1 -- 172.21.15.63:0/3837581086 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f8d2c01c820 con 0x7f8d2c0255d0 2024-04-26T04:18:48.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.334+0000 7f8d1ffff640 1 -- 172.21.15.63:0/3837581086 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f8d200c3030 con 0x7f8d2c0255d0 2024-04-26T04:18:48.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.334+0000 7f8d1ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-04-26T04:18:48.344 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.334+0000 7f8d1ffff640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:18:48.344 INFO:teuthology.orchestra.run.smithi063.stdout:51539607556 2024-04-26T04:18:48.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.338+0000 7f8d38d32640 1 -- 172.21.15.63:0/3837581086 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f8d18035e60 msgr2=0x7f8d18038320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:48.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.338+0000 7f8d38d32640 1 --2- 172.21.15.63:0/3837581086 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f8d18035e60 0x7f8d18038320 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8d28004f30 tx=0x7f8d28007ab0 comp rx=0 tx=0).stop 2024-04-26T04:18:48.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.338+0000 7f8d38d32640 10 monclient: shutdown 2024-04-26T04:18:48.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.338+0000 7f8d38d32640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:48.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.338+0000 7f8d38d32640 1 -- 172.21.15.63:0/3837581086 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d2c0255d0 msgr2=0x7f8d2c160d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:48.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.338+0000 7f8d38d32640 1 --2- 172.21.15.63:0/3837581086 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d2c0255d0 0x7f8d2c160d30 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8d20018f30 tx=0x7f8d200190c0 comp rx=0 tx=0).stop 2024-04-26T04:18:48.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.338+0000 7f8d38d32640 1 -- 172.21.15.63:0/3837581086 shutdown_connections 2024-04-26T04:18:48.346 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.338+0000 7f8d38d32640 1 --2- 172.21.15.63:0/3837581086 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f8d18035e60 0x7f8d18038320 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:48.346 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.338+0000 7f8d38d32640 1 --2- 172.21.15.63:0/3837581086 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f8d2c0c5460 0x7f8d2c161270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:48.346 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.338+0000 7f8d38d32640 1 --2- 172.21.15.63:0/3837581086 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d2c0255d0 0x7f8d2c160d30 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:48.346 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.338+0000 7f8d38d32640 1 --2- 172.21.15.63:0/3837581086 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d2c023d00 0x7f8d2c0c4520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:48.346 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.338+0000 7f8d38d32640 1 -- 172.21.15.63:0/3837581086 >> 172.21.15.63:0/3837581086 conn(0x7f8d2c0068d0 msgr2=0x7f8d2c028560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:48.346 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.338+0000 7f8d38d32640 1 -- 172.21.15.63:0/3837581086 shutdown_connections 2024-04-26T04:18:48.346 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.338+0000 7f8d38d32640 1 -- 172.21.15.63:0/3837581086 wait complete. 2024-04-26T04:18:48.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.346+0000 7f743a483640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-04-26T04:18:48.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.346+0000 7f743a483640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-04-26T04:18:48.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.346+0000 7f743a483640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:48.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.346+0000 7f743a483640 1 -- 172.21.15.63:0/1808898411 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f7434068260 con 0x7f7434071db0 2024-04-26T04:18:48.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.346+0000 7f7430ff9640 1 -- 172.21.15.63:0/1808898411 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f7434072a40 con 0x7f7434071db0 2024-04-26T04:18:48.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.346+0000 7f7430ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-04-26T04:18:48.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.346+0000 7f7430ff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:18:48.351 INFO:teuthology.orchestra.run.smithi063.stdout:47244640258 2024-04-26T04:18:48.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.346+0000 7f743a483640 1 -- 172.21.15.63:0/1808898411 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f742003a290 msgr2=0x7f742003c750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:48.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.346+0000 7f743a483640 1 --2- 172.21.15.63:0/1808898411 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f742003a290 0x7f742003c750 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7424009d50 tx=0x7f74240090e0 comp rx=0 tx=0).stop 2024-04-26T04:18:48.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.346+0000 7f743a483640 10 monclient: shutdown 2024-04-26T04:18:48.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.346+0000 7f743a483640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:48.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.346+0000 7f743a483640 1 -- 172.21.15.63:0/1808898411 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7434071db0 msgr2=0x7f74341373a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:48.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.346+0000 7f743a483640 1 --2- 172.21.15.63:0/1808898411 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7434071db0 0x7f74341373a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f742c00be00 tx=0x7f742c00bef0 comp rx=0 tx=0).stop 2024-04-26T04:18:48.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.346+0000 7f743a483640 1 -- 172.21.15.63:0/1808898411 shutdown_connections 2024-04-26T04:18:48.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.346+0000 7f743a483640 1 --2- 172.21.15.63:0/1808898411 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f742003a290 0x7f742003c750 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:48.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.346+0000 7f743a483640 1 --2- 172.21.15.63:0/1808898411 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7434071db0 0x7f74341373a0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:48.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.346+0000 7f743a483640 1 --2- 172.21.15.63:0/1808898411 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f7434071660 0x7f74341cd130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:48.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.346+0000 7f743a483640 1 --2- 172.21.15.63:0/1808898411 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7434071250 0x7f74341cca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:48.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.346+0000 7f743a483640 1 -- 172.21.15.63:0/1808898411 >> 172.21.15.63:0/1808898411 conn(0x7f74340a6170 msgr2=0x7f74340a1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:48.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.346+0000 7f743a483640 1 -- 172.21.15.63:0/1808898411 shutdown_connections 2024-04-26T04:18:48.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.346+0000 7f743a483640 1 -- 172.21.15.63:0/1808898411 wait complete. 2024-04-26T04:18:48.359 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607556 got 51539607555 for osd.2 2024-04-26T04:18:48.363 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607557 got 51539607556 for osd.0 2024-04-26T04:18:48.370 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640258 for osd.5 2024-04-26T04:18:48.373 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.366+0000 7f79bb0a4640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-04-26T04:18:48.373 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.366+0000 7f79bb0a4640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-04-26T04:18:48.373 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.366+0000 7f79bb0a4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:48.374 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.366+0000 7f79bb0a4640 1 -- 172.21.15.63:0/2635447425 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f7994005380 con 0x7f79b40a47d0 2024-04-26T04:18:48.374 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.366+0000 7f79b17fa640 1 -- 172.21.15.63:0/2635447425 <== mon.2 v2:172.21.15.63:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f79a0011520 con 0x7f79b40a47d0 2024-04-26T04:18:48.374 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.366+0000 7f79b17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-04-26T04:18:48.374 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.366+0000 7f79b17fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:18:48.374 INFO:teuthology.orchestra.run.smithi063.stdout:47244640258 2024-04-26T04:18:48.376 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.370+0000 7f79bb0a4640 1 -- 172.21.15.63:0/2635447425 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f7998035e30 msgr2=0x7f79980382f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:48.376 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.370+0000 7f79bb0a4640 1 --2- 172.21.15.63:0/2635447425 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f7998035e30 0x7f79980382f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f79a8006aa0 tx=0x7f79a8006a30 comp rx=0 tx=0).stop 2024-04-26T04:18:48.376 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.370+0000 7f79bb0a4640 10 monclient: shutdown 2024-04-26T04:18:48.376 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.370+0000 7f79bb0a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:48.376 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.370+0000 7f79bb0a4640 1 -- 172.21.15.63:0/2635447425 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f79b40a47d0 msgr2=0x7f79b41cf600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:48.377 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.370+0000 7f79bb0a4640 1 --2- 172.21.15.63:0/2635447425 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f79b40a47d0 0x7f79b41cf600 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f79a000c490 tx=0x7f79a0001e80 comp rx=0 tx=0).stop 2024-04-26T04:18:48.377 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.370+0000 7f79bb0a4640 1 -- 172.21.15.63:0/2635447425 shutdown_connections 2024-04-26T04:18:48.377 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.370+0000 7f79bb0a4640 1 --2- 172.21.15.63:0/2635447425 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f7998035e30 0x7f79980382f0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:48.377 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.370+0000 7f79bb0a4640 1 --2- 172.21.15.63:0/2635447425 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f79b40a4c00 0x7f79b413ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:48.377 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.370+0000 7f79bb0a4640 1 --2- 172.21.15.63:0/2635447425 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f79b40a47d0 0x7f79b41cf600 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:48.377 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.370+0000 7f79bb0a4640 1 --2- 172.21.15.63:0/2635447425 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79b40a2cd0 0x7f79b41ceed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:48.377 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.370+0000 7f79bb0a4640 1 -- 172.21.15.63:0/2635447425 >> 172.21.15.63:0/2635447425 conn(0x7f79b40a5f40 msgr2=0x7f79b40a2320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:48.377 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.370+0000 7f79bb0a4640 1 -- 172.21.15.63:0/2635447425 shutdown_connections 2024-04-26T04:18:48.377 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:48.370+0000 7f79bb0a4640 1 -- 172.21.15.63:0/2635447425 wait complete. 2024-04-26T04:18:48.391 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640258 for osd.4 2024-04-26T04:18:49.219 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-26T04:18:49.261 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-26T04:18:49.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.346+0000 7fbbcde5a640 10 monclient: get_monmap_and_config 2024-04-26T04:18:49.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.346+0000 7fbbcde5a640 10 monclient: build_initial_monmap 2024-04-26T04:18:49.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.346+0000 7fbbcde5a640 10 monclient: monmap: 2024-04-26T04:18:49.353 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:49.353 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:49.353 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:49.352771+0000 2024-04-26T04:18:49.353 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:49.352771+0000 2024-04-26T04:18:49.353 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:49.353 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:49.353 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:18:49.353 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:18:49.354 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:18:49.354 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:18:49.354 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:18:49.354 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:49.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.346+0000 7fbbcde5a640 1 Processor -- start 2024-04-26T04:18:49.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.346+0000 7fbbcde5a640 1 -- start start 2024-04-26T04:18:49.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.346+0000 7fbbcde5a640 10 monclient: init 2024-04-26T04:18:49.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.350+0000 7fbbcde5a640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:49.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.350+0000 7fbbcde5a640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-04-26T04:18:49.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.350+0000 7fbbcde5a640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fbbc8133e00 0x7fbbc81341e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:49.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.350+0000 7fbbcde5a640 10 monclient(hunting): picked mon.noname-b con 0x7fbbc8133e00 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:49.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.350+0000 7fbbcde5a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbbc80a26d0 addr v1:172.21.15.186:6789/0 2024-04-26T04:18:49.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.350+0000 7fbbcde5a640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fbbc809a8c0 0x7fbbc813bc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:49.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.350+0000 7fbbcde5a640 10 monclient(hunting): picked mon.noname-c con 0x7fbbc809a8c0 addr v2:172.21.15.186:3300/0 2024-04-26T04:18:49.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.350+0000 7fbbcde5a640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbc809bbf0 con 0x7fbbc80a26d0 2024-04-26T04:18:49.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.350+0000 7fbbcde5a640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:49.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.350+0000 7fbbcde5a640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fbbc809be60 con 0x7fbbc809a8c0 2024-04-26T04:18:49.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.350+0000 7fbbcde5a640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:49.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.350+0000 7fbbcde5a640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7fbbc809c140 con 0x7fbbc8133e00 2024-04-26T04:18:49.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.350+0000 7fbbcde5a640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:49.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.350+0000 7fbbcde5a640 10 monclient(hunting): authenticate will time out at 961.060561s 2024-04-26T04:18:49.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.350+0000 7fbbc77fe640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fbbc8133e00 0x7fbbc81341e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:49.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.350+0000 7fbbc77fe640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fbbc8133e00 0x7fbbc81341e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:36800/0 (socket says 172.21.15.63:36800) 2024-04-26T04:18:49.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.350+0000 7fbbc77fe640 1 -- 172.21.15.63:0/3426269777 learned_addr learned my addr 172.21.15.63:0/3426269777 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:18:49.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.350+0000 7fbbc77fe640 10 monclient(hunting): get_auth_request con 0x7fbbc8133e00 auth_method 0 2024-04-26T04:18:49.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.350+0000 7fbbc77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:49.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.350+0000 7fbbc77fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:49.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.350+0000 7fbbc77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:49.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.350+0000 7fbbc77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:49.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.350+0000 7fbbc77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:49.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.350+0000 7fbbc77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:49.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.350+0000 7fbbc77fe640 10 monclient(hunting): handle_auth_done global_id 4247 payload 274 2024-04-26T04:18:49.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.350+0000 7fbbc7fff640 1 --2- 172.21.15.63:0/3426269777 >> v2:172.21.15.186:3300/0 conn(0x7fbbc809a8c0 0x7fbbc813bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:49.359 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.350+0000 7fbbc77fe640 1 -- 172.21.15.63:0/3426269777 >> v2:172.21.15.186:3300/0 conn(0x7fbbc809a8c0 msgr2=0x7fbbc813bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:49.359 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.350+0000 7fbbc77fe640 1 --2- 172.21.15.63:0/3426269777 >> v2:172.21.15.186:3300/0 conn(0x7fbbc809a8c0 0x7fbbc813bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:49.359 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.350+0000 7fbbc77fe640 1 -- 172.21.15.63:0/3426269777 >> v1:172.21.15.186:6789/0 conn(0x7fbbc80a26d0 legacy=0x7fbbc809a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:49.359 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.350+0000 7fbbc77fe640 10 monclient: _finish_hunting 0 2024-04-26T04:18:49.359 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.350+0000 7fbbc77fe640 1 monclient: found mon.noname-b 2024-04-26T04:18:49.359 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.350+0000 7fbbc77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:49.359 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.350+0000 7fbbc77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.63:3301/0 2024-04-26T04:18:49.359 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.350+0000 7fbbc77fe640 1 -- 172.21.15.63:0/3426269777 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbc8134720 con 0x7fbbc8133e00 2024-04-26T04:18:49.359 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-26T04:18:49.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.354+0000 7fbbc77fe640 1 -- 172.21.15.63:0/3426269777 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fbbc8133e00 msgr2=0x7fbbc81341e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:49.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.354+0000 7fbbc77fe640 1 --2- 172.21.15.63:0/3426269777 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fbbc8133e00 0x7fbbc81341e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fbbb800bb10 tx=0x7fbbb801eb40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b016b82e86e7067 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:49.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.354+0000 7fbbc67fc640 1 -- 172.21.15.63:0/3426269777 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbb80b4070 con 0x7fbbc8133e00 2024-04-26T04:18:49.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.354+0000 7fbbc67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:49.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.354+0000 7fbbc67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T04:18:49.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.354+0000 7fbbc67fc640 10 monclient: dump: 2024-04-26T04:18:49.362 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:49.362 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:49.363 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:49.363 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:49.363 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:49.363 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:49.363 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:49.363 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:49.363 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:49.363 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:49.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.354+0000 7fbbc67fc640 10 monclient: _finish_auth 0 2024-04-26T04:18:49.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.354+0000 7fbbc67fc640 10 monclient: _check_auth_tickets 2024-04-26T04:18:49.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.354+0000 7fbbc67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:49.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.354+0000 7fbbc67fc640 1 -- 172.21.15.63:0/3426269777 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbb80a79b0 con 0x7fbbc8133e00 2024-04-26T04:18:49.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.354+0000 7fbbcde5a640 5 monclient: authenticate success, global_id 4247 2024-04-26T04:18:49.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.354+0000 7fbbcde5a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T04:18:49.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.354+0000 7fbbc67fc640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:49.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.354+0000 7fbbc67fc640 1 -- 172.21.15.63:0/3426269777 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbb80a7d80 con 0x7fbbc8133e00 2024-04-26T04:18:49.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.354+0000 7fbbcde5a640 10 monclient: get_monmap_and_config success 2024-04-26T04:18:49.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.354+0000 7fbbcde5a640 4 set_mon_vals no callback set 2024-04-26T04:18:49.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.354+0000 7fbbc67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:49.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.354+0000 7fbbc67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:18:49.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.354+0000 7fbbc67fc640 10 monclient: dump: 2024-04-26T04:18:49.364 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:49.364 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:49.364 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:49.364 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:49.364 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:49.364 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:49.364 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:49.365 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:49.365 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:49.365 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:49.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.354+0000 7fbbcde5a640 10 monclient: shutdown 2024-04-26T04:18:49.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.354+0000 7fbbcde5a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:49.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.354+0000 7fbbcde5a640 1 -- 172.21.15.63:0/3426269777 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fbbc8133e00 msgr2=0x7fbbc81341e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:49.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.354+0000 7fbbcde5a640 1 --2- 172.21.15.63:0/3426269777 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fbbc8133e00 0x7fbbc81341e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fbbb800bb10 tx=0x7fbbb801eb40 comp rx=0 tx=0).stop 2024-04-26T04:18:49.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.354+0000 7fbbcde5a640 1 -- 172.21.15.63:0/3426269777 shutdown_connections 2024-04-26T04:18:49.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.354+0000 7fbbcde5a640 1 --2- 172.21.15.63:0/3426269777 >> v2:172.21.15.186:3300/0 conn(0x7fbbc809a8c0 0x7fbbc813bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:49.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.354+0000 7fbbcde5a640 1 --2- 172.21.15.63:0/3426269777 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fbbc8133e00 0x7fbbc81341e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:49.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.354+0000 7fbbcde5a640 1 -- 172.21.15.63:0/3426269777 >> 172.21.15.63:0/3426269777 conn(0x7fbbc80a2c40 msgr2=0x7fbbc80a3020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:49.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.354+0000 7fbbcde5a640 1 -- 172.21.15.63:0/3426269777 shutdown_connections 2024-04-26T04:18:49.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.354+0000 7fbbcde5a640 1 -- 172.21.15.63:0/3426269777 wait complete. 2024-04-26T04:18:49.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.354+0000 7fbbcde5a640 10 monclient: build_initial_monmap 2024-04-26T04:18:49.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.354+0000 7fbbcde5a640 10 monclient: monmap: 2024-04-26T04:18:49.366 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:49.366 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:18:49.366 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:18:49.366 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:18:49.366 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:49.366 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:49.366 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:18:49.366 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:18:49.366 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:18:49.366 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:49.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.354+0000 7fbbcde5a640 1 Processor -- start 2024-04-26T04:18:49.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.354+0000 7fbbcde5a640 1 -- start start 2024-04-26T04:18:49.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.354+0000 7fbbcde5a640 10 monclient: init 2024-04-26T04:18:49.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.354+0000 7fbbcde5a640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:49.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.354+0000 7fbbcde5a640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T04:18:49.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.354+0000 7fbbcde5a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbbc809a8c0 0x7fbbc8097290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:49.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.354+0000 7fbbcde5a640 10 monclient(hunting): picked mon.noname-b con 0x7fbbc809a8c0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:18:49.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.354+0000 7fbbcde5a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbbc80a26d0 0x7fbbc80977d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:49.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.354+0000 7fbbcde5a640 10 monclient(hunting): picked mon.noname-a con 0x7fbbc80a26d0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:18:49.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.354+0000 7fbbcde5a640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fbbc8133e00 0x7fbbc8097d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:49.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.354+0000 7fbbcde5a640 10 monclient(hunting): picked mon.noname-c con 0x7fbbc8133e00 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:49.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.354+0000 7fbbcde5a640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:49.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.354+0000 7fbbcde5a640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fbbc806dd00 con 0x7fbbc80a26d0 2024-04-26T04:18:49.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.354+0000 7fbbcde5a640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:49.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.354+0000 7fbbcde5a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbbc806db80 con 0x7fbbc809a8c0 2024-04-26T04:18:49.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.354+0000 7fbbcde5a640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:49.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.354+0000 7fbbcde5a640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7fbbc806de80 con 0x7fbbc8133e00 2024-04-26T04:18:49.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.354+0000 7fbbcde5a640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:49.368 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.354+0000 7fbbcde5a640 10 monclient(hunting): authenticate will time out at 961.067993s 2024-04-26T04:18:49.368 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-26T04:18:49.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.358+0000 7fbbc7fff640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fbbc8133e00 0x7fbbc8097d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:49.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.358+0000 7fbbc7fff640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fbbc8133e00 0x7fbbc8097d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:36806/0 (socket says 172.21.15.63:36806) 2024-04-26T04:18:49.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.358+0000 7fbbc7fff640 1 -- 172.21.15.63:0/3414563736 learned_addr learned my addr 172.21.15.63:0/3414563736 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:18:49.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.358+0000 7fbbc7fff640 10 monclient(hunting): get_auth_request con 0x7fbbc8133e00 auth_method 0 2024-04-26T04:18:49.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.358+0000 7fbbc7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:49.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.358+0000 7fbbc7fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:49.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.358+0000 7fbbc7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:49.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.358+0000 7fbbc6ffd640 1 --2- 172.21.15.63:0/3414563736 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbbc80a26d0 0x7fbbc80977d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:49.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.358+0000 7fbbc6ffd640 10 monclient(hunting): get_auth_request con 0x7fbbc80a26d0 auth_method 0 2024-04-26T04:18:49.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.358+0000 7fbbc6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:49.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.358+0000 7fbbc6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:49.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.358+0000 7fbbc6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:49.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.358+0000 7fbbc7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:49.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.358+0000 7fbbc7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:49.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.358+0000 7fbbc7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:49.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.358+0000 7fbbc77fe640 1 --2- 172.21.15.63:0/3414563736 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbbc809a8c0 0x7fbbc8097290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:49.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.358+0000 7fbbc6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:49.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.358+0000 7fbbc6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:49.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.358+0000 7fbbc6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:49.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.358+0000 7fbbc77fe640 10 monclient(hunting): get_auth_request con 0x7fbbc809a8c0 auth_method 0 2024-04-26T04:18:49.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.358+0000 7fbbc77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:49.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.358+0000 7fbbc77fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:49.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.358+0000 7fbbc77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:49.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.358+0000 7fbbc7fff640 10 monclient(hunting): handle_auth_done global_id 4250 payload 824 2024-04-26T04:18:49.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.358+0000 7fbbc7fff640 1 -- 172.21.15.63:0/3414563736 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbbc809a8c0 msgr2=0x7fbbc8097290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:49.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.358+0000 7fbbc7fff640 1 --2- 172.21.15.63:0/3414563736 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbbc809a8c0 0x7fbbc8097290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:49.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.358+0000 7fbbc7fff640 1 -- 172.21.15.63:0/3414563736 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbbc80a26d0 msgr2=0x7fbbc80977d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:49.373 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.358+0000 7fbbc7fff640 1 --2- 172.21.15.63:0/3414563736 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbbc80a26d0 0x7fbbc80977d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:49.373 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.358+0000 7fbbc7fff640 10 monclient: _finish_hunting 0 2024-04-26T04:18:49.374 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.358+0000 7fbbc7fff640 1 monclient: found mon.noname-c 2024-04-26T04:18:49.374 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.358+0000 7fbbc7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:49.374 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.358+0000 7fbbc7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.63:3301/0 2024-04-26T04:18:49.374 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.358+0000 7fbbc7fff640 1 -- 172.21.15.63:0/3414563736 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbc8098430 con 0x7fbbc8133e00 2024-04-26T04:18:49.374 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.358+0000 7fbbc7fff640 1 -- 172.21.15.63:0/3414563736 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fbbc8133e00 msgr2=0x7fbbc8097d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:49.374 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.358+0000 7fbbc7fff640 1 --2- 172.21.15.63:0/3414563736 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fbbc8133e00 0x7fbbc8097d10 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fbbc000cd70 tx=0x7fbbc000a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:49.374 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.358+0000 7fbbaffff640 1 -- 172.21.15.63:0/3414563736 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbc0003ba0 con 0x7fbbc8133e00 2024-04-26T04:18:49.374 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.358+0000 7fbbaffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:49.374 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.358+0000 7fbbaffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:18:49.374 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.358+0000 7fbbaffff640 10 monclient: dump: 2024-04-26T04:18:49.374 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:49.374 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:49.374 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:49.374 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:49.375 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:49.375 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:49.375 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:49.375 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:49.375 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:49.375 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:49.375 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.358+0000 7fbbaffff640 10 monclient: _finish_auth 0 2024-04-26T04:18:49.375 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.358+0000 7fbbaffff640 10 monclient: _check_auth_tickets 2024-04-26T04:18:49.375 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.358+0000 7fbbaffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:49.375 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.358+0000 7fbbcde5a640 5 monclient: authenticate success, global_id 4250 2024-04-26T04:18:49.375 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.358+0000 7fbbaffff640 1 -- 172.21.15.63:0/3414563736 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbc0003d40 con 0x7fbbc8133e00 2024-04-26T04:18:49.375 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.358+0000 7fbbaffff640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:49.375 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.358+0000 7fbbaffff640 1 -- 172.21.15.63:0/3414563736 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbc0004240 con 0x7fbbc8133e00 2024-04-26T04:18:49.375 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.358+0000 7fbbaffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:49.376 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.358+0000 7fbbaffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:18:49.376 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.358+0000 7fbbaffff640 10 monclient: dump: 2024-04-26T04:18:49.376 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:49.376 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:49.376 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:49.376 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:49.376 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:49.376 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:49.376 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:49.376 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:49.376 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:49.376 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:49.376 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.358+0000 7fbbc5ffb640 4 set_mon_vals no callback set 2024-04-26T04:18:49.376 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.358+0000 7fbbcde5a640 10 monclient: _renew_subs 2024-04-26T04:18:49.377 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.358+0000 7fbbcde5a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:49.377 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.358+0000 7fbbcde5a640 1 -- 172.21.15.63:0/3414563736 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbc80997a0 con 0x7fbbc8133e00 2024-04-26T04:18:49.377 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.358+0000 7fbbcde5a640 10 monclient: _renew_subs 2024-04-26T04:18:49.377 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.358+0000 7fbbcde5a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:49.377 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.358+0000 7fbbcde5a640 1 -- 172.21.15.63:0/3414563736 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbc8099490 con 0x7fbbc8133e00 2024-04-26T04:18:49.377 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.362+0000 7fbbcde5a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:49.377 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.362+0000 7fbbcde5a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:49.377 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.362+0000 7fbbcde5a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:49.377 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.362+0000 7fbbcde5a640 1 -- 172.21.15.63:0/3414563736 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbc8062e10 con 0x7fbbc8133e00 2024-04-26T04:18:49.377 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.362+0000 7fbbaffff640 1 -- 172.21.15.63:0/3414563736 <== mon.2 v2:172.21.15.63:3301/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fbbc0004460 con 0x7fbbc8133e00 2024-04-26T04:18:49.377 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.362+0000 7fbbaffff640 1 --2- 172.21.15.63:0/3414563736 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fbbb4035ec0 0x7fbbb4038380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:49.377 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.362+0000 7fbbaffff640 1 -- 172.21.15.63:0/3414563736 <== mon.2 v2:172.21.15.63:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7fbbc0049760 con 0x7fbbc8133e00 2024-04-26T04:18:49.378 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.362+0000 7fbbc77fe640 1 --2- 172.21.15.63:0/3414563736 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fbbb4035ec0 0x7fbbb4038380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:49.378 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.362+0000 7fbbc77fe640 10 monclient: get_auth_request con 0x7fbbb4035ec0 auth_method 0 2024-04-26T04:18:49.378 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.366+0000 7fbbc77fe640 1 --2- 172.21.15.63:0/3414563736 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fbbb4035ec0 0x7fbbb4038380 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fbbc806b4c0 tx=0x7fbbb80a9070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:49.378 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-26T04:18:49.379 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.366+0000 7fbbaffff640 1 -- 172.21.15.63:0/3414563736 <== mon.2 v2:172.21.15.63:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fbbc0004750 con 0x7fbbc8133e00 2024-04-26T04:18:49.379 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.366+0000 7fbbaffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:49.379 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.366+0000 7fbbaffff640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:18:49.388 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.382+0000 7fed4dc0e640 10 monclient: get_monmap_and_config 2024-04-26T04:18:49.388 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.382+0000 7fed4dc0e640 10 monclient: build_initial_monmap 2024-04-26T04:18:49.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.382+0000 7fed4dc0e640 10 monclient: monmap: 2024-04-26T04:18:49.389 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:49.389 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:49.389 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:49.388839+0000 2024-04-26T04:18:49.389 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:49.388839+0000 2024-04-26T04:18:49.389 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:49.389 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:49.390 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:18:49.390 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:18:49.390 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:18:49.390 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:18:49.391 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:18:49.391 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:49.392 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.382+0000 7fed4dc0e640 1 Processor -- start 2024-04-26T04:18:49.392 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.382+0000 7fed4dc0e640 1 -- start start 2024-04-26T04:18:49.392 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.382+0000 7fed4dc0e640 10 monclient: init 2024-04-26T04:18:49.392 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.386+0000 7fed4dc0e640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:49.392 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.386+0000 7fed4dc0e640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-04-26T04:18:49.393 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.386+0000 7fed4dc0e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fed480a1c60 addr v1:172.21.15.63:6789/0 2024-04-26T04:18:49.393 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.386+0000 7fed4dc0e640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7fed4809bf30 0x7fed4813c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:49.393 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.386+0000 7fed4dc0e640 10 monclient(hunting): picked mon.noname-a con 0x7fed4809bf30 addr v2:172.21.15.63:3300/0 2024-04-26T04:18:49.393 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.386+0000 7fed4dc0e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fed4806dbf0 addr v1:172.21.15.186:6789/0 2024-04-26T04:18:49.393 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.386+0000 7fed4dc0e640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed4806d320 con 0x7fed480a1c60 2024-04-26T04:18:49.393 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.386+0000 7fed4dc0e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed4809b270 con 0x7fed4806dbf0 2024-04-26T04:18:49.393 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.386+0000 7fed4dc0e640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:49.393 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.386+0000 7fed4dc0e640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7fed4809b830 con 0x7fed4809bf30 2024-04-26T04:18:49.393 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.386+0000 7fed4dc0e640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:49.393 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.386+0000 7fed4dc0e640 10 monclient(hunting): authenticate will time out at 961.096518s 2024-04-26T04:18:49.393 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.386+0000 7fed477fe640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7fed480a1c60 0x7fed480a2040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:60998/0 (socket says 172.21.15.63:60998) 2024-04-26T04:18:49.393 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.386+0000 7fed477fe640 1 -- 172.21.15.63:0/597830860 learned_addr learned my addr 172.21.15.63:0/597830860 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-04-26T04:18:49.393 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.386+0000 7fed46ffd640 1 --2- 172.21.15.63:0/597830860 >> v2:172.21.15.63:3300/0 conn(0x7fed4809bf30 0x7fed4813c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:49.393 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.386+0000 7fed46ffd640 10 monclient(hunting): get_auth_request con 0x7fed4809bf30 auth_method 0 2024-04-26T04:18:49.393 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.386+0000 7fed46ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:49.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.386+0000 7fed46ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:49.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.386+0000 7fed46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:49.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.386+0000 7fed46ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:49.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.386+0000 7fed46ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:49.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.386+0000 7fed46ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:49.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.386+0000 7fed46ffd640 10 monclient(hunting): handle_auth_done global_id 4266 payload 274 2024-04-26T04:18:49.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.386+0000 7fed46ffd640 1 -- 172.21.15.63:0/597830860 >> v1:172.21.15.186:6789/0 conn(0x7fed4806dbf0 legacy=0x7fed4813e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:49.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.386+0000 7fed46ffd640 1 -- 172.21.15.63:0/597830860 >> v1:172.21.15.63:6789/0 conn(0x7fed480a1c60 legacy=0x7fed480a2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:49.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.386+0000 7fed46ffd640 10 monclient: _finish_hunting 0 2024-04-26T04:18:49.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.386+0000 7fed46ffd640 1 monclient: found mon.noname-a 2024-04-26T04:18:49.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.386+0000 7fed46ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:49.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.386+0000 7fed46ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:18:49.394 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.386+0000 7fed46ffd640 1 -- 172.21.15.63:0/597830860 --> v2:172.21.15.63:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed4809b550 con 0x7fed4809bf30 2024-04-26T04:18:49.395 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-26T04:18:49.396 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.390+0000 7fed46ffd640 1 -- 172.21.15.63:0/597830860 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fed4809bf30 msgr2=0x7fed4813c1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:49.396 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.390+0000 7fed46ffd640 1 --2- 172.21.15.63:0/597830860 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fed4809bf30 0x7fed4813c1f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fed480a3cf0 tx=0x7fed3c01c2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d642d3fda1cf6c0b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:49.396 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.390+0000 7fed467fc640 1 -- 172.21.15.63:0/597830860 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed3c0b0070 con 0x7fed4809bf30 2024-04-26T04:18:49.396 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.390+0000 7fed467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:49.396 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.390+0000 7fed467fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:18:49.396 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.390+0000 7fed467fc640 10 monclient: dump: 2024-04-26T04:18:49.397 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:49.397 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:49.397 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:49.397 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:49.397 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:49.397 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:49.397 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:49.397 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:49.397 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:49.397 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:49.397 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.390+0000 7fed467fc640 10 monclient: _finish_auth 0 2024-04-26T04:18:49.397 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.390+0000 7fed467fc640 10 monclient: _check_auth_tickets 2024-04-26T04:18:49.397 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.390+0000 7fed467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:49.397 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.390+0000 7fed4dc0e640 5 monclient: authenticate success, global_id 4266 2024-04-26T04:18:49.397 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.390+0000 7fed4dc0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T04:18:49.398 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.390+0000 7fed467fc640 1 -- 172.21.15.63:0/597830860 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed3c0ab600 con 0x7fed4809bf30 2024-04-26T04:18:49.398 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.390+0000 7fed467fc640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:49.398 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.390+0000 7fed4dc0e640 10 monclient: get_monmap_and_config success 2024-04-26T04:18:49.398 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.390+0000 7fed4dc0e640 4 set_mon_vals no callback set 2024-04-26T04:18:49.399 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.390+0000 7fed4dc0e640 10 monclient: shutdown 2024-04-26T04:18:49.399 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.390+0000 7fed4dc0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:49.399 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.390+0000 7fed4dc0e640 1 -- 172.21.15.63:0/597830860 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fed4809bf30 msgr2=0x7fed4813c1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:49.399 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.390+0000 7fed4dc0e640 1 --2- 172.21.15.63:0/597830860 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fed4809bf30 0x7fed4813c1f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fed480a3cf0 tx=0x7fed3c01c2c0 comp rx=0 tx=0).stop 2024-04-26T04:18:49.399 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.390+0000 7fed4dc0e640 1 -- 172.21.15.63:0/597830860 shutdown_connections 2024-04-26T04:18:49.399 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.390+0000 7fed4dc0e640 1 --2- 172.21.15.63:0/597830860 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fed4809bf30 0x7fed4813c1f0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:49.399 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.390+0000 7fed4dc0e640 1 -- 172.21.15.63:0/597830860 >> 172.21.15.63:0/597830860 conn(0x7fed480a6d80 msgr2=0x7fed480a2d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:49.399 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.390+0000 7fed4dc0e640 1 -- 172.21.15.63:0/597830860 shutdown_connections 2024-04-26T04:18:49.400 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.390+0000 7fed4dc0e640 1 -- 172.21.15.63:0/597830860 wait complete. 2024-04-26T04:18:49.400 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.394+0000 7fed4dc0e640 10 monclient: build_initial_monmap 2024-04-26T04:18:49.400 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.394+0000 7fed4dc0e640 10 monclient: monmap: 2024-04-26T04:18:49.400 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:49.400 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:18:49.400 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:18:49.400 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:18:49.400 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:49.400 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:49.400 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:18:49.400 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:18:49.400 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:18:49.400 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:49.400 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.394+0000 7fed4dc0e640 1 Processor -- start 2024-04-26T04:18:49.401 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.394+0000 7fed4dc0e640 1 -- start start 2024-04-26T04:18:49.401 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.394+0000 7fed4dc0e640 10 monclient: init 2024-04-26T04:18:49.401 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.394+0000 7fed4dc0e640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:49.401 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.394+0000 7fed4dc0e640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T04:18:49.401 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.394+0000 7fed4dc0e640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fed4806dbf0 0x7fed480724f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:49.401 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.394+0000 7fed4dc0e640 10 monclient(hunting): picked mon.noname-c con 0x7fed4806dbf0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:49.401 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.394+0000 7fed4dc0e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fed4809bf30 0x7fed48072a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:49.401 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.394+0000 7fed4dc0e640 10 monclient(hunting): picked mon.noname-a con 0x7fed4809bf30 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:18:49.401 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.394+0000 7fed4dc0e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed480a1c60 0x7fed4813ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:49.402 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.394+0000 7fed4dc0e640 10 monclient(hunting): picked mon.noname-b con 0x7fed480a1c60 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:18:49.402 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.394+0000 7fed4dc0e640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:49.402 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.394+0000 7fed4dc0e640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fed48142ab0 con 0x7fed4809bf30 2024-04-26T04:18:49.402 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.394+0000 7fed4dc0e640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:49.402 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.394+0000 7fed4dc0e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fed48142930 con 0x7fed480a1c60 2024-04-26T04:18:49.402 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.394+0000 7fed4dc0e640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:49.402 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.394+0000 7fed4dc0e640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7fed48142c30 con 0x7fed4806dbf0 2024-04-26T04:18:49.402 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.394+0000 7fed4dc0e640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:49.402 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.394+0000 7fed4dc0e640 10 monclient(hunting): authenticate will time out at 961.106158s 2024-04-26T04:18:49.402 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.394+0000 7fed46ffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fed4809bf30 0x7fed48072a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:49.402 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.394+0000 7fed46ffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fed4809bf30 0x7fed48072a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:39146/0 (socket says 172.21.15.63:39146) 2024-04-26T04:18:49.402 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.394+0000 7fed46ffd640 1 -- 172.21.15.63:0/3935969200 learned_addr learned my addr 172.21.15.63:0/3935969200 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:18:49.402 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.394+0000 7fed46ffd640 10 monclient(hunting): get_auth_request con 0x7fed4809bf30 auth_method 0 2024-04-26T04:18:49.402 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.394+0000 7fed46ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:49.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.394+0000 7fed46ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:49.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.394+0000 7fed46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:49.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.394+0000 7fed46ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:49.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.394+0000 7fed46ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:49.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.394+0000 7fed46ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:49.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.394+0000 7fed477fe640 1 --2- 172.21.15.63:0/3935969200 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fed4806dbf0 0x7fed480724f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:49.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.394+0000 7fed477fe640 10 monclient(hunting): get_auth_request con 0x7fed4806dbf0 auth_method 0 2024-04-26T04:18:49.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.394+0000 7fed477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:49.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.394+0000 7fed477fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:49.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.394+0000 7fed477fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:49.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.394+0000 7fed47fff640 1 --2- 172.21.15.63:0/3935969200 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed480a1c60 0x7fed4813ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:49.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.394+0000 7fed47fff640 10 monclient(hunting): get_auth_request con 0x7fed480a1c60 auth_method 0 2024-04-26T04:18:49.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.394+0000 7fed47fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:49.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.394+0000 7fed47fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:49.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.394+0000 7fed47fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:49.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.394+0000 7fed46ffd640 10 monclient(hunting): handle_auth_done global_id 4269 payload 824 2024-04-26T04:18:49.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.394+0000 7fed46ffd640 1 -- 172.21.15.63:0/3935969200 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fed4806dbf0 msgr2=0x7fed480724f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:49.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.394+0000 7fed46ffd640 1 --2- 172.21.15.63:0/3935969200 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fed4806dbf0 0x7fed480724f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:49.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.394+0000 7fed46ffd640 1 -- 172.21.15.63:0/3935969200 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed480a1c60 msgr2=0x7fed4813ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:49.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.394+0000 7fed46ffd640 1 --2- 172.21.15.63:0/3935969200 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed480a1c60 0x7fed4813ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:49.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.394+0000 7fed46ffd640 10 monclient: _finish_hunting 0 2024-04-26T04:18:49.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.394+0000 7fed46ffd640 1 monclient: found mon.noname-a 2024-04-26T04:18:49.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.394+0000 7fed46ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:49.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.394+0000 7fed46ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:18:49.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.394+0000 7fed46ffd640 1 -- 172.21.15.63:0/3935969200 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed4809c910 con 0x7fed4809bf30 2024-04-26T04:18:49.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.394+0000 7fed477fe640 1 --2- 172.21.15.63:0/3935969200 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fed4806dbf0 0x7fed480724f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T04:18:49.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.398+0000 7fed46ffd640 1 -- 172.21.15.63:0/3935969200 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fed4809bf30 msgr2=0x7fed48072a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:49.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.398+0000 7fed46ffd640 1 --2- 172.21.15.63:0/3935969200 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fed4809bf30 0x7fed48072a30 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fed3c01c7a0 tx=0x7fed3c019060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:49.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.398+0000 7fed2ffff640 1 -- 172.21.15.63:0/3935969200 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed3c0b9020 con 0x7fed4809bf30 2024-04-26T04:18:49.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.398+0000 7fed2ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:49.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.398+0000 7fed2ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:18:49.406 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.398+0000 7fed2ffff640 10 monclient: dump: 2024-04-26T04:18:49.406 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:49.406 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:49.406 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:49.406 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:49.406 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:49.407 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:49.407 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:49.407 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:49.407 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:49.407 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:49.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.398+0000 7fed2ffff640 10 monclient: _finish_auth 0 2024-04-26T04:18:49.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.398+0000 7fed2ffff640 10 monclient: _check_auth_tickets 2024-04-26T04:18:49.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.398+0000 7fed2ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:49.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.398+0000 7fed2ffff640 1 -- 172.21.15.63:0/3935969200 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed3c0aacf0 con 0x7fed4809bf30 2024-04-26T04:18:49.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.398+0000 7fed2ffff640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:49.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.398+0000 7fed2ffff640 1 -- 172.21.15.63:0/3935969200 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed3c0b0040 con 0x7fed4809bf30 2024-04-26T04:18:49.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.398+0000 7fed2ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:49.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.398+0000 7fed2ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:18:49.407 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.398+0000 7fed2ffff640 10 monclient: dump: 2024-04-26T04:18:49.407 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:49.408 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:49.408 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:49.408 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:49.408 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:49.408 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:49.408 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:49.408 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:49.408 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:49.408 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:49.408 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.398+0000 7fed45ffb640 4 set_mon_vals no callback set 2024-04-26T04:18:49.408 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.398+0000 7fed4dc0e640 5 monclient: authenticate success, global_id 4269 2024-04-26T04:18:49.408 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.398+0000 7fed4dc0e640 10 monclient: _renew_subs 2024-04-26T04:18:49.408 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.398+0000 7fed4dc0e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:49.408 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.398+0000 7fed4dc0e640 1 -- 172.21.15.63:0/3935969200 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed4806efa0 con 0x7fed4809bf30 2024-04-26T04:18:49.408 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.398+0000 7fed4dc0e640 10 monclient: _renew_subs 2024-04-26T04:18:49.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.398+0000 7fed4dc0e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:49.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.398+0000 7fed4dc0e640 1 -- 172.21.15.63:0/3935969200 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed4812ed80 con 0x7fed4809bf30 2024-04-26T04:18:49.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.398+0000 7fed2ffff640 1 -- 172.21.15.63:0/3935969200 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fed3c0bc020 con 0x7fed4809bf30 2024-04-26T04:18:49.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.398+0000 7fed4dc0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:49.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.398+0000 7fed4dc0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:49.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.398+0000 7fed2ffff640 1 --2- 172.21.15.63:0/3935969200 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fed34035f10 0x7fed340383d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:49.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.398+0000 7fed2ffff640 1 -- 172.21.15.63:0/3935969200 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7fed3c0e3230 con 0x7fed4809bf30 2024-04-26T04:18:49.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.398+0000 7fed4dc0e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:49.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.398+0000 7fed4dc0e640 1 -- 172.21.15.63:0/3935969200 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed48062e10 con 0x7fed4809bf30 2024-04-26T04:18:49.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.402+0000 7fed477fe640 1 --2- 172.21.15.63:0/3935969200 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fed34035f10 0x7fed340383d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:49.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.402+0000 7fed477fe640 10 monclient: get_auth_request con 0x7fed34035f10 auth_method 0 2024-04-26T04:18:49.410 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.402+0000 7fed477fe640 1 --2- 172.21.15.63:0/3935969200 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fed34035f10 0x7fed340383d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fed38007fd0 tx=0x7fed38009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:49.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.410+0000 7fed2ffff640 1 -- 172.21.15.63:0/3935969200 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fed3c0c3030 con 0x7fed4809bf30 2024-04-26T04:18:49.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.410+0000 7fed2ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:49.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.410+0000 7fed2ffff640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:18:49.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.482+0000 7f788ab4a640 10 monclient: get_monmap_and_config 2024-04-26T04:18:49.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.482+0000 7f788ab4a640 10 monclient: build_initial_monmap 2024-04-26T04:18:49.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.482+0000 7f788ab4a640 10 monclient: monmap: 2024-04-26T04:18:49.487 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:49.487 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:49.488 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:49.487080+0000 2024-04-26T04:18:49.488 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:49.487080+0000 2024-04-26T04:18:49.488 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:49.488 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:49.488 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:18:49.488 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:18:49.488 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:18:49.488 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:18:49.488 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:18:49.488 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:49.488 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.482+0000 7f788ab4a640 1 Processor -- start 2024-04-26T04:18:49.488 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.482+0000 7f788ab4a640 1 -- start start 2024-04-26T04:18:49.488 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.482+0000 7f788ab4a640 10 monclient: init 2024-04-26T04:18:49.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.482+0000 7f788ab4a640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:49.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.482+0000 7f788ab4a640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-04-26T04:18:49.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.482+0000 7f788ab4a640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f78840a2cd0 0x7f78840a30b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:49.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.482+0000 7f788ab4a640 10 monclient(hunting): picked mon.noname-a con 0x7f78840a2cd0 addr v2:172.21.15.63:3300/0 2024-04-26T04:18:49.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.482+0000 7f788ab4a640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f78840a47d0 0x7f7884136ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:49.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.482+0000 7f788ab4a640 10 monclient(hunting): picked mon.noname-b con 0x7f78840a47d0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:49.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.482+0000 7f788ab4a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f78840a4bd0 addr v1:172.21.15.186:6789/0 2024-04-26T04:18:49.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.482+0000 7f788ab4a640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78840a0fb0 con 0x7f78840a4bd0 2024-04-26T04:18:49.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.482+0000 7f788ab4a640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:49.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.482+0000 7f788ab4a640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7f78840a35f0 con 0x7f78840a2cd0 2024-04-26T04:18:49.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.482+0000 7f788ab4a640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:49.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.482+0000 7f7883fff640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f78840a47d0 0x7f7884136ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:49.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.482+0000 7f7883fff640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f78840a47d0 0x7f7884136ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:36824/0 (socket says 172.21.15.63:36824) 2024-04-26T04:18:49.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.482+0000 7f7883fff640 1 -- 172.21.15.63:0/3709940185 learned_addr learned my addr 172.21.15.63:0/3709940185 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:18:49.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.482+0000 7f788ab4a640 1 -- 172.21.15.63:0/3709940185 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f78840a5380 con 0x7f78840a47d0 2024-04-26T04:18:49.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.482+0000 7f788ab4a640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:49.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.482+0000 7f78888e6640 1 --2- 172.21.15.63:0/3709940185 >> v2:172.21.15.63:3300/0 conn(0x7f78840a2cd0 0x7f78840a30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:49.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.482+0000 7f788ab4a640 10 monclient(hunting): authenticate will time out at 961.194829s 2024-04-26T04:18:49.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.482+0000 7f78888e6640 10 monclient(hunting): get_auth_request con 0x7f78840a2cd0 auth_method 0 2024-04-26T04:18:49.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.482+0000 7f78888e6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:49.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.482+0000 7f78888e6640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:49.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.482+0000 7f78888e6640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:49.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.482+0000 7f7883fff640 10 monclient(hunting): get_auth_request con 0x7f78840a47d0 auth_method 0 2024-04-26T04:18:49.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.482+0000 7f7883fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:49.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.482+0000 7f7883fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:49.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.482+0000 7f7883fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:49.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.482+0000 7f7883fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:49.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.482+0000 7f7883fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:49.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.482+0000 7f7883fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:49.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.482+0000 7f78888e6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:49.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.482+0000 7f78888e6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:49.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.482+0000 7f78888e6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:49.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.482+0000 7f7883fff640 10 monclient(hunting): handle_auth_done global_id 4256 payload 274 2024-04-26T04:18:49.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.482+0000 7f7883fff640 1 -- 172.21.15.63:0/3709940185 >> v2:172.21.15.63:3300/0 conn(0x7f78840a2cd0 msgr2=0x7f78840a30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:49.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.482+0000 7f7883fff640 1 --2- 172.21.15.63:0/3709940185 >> v2:172.21.15.63:3300/0 conn(0x7f78840a2cd0 0x7f78840a30b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:49.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.482+0000 7f7883fff640 1 -- 172.21.15.63:0/3709940185 >> v1:172.21.15.186:6789/0 conn(0x7f78840a4bd0 legacy=0x7f7884137220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:49.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.482+0000 7f7883fff640 10 monclient: _finish_hunting 0 2024-04-26T04:18:49.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.482+0000 7f7883fff640 1 monclient: found mon.noname-b 2024-04-26T04:18:49.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.482+0000 7f7883fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:49.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.482+0000 7f7883fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.63:3301/0 2024-04-26T04:18:49.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.482+0000 7f7883fff640 1 -- 172.21.15.63:0/3709940185 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78840a1290 con 0x7f78840a47d0 2024-04-26T04:18:49.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.486+0000 7f7883fff640 1 -- 172.21.15.63:0/3709940185 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f78840a47d0 msgr2=0x7f7884136ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:49.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.486+0000 7f7883fff640 1 --2- 172.21.15.63:0/3709940185 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f78840a47d0 0x7f7884136ce0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7870009870 tx=0x7f787001c470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a019d3e57ef4f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:49.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.486+0000 7f78837fe640 1 -- 172.21.15.63:0/3709940185 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78700b0070 con 0x7f78840a47d0 2024-04-26T04:18:49.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.486+0000 7f78837fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:49.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.486+0000 7f78837fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T04:18:49.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.486+0000 7f78837fe640 10 monclient: dump: 2024-04-26T04:18:49.495 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:49.495 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:49.495 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:49.495 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:49.495 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:49.495 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:49.495 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:49.495 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:49.495 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:49.496 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:49.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.486+0000 7f78837fe640 10 monclient: _finish_auth 0 2024-04-26T04:18:49.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.486+0000 7f78837fe640 10 monclient: _check_auth_tickets 2024-04-26T04:18:49.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.486+0000 7f78837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:49.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.486+0000 7f78837fe640 1 -- 172.21.15.63:0/3709940185 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78700ab600 con 0x7f78840a47d0 2024-04-26T04:18:49.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.486+0000 7f788ab4a640 5 monclient: authenticate success, global_id 4256 2024-04-26T04:18:49.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.486+0000 7f788ab4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T04:18:49.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.486+0000 7f78837fe640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:49.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.486+0000 7f788ab4a640 10 monclient: get_monmap_and_config success 2024-04-26T04:18:49.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.486+0000 7f78837fe640 1 -- 172.21.15.63:0/3709940185 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78700ab9d0 con 0x7f78840a47d0 2024-04-26T04:18:49.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.486+0000 7f788ab4a640 4 set_mon_vals no callback set 2024-04-26T04:18:49.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.486+0000 7f78837fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:49.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.486+0000 7f78837fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:18:49.497 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.486+0000 7f78837fe640 10 monclient: dump: 2024-04-26T04:18:49.497 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:49.497 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:49.497 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:49.497 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:49.497 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:49.497 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:49.497 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:49.497 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:49.497 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:49.497 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:49.497 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.490+0000 7f788ab4a640 10 monclient: shutdown 2024-04-26T04:18:49.497 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.490+0000 7f788ab4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:49.497 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.490+0000 7f788ab4a640 1 -- 172.21.15.63:0/3709940185 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f78840a47d0 msgr2=0x7f7884136ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:49.498 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.490+0000 7f788ab4a640 1 --2- 172.21.15.63:0/3709940185 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f78840a47d0 0x7f7884136ce0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7870009870 tx=0x7f787001c470 comp rx=0 tx=0).stop 2024-04-26T04:18:49.498 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.490+0000 7f788ab4a640 1 -- 172.21.15.63:0/3709940185 shutdown_connections 2024-04-26T04:18:49.498 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.490+0000 7f788ab4a640 1 --2- 172.21.15.63:0/3709940185 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f78840a47d0 0x7f7884136ce0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:49.498 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.490+0000 7f788ab4a640 1 --2- 172.21.15.63:0/3709940185 >> v2:172.21.15.63:3300/0 conn(0x7f78840a2cd0 0x7f78840a30b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:49.498 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.490+0000 7f788ab4a640 1 -- 172.21.15.63:0/3709940185 >> 172.21.15.63:0/3709940185 conn(0x7f78840a5f40 msgr2=0x7f78840a6320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:49.498 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.490+0000 7f788ab4a640 1 -- 172.21.15.63:0/3709940185 shutdown_connections 2024-04-26T04:18:49.498 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.490+0000 7f788ab4a640 1 -- 172.21.15.63:0/3709940185 wait complete. 2024-04-26T04:18:49.498 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.490+0000 7f788ab4a640 10 monclient: build_initial_monmap 2024-04-26T04:18:49.498 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.490+0000 7f788ab4a640 10 monclient: monmap: 2024-04-26T04:18:49.498 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:49.498 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:18:49.498 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:18:49.498 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:18:49.498 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:49.499 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:49.499 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:18:49.499 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:18:49.499 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:18:49.499 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:49.503 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.490+0000 7f788ab4a640 1 Processor -- start 2024-04-26T04:18:49.503 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.490+0000 7f788ab4a640 1 -- start start 2024-04-26T04:18:49.503 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.490+0000 7f788ab4a640 10 monclient: init 2024-04-26T04:18:49.503 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.490+0000 7f788ab4a640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:49.503 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.490+0000 7f788ab4a640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T04:18:49.503 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.490+0000 7f788ab4a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f78840a2cd0 0x7f788412ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:49.503 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.490+0000 7f788ab4a640 10 monclient(hunting): picked mon.noname-b con 0x7f78840a2cd0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:18:49.503 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.490+0000 7f788ab4a640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f78840a47d0 0x7f788412e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:49.503 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.490+0000 7f788ab4a640 10 monclient(hunting): picked mon.noname-c con 0x7f78840a47d0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:49.503 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.490+0000 7f788ab4a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f78840a4bd0 0x7f788406f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:49.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.490+0000 7f788ab4a640 10 monclient(hunting): picked mon.noname-a con 0x7f78840a4bd0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:18:49.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.490+0000 7f788ab4a640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:49.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.490+0000 7f788ab4a640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f7884141bd0 con 0x7f78840a4bd0 2024-04-26T04:18:49.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.490+0000 7f7883fff640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f78840a47d0 0x7f788412e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:49.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.490+0000 7f788ab4a640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:49.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.490+0000 7f7883fff640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f78840a47d0 0x7f788412e310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:36834/0 (socket says 172.21.15.63:36834) 2024-04-26T04:18:49.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.490+0000 7f7883fff640 1 -- 172.21.15.63:0/3268446327 learned_addr learned my addr 172.21.15.63:0/3268446327 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:18:49.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.490+0000 7f788ab4a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7884141a50 con 0x7f78840a2cd0 2024-04-26T04:18:49.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.490+0000 7f788ab4a640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:49.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.490+0000 7f788ab4a640 1 -- 172.21.15.63:0/3268446327 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f7884141d50 con 0x7f78840a47d0 2024-04-26T04:18:49.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.490+0000 7f78888e6640 1 --2- 172.21.15.63:0/3268446327 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f78840a2cd0 0x7f788412ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:49.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.490+0000 7f788ab4a640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:49.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.490+0000 7f78890e7640 1 --2- 172.21.15.63:0/3268446327 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f78840a4bd0 0x7f788406f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:49.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.490+0000 7f788ab4a640 10 monclient(hunting): authenticate will time out at 961.203027s 2024-04-26T04:18:49.505 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.490+0000 7f7883fff640 10 monclient(hunting): get_auth_request con 0x7f78840a47d0 auth_method 0 2024-04-26T04:18:49.505 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.490+0000 7f7883fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:49.505 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.490+0000 7f7883fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:49.505 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.490+0000 7f7883fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:49.505 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.490+0000 7f78890e7640 10 monclient(hunting): get_auth_request con 0x7f78840a4bd0 auth_method 0 2024-04-26T04:18:49.505 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.490+0000 7f78890e7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:49.505 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.490+0000 7f78890e7640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:49.505 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.490+0000 7f78890e7640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:49.505 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.490+0000 7f78888e6640 10 monclient(hunting): get_auth_request con 0x7f78840a2cd0 auth_method 0 2024-04-26T04:18:49.505 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.490+0000 7f78888e6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:49.505 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.490+0000 7f78888e6640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:49.505 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.490+0000 7f78888e6640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:49.505 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.490+0000 7f78890e7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:49.505 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.490+0000 7f78890e7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:49.506 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.490+0000 7f78890e7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:49.506 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.490+0000 7f7883fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:49.506 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.490+0000 7f7883fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:49.506 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.490+0000 7f7883fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:49.506 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.490+0000 7f7883fff640 10 monclient(hunting): handle_auth_done global_id 4259 payload 824 2024-04-26T04:18:49.506 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.490+0000 7f7883fff640 1 -- 172.21.15.63:0/3268446327 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f78840a2cd0 msgr2=0x7f788412ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:49.506 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.490+0000 7f7883fff640 1 --2- 172.21.15.63:0/3268446327 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f78840a2cd0 0x7f788412ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:49.506 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.490+0000 7f7883fff640 1 -- 172.21.15.63:0/3268446327 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f78840a4bd0 msgr2=0x7f788406f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:49.506 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.490+0000 7f7883fff640 1 --2- 172.21.15.63:0/3268446327 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f78840a4bd0 0x7f788406f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:49.506 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.490+0000 7f7883fff640 10 monclient: _finish_hunting 0 2024-04-26T04:18:49.506 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.490+0000 7f7883fff640 1 monclient: found mon.noname-c 2024-04-26T04:18:49.506 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.490+0000 7f7883fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:49.506 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.490+0000 7f7883fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.63:3301/0 2024-04-26T04:18:49.506 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.490+0000 7f7883fff640 1 -- 172.21.15.63:0/3268446327 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7884131f50 con 0x7f78840a47d0 2024-04-26T04:18:49.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.490+0000 7f78890e7640 1 --2- 172.21.15.63:0/3268446327 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f78840a4bd0 0x7f788406f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T04:18:49.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.494+0000 7f7883fff640 1 -- 172.21.15.63:0/3268446327 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f78840a47d0 msgr2=0x7f788412e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:49.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.494+0000 7f7883fff640 1 --2- 172.21.15.63:0/3268446327 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f78840a47d0 0x7f788412e310 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f787001a0a0 tx=0x7f7870019210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:49.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.494+0000 7f78817fa640 1 -- 172.21.15.63:0/3268446327 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78700b0050 con 0x7f78840a47d0 2024-04-26T04:18:49.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.494+0000 7f78817fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:49.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.494+0000 7f78817fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:18:49.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.494+0000 7f78817fa640 10 monclient: dump: 2024-04-26T04:18:49.507 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:49.507 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:49.507 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:49.507 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:49.507 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:49.507 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:49.508 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:49.508 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:49.508 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:49.508 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:49.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.494+0000 7f78817fa640 10 monclient: _finish_auth 0 2024-04-26T04:18:49.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.494+0000 7f78817fa640 10 monclient: _check_auth_tickets 2024-04-26T04:18:49.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.494+0000 7f78817fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:49.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.494+0000 7f788ab4a640 5 monclient: authenticate success, global_id 4259 2024-04-26T04:18:49.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.494+0000 7f78817fa640 1 -- 172.21.15.63:0/3268446327 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78700b1040 con 0x7f78840a47d0 2024-04-26T04:18:49.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.494+0000 7f78817fa640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:49.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.494+0000 7f78817fa640 1 -- 172.21.15.63:0/3268446327 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78700b49e0 con 0x7f78840a47d0 2024-04-26T04:18:49.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.494+0000 7f78817fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:49.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.494+0000 7f78817fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:18:49.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.494+0000 7f78817fa640 10 monclient: dump: 2024-04-26T04:18:49.509 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:49.509 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:49.509 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:49.509 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:49.509 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:49.509 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:49.509 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:49.509 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:49.509 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:49.509 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:49.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.494+0000 7f7882ffd640 4 set_mon_vals no callback set 2024-04-26T04:18:49.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.494+0000 7f788ab4a640 10 monclient: _renew_subs 2024-04-26T04:18:49.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.494+0000 7f788ab4a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:49.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.494+0000 7f788ab4a640 1 -- 172.21.15.63:0/3268446327 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7884072950 con 0x7f78840a47d0 2024-04-26T04:18:49.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.494+0000 7f788ab4a640 10 monclient: _renew_subs 2024-04-26T04:18:49.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.494+0000 7f788ab4a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:49.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.494+0000 7f788ab4a640 1 -- 172.21.15.63:0/3268446327 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f788412e850 con 0x7f78840a47d0 2024-04-26T04:18:49.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.498+0000 7f78817fa640 1 -- 172.21.15.63:0/3268446327 <== mon.2 v2:172.21.15.63:3301/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f78700bc020 con 0x7f78840a47d0 2024-04-26T04:18:49.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.498+0000 7f788ab4a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:49.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.498+0000 7f788ab4a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:49.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.498+0000 7f788ab4a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:49.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.498+0000 7f788ab4a640 1 -- 172.21.15.63:0/3268446327 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7884062e10 con 0x7f78840a47d0 2024-04-26T04:18:49.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.498+0000 7f78817fa640 1 --2- 172.21.15.63:0/3268446327 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f7868035ef0 0x7f78680383b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:49.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.498+0000 7f78817fa640 1 -- 172.21.15.63:0/3268446327 <== mon.2 v2:172.21.15.63:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7f78700e2640 con 0x7f78840a47d0 2024-04-26T04:18:49.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.506+0000 7f78888e6640 1 --2- 172.21.15.63:0/3268446327 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f7868035ef0 0x7f78680383b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:49.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.506+0000 7f78888e6640 10 monclient: get_auth_request con 0x7f7868035ef0 auth_method 0 2024-04-26T04:18:49.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.506+0000 7f78817fa640 1 -- 172.21.15.63:0/3268446327 <== mon.2 v2:172.21.15.63:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f78700160f0 con 0x7f78840a47d0 2024-04-26T04:18:49.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.506+0000 7f78817fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:49.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.506+0000 7f78817fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:18:49.512 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.506+0000 7f62262bd640 10 monclient: get_monmap_and_config 2024-04-26T04:18:49.512 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.506+0000 7f62262bd640 10 monclient: build_initial_monmap 2024-04-26T04:18:49.512 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.506+0000 7f62262bd640 10 monclient: monmap: 2024-04-26T04:18:49.513 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:49.513 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:49.513 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:49.511858+0000 2024-04-26T04:18:49.513 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:49.511858+0000 2024-04-26T04:18:49.513 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:49.513 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:49.513 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:18:49.513 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:18:49.513 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:18:49.513 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:18:49.513 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:18:49.513 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:49.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.506+0000 7f62262bd640 1 Processor -- start 2024-04-26T04:18:49.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.506+0000 7f62262bd640 1 -- start start 2024-04-26T04:18:49.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.506+0000 7f62262bd640 10 monclient: init 2024-04-26T04:18:49.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.506+0000 7f78888e6640 1 --2- 172.21.15.63:0/3268446327 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f7868035ef0 0x7f78680383b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7878009a50 tx=0x7f7878008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:49.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.506+0000 7f62262bd640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:49.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.506+0000 7f62262bd640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-04-26T04:18:49.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.506+0000 7f62262bd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f62200a0bb0 addr v1:172.21.15.186:6789/0 2024-04-26T04:18:49.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.506+0000 7f62262bd640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f622009aff0 0x7f6220138320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:49.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.506+0000 7f62262bd640 10 monclient(hunting): picked mon.noname-c con 0x7f622009aff0 addr v2:172.21.15.186:3300/0 2024-04-26T04:18:49.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.510+0000 7f62262bd640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f622009b400 0x7f622013a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:49.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.510+0000 7f62262bd640 10 monclient(hunting): picked mon.noname-a con 0x7f622009b400 addr v2:172.21.15.63:3300/0 2024-04-26T04:18:49.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.510+0000 7f621f7fe640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f622009aff0 0x7f6220138320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:49.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.510+0000 7f622485a640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f622009b400 0x7f622013a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:49.517 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.510+0000 7f621ffff640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f62200a0bb0 0x7f622009a8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.63:36222/0 (socket says 172.21.15.63:36222) 2024-04-26T04:18:49.517 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.510+0000 7f621ffff640 1 -- 172.21.15.63:0/4136736087 learned_addr learned my addr 172.21.15.63:0/4136736087 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-04-26T04:18:49.517 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.510+0000 7f62262bd640 1 -- 172.21.15.63:0/4136736087 --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62200a0f90 con 0x7f62200a0bb0 2024-04-26T04:18:49.517 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.510+0000 7f62262bd640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:49.517 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.510+0000 7f62262bd640 1 -- 172.21.15.63:0/4136736087 --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7f622009eae0 con 0x7f622009b400 2024-04-26T04:18:49.517 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.510+0000 7f62262bd640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:49.517 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.510+0000 7f62262bd640 1 -- 172.21.15.63:0/4136736087 --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f62200a18f0 con 0x7f622009aff0 2024-04-26T04:18:49.517 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.510+0000 7f62262bd640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:49.517 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.510+0000 7f62262bd640 10 monclient(hunting): authenticate will time out at 961.220918s 2024-04-26T04:18:49.517 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.510+0000 7f621f7fe640 10 monclient(hunting): get_auth_request con 0x7f622009aff0 auth_method 0 2024-04-26T04:18:49.517 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.510+0000 7f621f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:49.517 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.510+0000 7f621f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:49.517 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.510+0000 7f621f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:49.517 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.510+0000 7f622485a640 10 monclient(hunting): get_auth_request con 0x7f622009b400 auth_method 0 2024-04-26T04:18:49.517 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.510+0000 7f622485a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:49.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.510+0000 7f622485a640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:49.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.510+0000 7f622485a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:49.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.510+0000 7f622485a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:49.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.510+0000 7f622485a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:49.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.510+0000 7f622485a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:49.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.510+0000 7f621f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:49.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.510+0000 7f621f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:49.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.510+0000 7f621f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:49.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.510+0000 7f622485a640 10 monclient(hunting): handle_auth_done global_id 4278 payload 274 2024-04-26T04:18:49.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.510+0000 7f622485a640 1 -- 172.21.15.63:0/4136736087 >> v2:172.21.15.186:3300/0 conn(0x7f622009aff0 msgr2=0x7f6220138320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:49.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.510+0000 7f622485a640 1 --2- 172.21.15.63:0/4136736087 >> v2:172.21.15.186:3300/0 conn(0x7f622009aff0 0x7f6220138320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:49.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.510+0000 7f622485a640 1 -- 172.21.15.63:0/4136736087 >> v1:172.21.15.186:6789/0 conn(0x7f62200a0bb0 legacy=0x7f622009a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:49.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.510+0000 7f622485a640 10 monclient: _finish_hunting 0 2024-04-26T04:18:49.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.510+0000 7f622485a640 1 monclient: found mon.noname-a 2024-04-26T04:18:49.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.510+0000 7f622485a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:49.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.510+0000 7f622485a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:18:49.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.510+0000 7f622485a640 1 -- 172.21.15.63:0/4136736087 --> v2:172.21.15.63:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f622009e800 con 0x7f622009b400 2024-04-26T04:18:49.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.514+0000 7f622485a640 1 -- 172.21.15.63:0/4136736087 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f622009b400 msgr2=0x7f622013a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:49.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.514+0000 7f622485a640 1 --2- 172.21.15.63:0/4136736087 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f622009b400 0x7f622013a870 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f622009d740 tx=0x7f621801dad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13b762490a4e19fa server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:49.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.514+0000 7f621effd640 1 -- 172.21.15.63:0/4136736087 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62180a6a30 con 0x7f622009b400 2024-04-26T04:18:49.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.514+0000 7f621effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:49.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.514+0000 7f621effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:18:49.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.514+0000 7f621effd640 10 monclient: dump: 2024-04-26T04:18:49.524 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:49.524 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:49.524 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:49.524 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:49.524 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:49.524 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:49.524 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:49.524 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:49.525 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:49.525 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:49.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.514+0000 7f621effd640 10 monclient: _finish_auth 0 2024-04-26T04:18:49.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.514+0000 7f621effd640 10 monclient: _check_auth_tickets 2024-04-26T04:18:49.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.514+0000 7f621effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:49.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.514+0000 7f621effd640 1 -- 172.21.15.63:0/4136736087 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62180a6bd0 con 0x7f622009b400 2024-04-26T04:18:49.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.514+0000 7f621effd640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:49.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.514+0000 7f621effd640 1 -- 172.21.15.63:0/4136736087 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f621800a0d0 con 0x7f622009b400 2024-04-26T04:18:49.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.514+0000 7f621effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:49.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.514+0000 7f621effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:18:49.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.514+0000 7f621effd640 10 monclient: dump: 2024-04-26T04:18:49.525 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:49.525 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:49.525 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:49.526 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:49.526 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:49.526 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:49.526 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:49.526 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:49.526 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:49.526 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:49.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.514+0000 7f62262bd640 5 monclient: authenticate success, global_id 4278 2024-04-26T04:18:49.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.514+0000 7f62262bd640 10 monclient: get_monmap_and_config success 2024-04-26T04:18:49.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.514+0000 7f62262bd640 4 set_mon_vals no callback set 2024-04-26T04:18:49.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.514+0000 7f62262bd640 10 monclient: shutdown 2024-04-26T04:18:49.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.514+0000 7f62262bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:49.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.514+0000 7f62262bd640 1 -- 172.21.15.63:0/4136736087 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f622009b400 msgr2=0x7f622013a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:49.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.514+0000 7f62262bd640 1 --2- 172.21.15.63:0/4136736087 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f622009b400 0x7f622013a870 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f622009d740 tx=0x7f621801dad0 comp rx=0 tx=0).stop 2024-04-26T04:18:49.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.514+0000 7f62262bd640 1 -- 172.21.15.63:0/4136736087 shutdown_connections 2024-04-26T04:18:49.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.514+0000 7f62262bd640 1 --2- 172.21.15.63:0/4136736087 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f622009b400 0x7f622013a870 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:49.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.514+0000 7f62262bd640 1 --2- 172.21.15.63:0/4136736087 >> v2:172.21.15.186:3300/0 conn(0x7f622009aff0 0x7f6220138320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:49.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.514+0000 7f62262bd640 1 -- 172.21.15.63:0/4136736087 >> 172.21.15.63:0/4136736087 conn(0x7f62200a1e70 msgr2=0x7f62200a2280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:49.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.514+0000 7f62262bd640 1 -- 172.21.15.63:0/4136736087 shutdown_connections 2024-04-26T04:18:49.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.514+0000 7f62262bd640 1 -- 172.21.15.63:0/4136736087 wait complete. 2024-04-26T04:18:49.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.518+0000 7f62262bd640 10 monclient: build_initial_monmap 2024-04-26T04:18:49.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.518+0000 7f62262bd640 10 monclient: monmap: 2024-04-26T04:18:49.527 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:49.527 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:18:49.527 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:18:49.527 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:18:49.527 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:49.528 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:49.528 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:18:49.528 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:18:49.528 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:18:49.528 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:49.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.518+0000 7f62262bd640 1 Processor -- start 2024-04-26T04:18:49.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.518+0000 7f62262bd640 1 -- start start 2024-04-26T04:18:49.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.518+0000 7f62262bd640 10 monclient: init 2024-04-26T04:18:49.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.518+0000 7f62262bd640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:49.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.518+0000 7f62262bd640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T04:18:49.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.518+0000 7f62262bd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f622009aff0 0x7f6220144fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:49.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.518+0000 7f621ffff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f622009aff0 0x7f6220144fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:49.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.518+0000 7f621ffff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f622009aff0 0x7f6220144fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:39198/0 (socket says 172.21.15.63:39198) 2024-04-26T04:18:49.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.518+0000 7f621ffff640 1 -- 172.21.15.63:0/3835818408 learned_addr learned my addr 172.21.15.63:0/3835818408 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:18:49.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.518+0000 7f62262bd640 10 monclient(hunting): picked mon.noname-a con 0x7f622009aff0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:18:49.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.518+0000 7f62262bd640 1 --2- 172.21.15.63:0/3835818408 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f622009b400 0x7f6220140c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:49.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.518+0000 7f62262bd640 10 monclient(hunting): picked mon.noname-c con 0x7f622009b400 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:49.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.518+0000 7f62262bd640 1 --2- 172.21.15.63:0/3835818408 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f62200a0bb0 0x7f62201411c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:49.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.518+0000 7f62262bd640 10 monclient(hunting): picked mon.noname-b con 0x7f62200a0bb0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:18:49.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.518+0000 7f62262bd640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:49.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.518+0000 7f62262bd640 1 -- 172.21.15.63:0/3835818408 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f622013d750 con 0x7f622009aff0 2024-04-26T04:18:49.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.518+0000 7f62262bd640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:49.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.518+0000 7f62262bd640 1 -- 172.21.15.63:0/3835818408 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f622013d5d0 con 0x7f62200a0bb0 2024-04-26T04:18:49.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.518+0000 7f62262bd640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:49.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.518+0000 7f62262bd640 1 -- 172.21.15.63:0/3835818408 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f622013d8d0 con 0x7f622009b400 2024-04-26T04:18:49.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.518+0000 7f621f7fe640 1 --2- 172.21.15.63:0/3835818408 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f622009b400 0x7f6220140c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:49.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.518+0000 7f62262bd640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:49.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.518+0000 7f622485a640 1 --2- 172.21.15.63:0/3835818408 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f62200a0bb0 0x7f62201411c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:49.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.518+0000 7f62262bd640 10 monclient(hunting): authenticate will time out at 961.230322s 2024-04-26T04:18:49.531 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.518+0000 7f621ffff640 10 monclient(hunting): get_auth_request con 0x7f622009aff0 auth_method 0 2024-04-26T04:18:49.531 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.518+0000 7f621ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:49.531 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.518+0000 7f621ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:49.531 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.518+0000 7f621ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:49.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.518+0000 7f621f7fe640 10 monclient(hunting): get_auth_request con 0x7f622009b400 auth_method 0 2024-04-26T04:18:49.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.518+0000 7f621f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:49.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.518+0000 7f621f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:49.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.518+0000 7f621f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:49.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.518+0000 7f622485a640 10 monclient(hunting): get_auth_request con 0x7f62200a0bb0 auth_method 0 2024-04-26T04:18:49.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.518+0000 7f622485a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:49.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.518+0000 7f622485a640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:49.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.518+0000 7f622485a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:49.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.518+0000 7f621f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:49.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.518+0000 7f621f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:49.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.518+0000 7f621f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:49.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.518+0000 7f622485a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:49.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.518+0000 7f622485a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:49.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.518+0000 7f622485a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:49.533 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.518+0000 7f621ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:49.533 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.518+0000 7f621ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:49.533 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.518+0000 7f621ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:49.533 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.518+0000 7f622485a640 10 monclient(hunting): handle_auth_done global_id 4276 payload 824 2024-04-26T04:18:49.533 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.518+0000 7f622485a640 1 -- 172.21.15.63:0/3835818408 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f622009b400 msgr2=0x7f6220140c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:49.533 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.518+0000 7f622485a640 1 --2- 172.21.15.63:0/3835818408 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f622009b400 0x7f6220140c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:49.533 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.518+0000 7f622485a640 1 -- 172.21.15.63:0/3835818408 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f622009aff0 msgr2=0x7f6220144fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:49.533 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.518+0000 7f622485a640 1 --2- 172.21.15.63:0/3835818408 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f622009aff0 0x7f6220144fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:49.533 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.518+0000 7f622485a640 10 monclient: _finish_hunting 0 2024-04-26T04:18:49.533 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.518+0000 7f622485a640 1 monclient: found mon.noname-b 2024-04-26T04:18:49.533 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.518+0000 7f622485a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:49.533 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.518+0000 7f622485a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.186:3300/0 2024-04-26T04:18:49.533 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.518+0000 7f622485a640 1 -- 172.21.15.63:0/3835818408 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62200a33f0 con 0x7f62200a0bb0 2024-04-26T04:18:49.534 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.518+0000 7f621f7fe640 1 --2- 172.21.15.63:0/3835818408 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f622009b400 0x7f6220140c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T04:18:49.534 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.522+0000 7f622485a640 1 -- 172.21.15.63:0/3835818408 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f62200a0bb0 msgr2=0x7f62201411c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:49.534 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.522+0000 7f622485a640 1 --2- 172.21.15.63:0/3835818408 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f62200a0bb0 0x7f62201411c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f62200a11a0 tx=0x7f621801a870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:49.534 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.522+0000 7f621cff9640 1 -- 172.21.15.63:0/3835818408 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62180bc030 con 0x7f62200a0bb0 2024-04-26T04:18:49.534 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.522+0000 7f621cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:49.534 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.522+0000 7f621cff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T04:18:49.534 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.522+0000 7f621cff9640 10 monclient: dump: 2024-04-26T04:18:49.534 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:49.534 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:49.534 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:49.534 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:49.534 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:49.534 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:49.534 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:49.535 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:49.535 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:49.535 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:49.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.522+0000 7f621cff9640 10 monclient: _finish_auth 0 2024-04-26T04:18:49.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.522+0000 7f621cff9640 10 monclient: _check_auth_tickets 2024-04-26T04:18:49.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.522+0000 7f621cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:49.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.522+0000 7f62262bd640 5 monclient: authenticate success, global_id 4276 2024-04-26T04:18:49.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.522+0000 7f621cff9640 1 -- 172.21.15.63:0/3835818408 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f621800a4f0 con 0x7f62200a0bb0 2024-04-26T04:18:49.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.522+0000 7f62262bd640 10 monclient: _renew_subs 2024-04-26T04:18:49.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.522+0000 7f62262bd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-04-26T04:18:49.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.522+0000 7f62262bd640 1 -- 172.21.15.63:0/3835818408 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f622009dc10 con 0x7f62200a0bb0 2024-04-26T04:18:49.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.522+0000 7f621cff9640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:49.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.522+0000 7f621cff9640 1 -- 172.21.15.63:0/3835818408 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f621800a8c0 con 0x7f62200a0bb0 2024-04-26T04:18:49.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.522+0000 7f621cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:49.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.522+0000 7f621cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T04:18:49.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.522+0000 7f621cff9640 10 monclient: dump: 2024-04-26T04:18:49.536 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:49.536 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:49.536 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:49.536 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:49.536 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:49.536 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:49.536 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:49.536 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:49.536 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:49.536 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:49.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.522+0000 7f62262bd640 10 monclient: _renew_subs 2024-04-26T04:18:49.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.522+0000 7f62262bd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-04-26T04:18:49.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.522+0000 7f62262bd640 1 -- 172.21.15.63:0/3835818408 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6220144b90 con 0x7f62200a0bb0 2024-04-26T04:18:49.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.522+0000 7f621e7fc640 4 set_mon_vals no callback set 2024-04-26T04:18:49.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.522+0000 7f62262bd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:49.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.522+0000 7f62262bd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:49.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.522+0000 7f62262bd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-04-26T04:18:49.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.522+0000 7f62262bd640 1 -- 172.21.15.63:0/3835818408 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6220144320 con 0x7f62200a0bb0 2024-04-26T04:18:49.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.522+0000 7f622485a640 1 --2- 172.21.15.63:0/3835818408 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f62200a0bb0 0x7f62201411c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f62200a11a0 tx=0x7f621801a870 comp rx=0 tx=0).queue_received will delay after 1.71411e+09 on 0x7f62180e39b0 osd_map(15..15 src has 1..15) 2024-04-26T04:18:49.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.522+0000 7f621cff9640 1 -- 172.21.15.63:0/3835818408 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f62180bf070 con 0x7f62200a0bb0 2024-04-26T04:18:49.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.526+0000 7f621cff9640 1 --2- 172.21.15.63:0/3835818408 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f620c035ec0 0x7f620c038380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:49.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.526+0000 7f621ffff640 1 --2- 172.21.15.63:0/3835818408 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f620c035ec0 0x7f620c038380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:49.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.526+0000 7f621ffff640 10 monclient: get_auth_request con 0x7f620c035ec0 auth_method 0 2024-04-26T04:18:49.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.526+0000 7f621ffff640 1 --2- 172.21.15.63:0/3835818408 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f620c035ec0 0x7f620c038380 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6210007d60 tx=0x7f6210007cb0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:49.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.538+0000 7ffb354b0640 10 monclient: get_monmap_and_config 2024-04-26T04:18:49.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.538+0000 7ffb354b0640 10 monclient: build_initial_monmap 2024-04-26T04:18:49.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.538+0000 7ffb354b0640 10 monclient: monmap: 2024-04-26T04:18:49.545 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:49.545 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:49.545 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:49.542874+0000 2024-04-26T04:18:49.545 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:49.542874+0000 2024-04-26T04:18:49.545 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:49.545 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:49.546 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:18:49.546 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:18:49.546 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:18:49.546 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:18:49.546 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:18:49.546 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:49.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.538+0000 7ffb354b0640 1 Processor -- start 2024-04-26T04:18:49.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.538+0000 7ffb354b0640 1 -- start start 2024-04-26T04:18:49.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.538+0000 7ffb354b0640 10 monclient: init 2024-04-26T04:18:49.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.538+0000 7f621cff9640 1 -- 172.21.15.63:0/3835818408 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7f62180e39b0 con 0x7f62200a0bb0 2024-04-26T04:18:49.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.538+0000 7ffb354b0640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:49.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.538+0000 7ffb354b0640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-04-26T04:18:49.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.538+0000 7ffb354b0640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ffb300a2cd0 0x7ffb300a30b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:49.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.538+0000 7ffb354b0640 10 monclient(hunting): picked mon.noname-c con 0x7ffb300a2cd0 addr v2:172.21.15.186:3300/0 2024-04-26T04:18:49.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.538+0000 7ffb354b0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ffb300a47d0 addr v1:172.21.15.63:6789/0 2024-04-26T04:18:49.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.538+0000 7ffb354b0640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ffb300a4c00 0x7ffb301373f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:49.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.538+0000 7ffb354b0640 10 monclient(hunting): picked mon.noname-b con 0x7ffb300a4c00 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:49.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.538+0000 7ffb354b0640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb300a0fb0 con 0x7ffb300a47d0 2024-04-26T04:18:49.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.538+0000 7ffb354b0640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:49.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.538+0000 7ffb354b0640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7ffb300a35f0 con 0x7ffb300a2cd0 2024-04-26T04:18:49.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.538+0000 7ffb354b0640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:49.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.538+0000 7ffb354b0640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7ffb300a5380 con 0x7ffb300a4c00 2024-04-26T04:18:49.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.538+0000 7ffb354b0640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:49.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.538+0000 7ffb354b0640 10 monclient(hunting): authenticate will time out at 961.251402s 2024-04-26T04:18:49.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.538+0000 7ffb2e7fc640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7ffb300a47d0 0x7ffb30136ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:32782/0 (socket says 172.21.15.63:32782) 2024-04-26T04:18:49.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.538+0000 7ffb2e7fc640 1 -- 172.21.15.63:0/4190097023 learned_addr learned my addr 172.21.15.63:0/4190097023 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-04-26T04:18:49.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.538+0000 7ffb2f7fe640 1 --2- 172.21.15.63:0/4190097023 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ffb300a4c00 0x7ffb301373f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:49.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.538+0000 7ffb2f7fe640 10 monclient(hunting): get_auth_request con 0x7ffb300a4c00 auth_method 0 2024-04-26T04:18:49.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.538+0000 7ffb2f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:49.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.538+0000 7ffb2f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:49.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.538+0000 7ffb2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:49.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.538+0000 7ffb2effd640 1 --2- 172.21.15.63:0/4190097023 >> v2:172.21.15.186:3300/0 conn(0x7ffb300a2cd0 0x7ffb300a30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:49.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.538+0000 7ffb2effd640 10 monclient(hunting): get_auth_request con 0x7ffb300a2cd0 auth_method 0 2024-04-26T04:18:49.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.538+0000 7ffb2effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:49.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.538+0000 7ffb2effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:49.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.538+0000 7ffb2effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:49.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.538+0000 7ffb2f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:49.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.538+0000 7ffb2f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:49.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.538+0000 7ffb2f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:49.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.538+0000 7ffb2effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:49.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.538+0000 7ffb2effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:49.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.538+0000 7ffb2effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:49.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.538+0000 7ffb2f7fe640 10 monclient(hunting): handle_auth_done global_id 4265 payload 274 2024-04-26T04:18:49.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.538+0000 7ffb2f7fe640 1 -- 172.21.15.63:0/4190097023 >> v2:172.21.15.186:3300/0 conn(0x7ffb300a2cd0 msgr2=0x7ffb300a30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:49.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.538+0000 7ffb2f7fe640 1 --2- 172.21.15.63:0/4190097023 >> v2:172.21.15.186:3300/0 conn(0x7ffb300a2cd0 0x7ffb300a30b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:49.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.538+0000 7ffb2f7fe640 1 -- 172.21.15.63:0/4190097023 >> v1:172.21.15.63:6789/0 conn(0x7ffb300a47d0 legacy=0x7ffb30136ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:49.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.542+0000 7ffb2f7fe640 10 monclient: _finish_hunting 0 2024-04-26T04:18:49.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.542+0000 7ffb2f7fe640 1 monclient: found mon.noname-b 2024-04-26T04:18:49.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.542+0000 7ffb2f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:49.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.542+0000 7ffb2f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.63:3301/0 2024-04-26T04:18:49.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.542+0000 7ffb2f7fe640 1 -- 172.21.15.63:0/4190097023 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb300a1290 con 0x7ffb300a4c00 2024-04-26T04:18:49.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ff4aca6d640 10 monclient: get_monmap_and_config 2024-04-26T04:18:49.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ff4aca6d640 10 monclient: build_initial_monmap 2024-04-26T04:18:49.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ff4aca6d640 10 monclient: monmap: 2024-04-26T04:18:49.554 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:49.554 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:49.554 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:49.550279+0000 2024-04-26T04:18:49.554 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:49.550279+0000 2024-04-26T04:18:49.554 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:49.554 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:49.554 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:18:49.554 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:18:49.554 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:18:49.554 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:18:49.554 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:18:49.554 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:49.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ff4aca6d640 1 Processor -- start 2024-04-26T04:18:49.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ff4aca6d640 1 -- start start 2024-04-26T04:18:49.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ff4aca6d640 10 monclient: init 2024-04-26T04:18:49.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ffb2f7fe640 1 -- 172.21.15.63:0/4190097023 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ffb300a4c00 msgr2=0x7ffb301373f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:49.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ffb2f7fe640 1 --2- 172.21.15.63:0/4190097023 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ffb300a4c00 0x7ffb301373f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ffb28009d90 tx=0x7ffb2801c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d5ec6855cfe0fa5 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:49.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ff4aca6d640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:49.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ff4aca6d640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-04-26T04:18:49.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ff4aca6d640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7ff4a8134b90 0x7ff4a809d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:49.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ff4aca6d640 10 monclient(hunting): picked mon.noname-a con 0x7ff4a8134b90 addr v2:172.21.15.63:3300/0 2024-04-26T04:18:49.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ff4aca6d640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff4a8134f70 0x7ff4a8135370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:49.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ff4aca6d640 10 monclient(hunting): picked mon.noname-b con 0x7ff4a8134f70 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:49.557 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ff4aca6d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff4a8133b10 addr v1:172.21.15.186:6789/0 2024-04-26T04:18:49.557 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ff4aca6d640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4a80a3630 con 0x7ff4a8133b10 2024-04-26T04:18:49.557 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ff4aca6d640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:49.557 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ffb2dffb640 1 -- 172.21.15.63:0/4190097023 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb280a6710 con 0x7ffb300a4c00 2024-04-26T04:18:49.557 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ffb2dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:49.557 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ff4a5d9b640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff4a8134f70 0x7ff4a8135370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:49.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ff4a5d9b640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff4a8134f70 0x7ff4a8135370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:36846/0 (socket says 172.21.15.63:36846) 2024-04-26T04:18:49.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ff4a5d9b640 1 -- 172.21.15.63:0/2259883209 learned_addr learned my addr 172.21.15.63:0/2259883209 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:18:49.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ffb2dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T04:18:49.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ff4aca6d640 1 -- 172.21.15.63:0/2259883209 --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7ff4a80a3840 con 0x7ff4a8134b90 2024-04-26T04:18:49.559 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ffb2dffb640 10 monclient: dump: 2024-04-26T04:18:49.559 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:49.559 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:49.559 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:49.559 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:49.559 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:49.559 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:49.559 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:49.559 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:49.559 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:49.559 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:49.559 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ff4aca6d640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:49.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ffb2dffb640 10 monclient: _finish_auth 0 2024-04-26T04:18:49.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ff4aca6d640 1 -- 172.21.15.63:0/2259883209 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7ff4a809c350 con 0x7ff4a8134f70 2024-04-26T04:18:49.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ff4a659c640 1 --2- 172.21.15.63:0/2259883209 >> v2:172.21.15.63:3300/0 conn(0x7ff4a8134b90 0x7ff4a809d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:49.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ff4aca6d640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:49.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ffb2dffb640 10 monclient: _check_auth_tickets 2024-04-26T04:18:49.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ffb2dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:49.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ff4aca6d640 10 monclient(hunting): authenticate will time out at 961.257926s 2024-04-26T04:18:49.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ffb2dffb640 1 -- 172.21.15.63:0/4190097023 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb280a68b0 con 0x7ffb300a4c00 2024-04-26T04:18:49.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ffb354b0640 5 monclient: authenticate success, global_id 4265 2024-04-26T04:18:49.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ffb354b0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T04:18:49.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ff4a5d9b640 10 monclient(hunting): get_auth_request con 0x7ff4a8134f70 auth_method 0 2024-04-26T04:18:49.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ff4a5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:49.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ff4a5d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:49.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ff4a5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:49.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ffb2dffb640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:49.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ff4a659c640 10 monclient(hunting): get_auth_request con 0x7ff4a8134b90 auth_method 0 2024-04-26T04:18:49.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ff4a659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:49.563 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ff4a659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:49.563 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ff4a659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:49.563 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ffb2dffb640 1 -- 172.21.15.63:0/4190097023 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb280a6c80 con 0x7ffb300a4c00 2024-04-26T04:18:49.563 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ffb354b0640 10 monclient: get_monmap_and_config success 2024-04-26T04:18:49.563 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ffb354b0640 4 set_mon_vals no callback set 2024-04-26T04:18:49.563 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ffb2dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:49.564 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ffb2dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:18:49.564 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ff4a5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:49.564 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ff4a5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:49.564 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ffb2dffb640 10 monclient: dump: 2024-04-26T04:18:49.564 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:49.564 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:49.565 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:49.565 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:49.565 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:49.565 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:49.565 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:49.565 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:49.565 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:49.565 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:49.566 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ff4a5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:49.566 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ff4a5d9b640 10 monclient(hunting): handle_auth_done global_id 4268 payload 274 2024-04-26T04:18:49.566 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ff4a5d9b640 1 -- 172.21.15.63:0/2259883209 >> v2:172.21.15.63:3300/0 conn(0x7ff4a8134b90 msgr2=0x7ff4a809d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:49.566 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ff4a5d9b640 1 --2- 172.21.15.63:0/2259883209 >> v2:172.21.15.63:3300/0 conn(0x7ff4a8134b90 0x7ff4a809d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:49.566 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ff4a5d9b640 1 -- 172.21.15.63:0/2259883209 >> v1:172.21.15.186:6789/0 conn(0x7ff4a8133b10 legacy=0x7ff4a8133f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:49.566 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ff4a5d9b640 10 monclient: _finish_hunting 0 2024-04-26T04:18:49.566 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ff4a5d9b640 1 monclient: found mon.noname-b 2024-04-26T04:18:49.566 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ff4a5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:49.566 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ff4a5d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.63:3301/0 2024-04-26T04:18:49.566 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ff4a5d9b640 1 -- 172.21.15.63:0/2259883209 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4a809ba50 con 0x7ff4a8134f70 2024-04-26T04:18:49.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ffb354b0640 10 monclient: shutdown 2024-04-26T04:18:49.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ffb354b0640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:49.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ffb354b0640 1 -- 172.21.15.63:0/4190097023 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ffb300a4c00 msgr2=0x7ffb301373f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:49.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ffb354b0640 1 --2- 172.21.15.63:0/4190097023 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ffb300a4c00 0x7ffb301373f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ffb28009d90 tx=0x7ffb2801c880 comp rx=0 tx=0).stop 2024-04-26T04:18:49.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ffb354b0640 1 -- 172.21.15.63:0/4190097023 shutdown_connections 2024-04-26T04:18:49.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ffb354b0640 1 --2- 172.21.15.63:0/4190097023 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ffb300a4c00 0x7ffb301373f0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:49.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ffb354b0640 1 --2- 172.21.15.63:0/4190097023 >> v2:172.21.15.186:3300/0 conn(0x7ffb300a2cd0 0x7ffb300a30b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:49.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ffb354b0640 1 -- 172.21.15.63:0/4190097023 >> 172.21.15.63:0/4190097023 conn(0x7ffb300a5f40 msgr2=0x7ffb300a6320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:49.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ffb354b0640 1 -- 172.21.15.63:0/4190097023 shutdown_connections 2024-04-26T04:18:49.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.546+0000 7ffb354b0640 1 -- 172.21.15.63:0/4190097023 wait complete. 2024-04-26T04:18:49.571 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ffb354b0640 10 monclient: build_initial_monmap 2024-04-26T04:18:49.571 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ffb354b0640 10 monclient: monmap: 2024-04-26T04:18:49.571 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:49.571 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:18:49.571 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:18:49.571 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:18:49.571 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:49.571 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:49.571 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:18:49.571 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:18:49.571 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:18:49.571 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:49.571 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ffb354b0640 1 Processor -- start 2024-04-26T04:18:49.571 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ffb354b0640 1 -- start start 2024-04-26T04:18:49.572 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ffb354b0640 10 monclient: init 2024-04-26T04:18:49.572 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ffb354b0640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:49.572 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ffb354b0640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T04:18:49.572 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ffb354b0640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ffb300a2cd0 0x7ffb301ceed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:49.572 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ffb354b0640 10 monclient(hunting): picked mon.noname-c con 0x7ffb300a2cd0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:49.572 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ffb354b0640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffb300a47d0 0x7ffb301cf600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:49.572 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ffb2effd640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ffb300a2cd0 0x7ffb301ceed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:49.572 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ffb2effd640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ffb300a2cd0 0x7ffb301ceed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:36860/0 (socket says 172.21.15.63:36860) 2024-04-26T04:18:49.572 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ffb2e7fc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffb300a47d0 0x7ffb301cf600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:49.572 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ffb2e7fc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffb300a47d0 0x7ffb301cf600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:39226/0 (socket says 172.21.15.63:39226) 2024-04-26T04:18:49.572 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ffb354b0640 10 monclient(hunting): picked mon.noname-a con 0x7ffb300a47d0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:18:49.572 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ffb2effd640 1 -- 172.21.15.63:0/84734618 learned_addr learned my addr 172.21.15.63:0/84734618 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:18:49.572 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ffb354b0640 1 --2- 172.21.15.63:0/84734618 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffb300a4c00 0x7ffb3013ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:49.572 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ffb354b0640 10 monclient(hunting): picked mon.noname-b con 0x7ffb300a4c00 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:18:49.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ffb354b0640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:49.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ffb354b0640 1 -- 172.21.15.63:0/84734618 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ffb30141a50 con 0x7ffb300a47d0 2024-04-26T04:18:49.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ffb354b0640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:49.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ffb354b0640 1 -- 172.21.15.63:0/84734618 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ffb30137b40 con 0x7ffb300a4c00 2024-04-26T04:18:49.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ffb354b0640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:49.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ffb354b0640 1 -- 172.21.15.63:0/84734618 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7ffb30141bd0 con 0x7ffb300a2cd0 2024-04-26T04:18:49.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ffb354b0640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:49.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ffb354b0640 10 monclient(hunting): authenticate will time out at 961.261942s 2024-04-26T04:18:49.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ffb2effd640 10 monclient(hunting): get_auth_request con 0x7ffb300a2cd0 auth_method 0 2024-04-26T04:18:49.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ffb2effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:49.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ffb2effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:49.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ffb2effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:49.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ffb2e7fc640 10 monclient(hunting): get_auth_request con 0x7ffb300a47d0 auth_method 0 2024-04-26T04:18:49.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ffb2e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:49.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ffb2e7fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:49.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ffb2e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:49.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ffb2effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:49.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ffb2effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:49.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ffb2effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:49.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ffb2e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:49.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ffb2e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:49.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ffb2e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:49.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ffb2effd640 10 monclient(hunting): handle_auth_done global_id 4271 payload 824 2024-04-26T04:18:49.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ffb2effd640 1 -- 172.21.15.63:0/84734618 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffb300a4c00 msgr2=0x7ffb3013ba80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-26T04:18:49.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ffb2effd640 1 --2- 172.21.15.63:0/84734618 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffb300a4c00 0x7ffb3013ba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:49.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ffb2effd640 1 -- 172.21.15.63:0/84734618 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffb300a47d0 msgr2=0x7ffb301cf600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:49.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ffb2effd640 1 --2- 172.21.15.63:0/84734618 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffb300a47d0 0x7ffb301cf600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:49.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ffb2effd640 10 monclient: _finish_hunting 0 2024-04-26T04:18:49.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ffb2effd640 1 monclient: found mon.noname-c 2024-04-26T04:18:49.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ffb2effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:49.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ffb2effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.63:3301/0 2024-04-26T04:18:49.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ffb2effd640 1 -- 172.21.15.63:0/84734618 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb30072a90 con 0x7ffb300a2cd0 2024-04-26T04:18:49.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ff4a5d9b640 1 -- 172.21.15.63:0/2259883209 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff4a8134f70 msgr2=0x7ff4a8135370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:49.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ff4a5d9b640 1 --2- 172.21.15.63:0/2259883209 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff4a8134f70 0x7ff4a8135370 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff49c009870 tx=0x7ff49c01c400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4faf3c63b48fb798 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:49.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ff4a559a640 1 -- 172.21.15.63:0/2259883209 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff49c0b0070 con 0x7ff4a8134f70 2024-04-26T04:18:49.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ff4a559a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:49.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ff4a559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T04:18:49.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ff4a559a640 10 monclient: dump: 2024-04-26T04:18:49.577 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:49.577 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:49.577 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:49.577 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:49.577 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:49.577 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:49.577 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:49.577 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:49.578 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:49.578 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:49.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ff4a559a640 10 monclient: _finish_auth 0 2024-04-26T04:18:49.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ff4a559a640 10 monclient: _check_auth_tickets 2024-04-26T04:18:49.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ff4a559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:49.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ff4a559a640 1 -- 172.21.15.63:0/2259883209 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff49c0ab600 con 0x7ff4a8134f70 2024-04-26T04:18:49.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ff4a559a640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:49.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ff4aca6d640 5 monclient: authenticate success, global_id 4268 2024-04-26T04:18:49.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ff4a559a640 1 -- 172.21.15.63:0/2259883209 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff49c0ab9d0 con 0x7ff4a8134f70 2024-04-26T04:18:49.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ff4a559a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:49.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ff4a559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:18:49.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ff4a559a640 10 monclient: dump: 2024-04-26T04:18:49.578 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:49.579 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:49.579 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:49.579 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:49.579 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:49.579 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:49.579 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:49.579 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:49.579 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:49.579 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:49.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ff4aca6d640 10 monclient: get_monmap_and_config success 2024-04-26T04:18:49.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ff4aca6d640 4 set_mon_vals no callback set 2024-04-26T04:18:49.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ff4aca6d640 10 monclient: shutdown 2024-04-26T04:18:49.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ff4aca6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:49.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ff4aca6d640 1 -- 172.21.15.63:0/2259883209 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff4a8134f70 msgr2=0x7ff4a8135370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:49.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ff4aca6d640 1 --2- 172.21.15.63:0/2259883209 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff4a8134f70 0x7ff4a8135370 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff49c009870 tx=0x7ff49c01c400 comp rx=0 tx=0).stop 2024-04-26T04:18:49.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ff4aca6d640 1 -- 172.21.15.63:0/2259883209 shutdown_connections 2024-04-26T04:18:49.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ff4aca6d640 1 --2- 172.21.15.63:0/2259883209 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff4a8134f70 0x7ff4a8135370 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:49.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ff4aca6d640 1 --2- 172.21.15.63:0/2259883209 >> v2:172.21.15.63:3300/0 conn(0x7ff4a8134b90 0x7ff4a809d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:49.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ff4aca6d640 1 -- 172.21.15.63:0/2259883209 >> 172.21.15.63:0/2259883209 conn(0x7ff4a80a7360 msgr2=0x7ff4a80a7770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:49.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ff4aca6d640 1 -- 172.21.15.63:0/2259883209 shutdown_connections 2024-04-26T04:18:49.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.550+0000 7ff4aca6d640 1 -- 172.21.15.63:0/2259883209 wait complete. 2024-04-26T04:18:49.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ff4aca6d640 10 monclient: build_initial_monmap 2024-04-26T04:18:49.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ff4aca6d640 10 monclient: monmap: 2024-04-26T04:18:49.581 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:49.581 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:18:49.581 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:18:49.581 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:18:49.581 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:49.581 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:49.581 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:18:49.581 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:18:49.581 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:18:49.581 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:49.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ff4aca6d640 1 Processor -- start 2024-04-26T04:18:49.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ffb2effd640 1 -- 172.21.15.63:0/84734618 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ffb300a2cd0 msgr2=0x7ffb301ceed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:49.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ffb2effd640 1 --2- 172.21.15.63:0/84734618 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ffb300a2cd0 0x7ffb301ceed0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ffb2000e8d0 tx=0x7ffb2000ed90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:49.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ff4aca6d640 1 -- start start 2024-04-26T04:18:49.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ff4aca6d640 10 monclient: init 2024-04-26T04:18:49.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ffb177fe640 1 -- 172.21.15.63:0/84734618 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb20017070 con 0x7ffb300a2cd0 2024-04-26T04:18:49.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ffb177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:49.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ffb177fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:18:49.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ffb177fe640 10 monclient: dump: 2024-04-26T04:18:49.584 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:49.584 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:49.584 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:49.584 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:49.584 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:49.584 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:49.584 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:49.584 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:49.584 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:49.584 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:49.585 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ff4aca6d640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:49.585 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ff4aca6d640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T04:18:49.585 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ff4aca6d640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff4a8133b10 0x7ff4a81ccd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:49.585 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ff4aca6d640 10 monclient(hunting): picked mon.noname-c con 0x7ff4a8133b10 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:49.585 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ff4aca6d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff4a8134b90 0x7ff4a81cd4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:49.585 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ff4aca6d640 10 monclient(hunting): picked mon.noname-b con 0x7ff4a8134b90 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:18:49.585 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ffb177fe640 10 monclient: _finish_auth 0 2024-04-26T04:18:49.585 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ffb177fe640 10 monclient: _check_auth_tickets 2024-04-26T04:18:49.585 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ffb177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:49.586 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ffb177fe640 1 -- 172.21.15.63:0/84734618 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb2000cb50 con 0x7ffb300a2cd0 2024-04-26T04:18:49.586 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ffb177fe640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:49.586 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ffb177fe640 1 -- 172.21.15.63:0/84734618 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb20013470 con 0x7ffb300a2cd0 2024-04-26T04:18:49.586 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ffb354b0640 5 monclient: authenticate success, global_id 4271 2024-04-26T04:18:49.586 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ffb354b0640 10 monclient: _renew_subs 2024-04-26T04:18:49.586 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ff4aca6d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff4a8134f70 0x7ff4a8138850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:49.586 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ff4aca6d640 10 monclient(hunting): picked mon.noname-a con 0x7ff4a8134f70 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:18:49.586 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ff4aca6d640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:49.586 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ff4aca6d640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff4a813fc90 con 0x7ff4a8134f70 2024-04-26T04:18:49.587 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ff4aca6d640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:49.587 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ff4aca6d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff4a813fb10 con 0x7ff4a8134b90 2024-04-26T04:18:49.587 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ff4aca6d640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:49.587 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ff4aca6d640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7ff4a813fe10 con 0x7ff4a8133b10 2024-04-26T04:18:49.587 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ff4aca6d640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:49.587 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ff4aca6d640 10 monclient(hunting): authenticate will time out at 961.265037s 2024-04-26T04:18:49.588 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ffb354b0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:49.588 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ffb354b0640 1 -- 172.21.15.63:0/84734618 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb3013d720 con 0x7ffb300a2cd0 2024-04-26T04:18:49.588 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ffb2d7fa640 4 set_mon_vals no callback set 2024-04-26T04:18:49.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ff4a6d9d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff4a8134f70 0x7ff4a8138850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:49.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ff4a6d9d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff4a8134f70 0x7ff4a8138850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:39230/0 (socket says 172.21.15.63:39230) 2024-04-26T04:18:49.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ffb177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:49.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ff4a6d9d640 1 -- 172.21.15.63:0/2035668620 learned_addr learned my addr 172.21.15.63:0/2035668620 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:18:49.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ff4a6d9d640 10 monclient(hunting): get_auth_request con 0x7ff4a8134f70 auth_method 0 2024-04-26T04:18:49.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ff4a6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:49.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ff4a6d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:49.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ff4a6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:49.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ffb177fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:18:49.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ffb177fe640 10 monclient: dump: 2024-04-26T04:18:49.590 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:49.590 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:49.590 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:49.590 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:49.590 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:49.590 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:49.590 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:49.590 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:49.591 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:49.591 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:49.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ff4a659c640 1 --2- 172.21.15.63:0/2035668620 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff4a8133b10 0x7ff4a81ccd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:49.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ff4a659c640 10 monclient(hunting): get_auth_request con 0x7ff4a8133b10 auth_method 0 2024-04-26T04:18:49.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ff4a659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:49.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ff4a659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:49.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ff4a659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:49.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ff4a6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:49.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ff4a6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:49.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ff4a6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:49.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ff4a659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:49.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ff4a659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:49.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ff4a659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:49.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ff4a6d9d640 10 monclient(hunting): handle_auth_done global_id 4290 payload 824 2024-04-26T04:18:49.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ff4a5d9b640 1 --2- 172.21.15.63:0/2035668620 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff4a8134b90 0x7ff4a81cd4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:49.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ff4a6d9d640 1 -- 172.21.15.63:0/2035668620 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff4a8133b10 msgr2=0x7ff4a81ccd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:49.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ff4a6d9d640 1 --2- 172.21.15.63:0/2035668620 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff4a8133b10 0x7ff4a81ccd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:49.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ff4a6d9d640 1 -- 172.21.15.63:0/2035668620 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff4a8134b90 msgr2=0x7ff4a81cd4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:49.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ff4a6d9d640 1 --2- 172.21.15.63:0/2035668620 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff4a8134b90 0x7ff4a81cd4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:49.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ff4a6d9d640 10 monclient: _finish_hunting 0 2024-04-26T04:18:49.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ff4a6d9d640 1 monclient: found mon.noname-a 2024-04-26T04:18:49.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ff4a6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:49.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ff4a6d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:18:49.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ff4a6d9d640 1 -- 172.21.15.63:0/2035668620 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4a813c7a0 con 0x7ff4a8134f70 2024-04-26T04:18:49.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ff4a5d9b640 10 monclient: get_auth_request con 0x7ff4a8134b90 auth_method 0 2024-04-26T04:18:49.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ff4a5d9b640 1 --2- 172.21.15.63:0/2035668620 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff4a8134b90 0x7ff4a81cd4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T04:18:49.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ffb354b0640 10 monclient: _renew_subs 2024-04-26T04:18:49.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ffb354b0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:49.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ffb354b0640 1 -- 172.21.15.63:0/84734618 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb30120580 con 0x7ffb300a2cd0 2024-04-26T04:18:49.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ffb354b0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:49.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ffb354b0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:49.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ffb354b0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:49.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.558+0000 7ff4a6d9d640 1 -- 172.21.15.63:0/2035668620 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff4a8134f70 msgr2=0x7ff4a8138850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:49.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.558+0000 7ff4a6d9d640 1 --2- 172.21.15.63:0/2035668620 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff4a8134f70 0x7ff4a8138850 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff4a000c780 tx=0x7ff4a000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:49.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.554+0000 7ffb354b0640 1 -- 172.21.15.63:0/84734618 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb30062e10 con 0x7ffb300a2cd0 2024-04-26T04:18:49.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.558+0000 7ff496ffd640 1 -- 172.21.15.63:0/2035668620 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4a000ac60 con 0x7ff4a8134f70 2024-04-26T04:18:49.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.558+0000 7ff496ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:49.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.558+0000 7ff496ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:18:49.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.558+0000 7ff496ffd640 10 monclient: dump: 2024-04-26T04:18:49.597 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:49.597 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:49.597 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:49.597 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:49.597 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:49.597 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:49.597 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:49.597 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:49.597 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:49.597 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:49.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.558+0000 7ff496ffd640 10 monclient: _finish_auth 0 2024-04-26T04:18:49.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.558+0000 7ff496ffd640 10 monclient: _check_auth_tickets 2024-04-26T04:18:49.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.558+0000 7ff496ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:49.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.558+0000 7ff496ffd640 1 -- 172.21.15.63:0/2035668620 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4a000ae00 con 0x7ff4a8134f70 2024-04-26T04:18:49.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.558+0000 7ff496ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:49.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.558+0000 7ff496ffd640 1 -- 172.21.15.63:0/2035668620 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4a0004bb0 con 0x7ff4a8134f70 2024-04-26T04:18:49.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.558+0000 7ff496ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:49.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.558+0000 7ff496ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:18:49.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.558+0000 7ff496ffd640 10 monclient: dump: 2024-04-26T04:18:49.598 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:49.598 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:49.599 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:49.599 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:49.599 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:49.599 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:49.599 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:49.599 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:49.599 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:49.599 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:49.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.558+0000 7ff4a4d99640 4 set_mon_vals no callback set 2024-04-26T04:18:49.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.558+0000 7ff4aca6d640 5 monclient: authenticate success, global_id 4290 2024-04-26T04:18:49.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.558+0000 7ff4aca6d640 10 monclient: _renew_subs 2024-04-26T04:18:49.600 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.558+0000 7ff4aca6d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:49.600 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.558+0000 7ff4aca6d640 1 -- 172.21.15.63:0/2035668620 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4a809bd60 con 0x7ff4a8134f70 2024-04-26T04:18:49.600 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.558+0000 7ff4aca6d640 10 monclient: _renew_subs 2024-04-26T04:18:49.600 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.558+0000 7ff4aca6d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:49.600 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.558+0000 7ff4aca6d640 1 -- 172.21.15.63:0/2035668620 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4a81cda00 con 0x7ff4a8134f70 2024-04-26T04:18:49.600 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.558+0000 7ff4aca6d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:49.600 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.558+0000 7ff4aca6d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:49.600 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.558+0000 7ff4aca6d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:49.600 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.558+0000 7ff4aca6d640 1 -- 172.21.15.63:0/2035668620 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4a8062e10 con 0x7ff4a8134f70 2024-04-26T04:18:49.600 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.558+0000 7ff496ffd640 1 -- 172.21.15.63:0/2035668620 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7ff4a0005030 con 0x7ff4a8134f70 2024-04-26T04:18:49.600 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.558+0000 7ff496ffd640 1 --2- 172.21.15.63:0/2035668620 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7ff48803a320 0x7ff48803c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:49.600 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.558+0000 7ff496ffd640 1 -- 172.21.15.63:0/2035668620 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7ff4a0049230 con 0x7ff4a8134f70 2024-04-26T04:18:49.600 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.558+0000 7ff4a659c640 1 --2- 172.21.15.63:0/2035668620 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7ff48803a320 0x7ff48803c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:49.600 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.558+0000 7ff4a659c640 10 monclient: get_auth_request con 0x7ff48803a320 auth_method 0 2024-04-26T04:18:49.600 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.562+0000 7ff4a659c640 1 --2- 172.21.15.63:0/2035668620 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7ff48803a320 0x7ff48803c7e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff498007a60 tx=0x7ff498008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:49.601 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.566+0000 7fbbcde5a640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-04-26T04:18:49.601 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.566+0000 7fbbcde5a640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-04-26T04:18:49.601 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.566+0000 7fbbcde5a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:49.601 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.566+0000 7fbbcde5a640 1 -- 172.21.15.63:0/3414563736 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fbbc8068480 con 0x7fbbc8133e00 2024-04-26T04:18:49.601 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.566+0000 7fbbaffff640 1 -- 172.21.15.63:0/3414563736 <== mon.2 v2:172.21.15.63:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fbbc0049360 con 0x7fbbc8133e00 2024-04-26T04:18:49.601 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.566+0000 7fbbaffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-04-26T04:18:49.601 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.566+0000 7fbbaffff640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:18:49.602 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.566+0000 7ffb177fe640 1 -- 172.21.15.63:0/84734618 <== mon.2 v2:172.21.15.63:3301/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7ffb2001a070 con 0x7ffb300a2cd0 2024-04-26T04:18:49.602 INFO:teuthology.orchestra.run.smithi063.stdout:47244640259 2024-04-26T04:18:49.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.566+0000 7ffb177fe640 1 --2- 172.21.15.63:0/84734618 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7ffb1c035f10 0x7ffb1c0383d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:49.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.566+0000 7ffb177fe640 1 -- 172.21.15.63:0/84734618 <== mon.2 v2:172.21.15.63:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7ffb20049480 con 0x7ffb300a2cd0 2024-04-26T04:18:49.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.566+0000 7ffb177fe640 1 -- 172.21.15.63:0/84734618 <== mon.2 v2:172.21.15.63:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7ffb200498d0 con 0x7ffb300a2cd0 2024-04-26T04:18:49.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.566+0000 7ffb177fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:49.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.566+0000 7ffb177fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:18:49.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.566+0000 7ff496ffd640 1 -- 172.21.15.63:0/2035668620 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7ff4a001a770 con 0x7ff4a8134f70 2024-04-26T04:18:49.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.566+0000 7ff496ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:49.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.566+0000 7ff496ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:18:49.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.566+0000 7ffb2e7fc640 1 --2- 172.21.15.63:0/84734618 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7ffb1c035f10 0x7ffb1c0383d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:49.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.566+0000 7ffb2e7fc640 10 monclient: get_auth_request con 0x7ffb1c035f10 auth_method 0 2024-04-26T04:18:49.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.570+0000 7fbbcde5a640 1 -- 172.21.15.63:0/3414563736 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fbbb4035ec0 msgr2=0x7fbbb4038380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:49.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.570+0000 7fbbcde5a640 1 --2- 172.21.15.63:0/3414563736 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fbbb4035ec0 0x7fbbb4038380 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fbbc806b4c0 tx=0x7fbbb80a9070 comp rx=0 tx=0).stop 2024-04-26T04:18:49.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.570+0000 7fbbcde5a640 10 monclient: shutdown 2024-04-26T04:18:49.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.570+0000 7fbbcde5a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:49.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.570+0000 7fbbcde5a640 1 -- 172.21.15.63:0/3414563736 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fbbc8133e00 msgr2=0x7fbbc8097d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:49.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.570+0000 7fbbcde5a640 1 --2- 172.21.15.63:0/3414563736 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fbbc8133e00 0x7fbbc8097d10 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fbbc000cd70 tx=0x7fbbc000a660 comp rx=0 tx=0).stop 2024-04-26T04:18:49.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.570+0000 7ffb2e7fc640 1 --2- 172.21.15.63:0/84734618 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7ffb1c035f10 0x7ffb1c0383d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ffb24007ca0 tx=0x7ffb24007bf0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:49.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.574+0000 7fbbcde5a640 1 -- 172.21.15.63:0/3414563736 shutdown_connections 2024-04-26T04:18:49.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.574+0000 7fbbcde5a640 1 --2- 172.21.15.63:0/3414563736 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fbbb4035ec0 0x7fbbb4038380 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:49.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.574+0000 7fbbcde5a640 1 --2- 172.21.15.63:0/3414563736 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fbbc8133e00 0x7fbbc8097d10 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:49.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.574+0000 7fbbcde5a640 1 --2- 172.21.15.63:0/3414563736 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbbc80a26d0 0x7fbbc80977d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:49.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.574+0000 7fbbcde5a640 1 --2- 172.21.15.63:0/3414563736 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbbc809a8c0 0x7fbbc8097290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:49.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.574+0000 7fbbcde5a640 1 -- 172.21.15.63:0/3414563736 >> 172.21.15.63:0/3414563736 conn(0x7fbbc80a2c40 msgr2=0x7fbbc80a4dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:49.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.574+0000 7fbbcde5a640 1 -- 172.21.15.63:0/3414563736 shutdown_connections 2024-04-26T04:18:49.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.574+0000 7fbbcde5a640 1 -- 172.21.15.63:0/3414563736 wait complete. 2024-04-26T04:18:49.608 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.3 2024-04-26T04:18:49.608 DEBUG:teuthology.parallel:result is None 2024-04-26T04:18:49.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.666+0000 7fed4dc0e640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-04-26T04:18:49.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.666+0000 7fed4dc0e640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-04-26T04:18:49.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.666+0000 7fed4dc0e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:49.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.666+0000 7fed4dc0e640 1 -- 172.21.15.63:0/3935969200 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fed480684a0 con 0x7fed4809bf30 2024-04-26T04:18:49.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.666+0000 7fed2ffff640 1 -- 172.21.15.63:0/3935969200 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fed4809c910 con 0x7fed4809bf30 2024-04-26T04:18:49.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.666+0000 7fed2ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-04-26T04:18:49.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.666+0000 7fed2ffff640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:18:49.671 INFO:teuthology.orchestra.run.smithi063.stdout:51539607556 2024-04-26T04:18:49.672 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.666+0000 7fed4dc0e640 1 -- 172.21.15.63:0/3935969200 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fed34035f10 msgr2=0x7fed340383d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:49.672 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.666+0000 7fed4dc0e640 1 --2- 172.21.15.63:0/3935969200 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fed34035f10 0x7fed340383d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fed38007fd0 tx=0x7fed38009040 comp rx=0 tx=0).stop 2024-04-26T04:18:49.672 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.666+0000 7fed4dc0e640 10 monclient: shutdown 2024-04-26T04:18:49.673 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.666+0000 7fed4dc0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:49.673 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.666+0000 7fed4dc0e640 1 -- 172.21.15.63:0/3935969200 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fed4809bf30 msgr2=0x7fed48072a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:49.673 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.666+0000 7fed4dc0e640 1 --2- 172.21.15.63:0/3935969200 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fed4809bf30 0x7fed48072a30 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fed3c01c7a0 tx=0x7fed3c019060 comp rx=0 tx=0).stop 2024-04-26T04:18:49.673 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.666+0000 7fed4dc0e640 1 -- 172.21.15.63:0/3935969200 shutdown_connections 2024-04-26T04:18:49.673 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.666+0000 7fed4dc0e640 1 --2- 172.21.15.63:0/3935969200 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fed34035f10 0x7fed340383d0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:49.673 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.666+0000 7fed4dc0e640 1 --2- 172.21.15.63:0/3935969200 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed480a1c60 0x7fed4813ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:49.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.666+0000 7fed4dc0e640 1 --2- 172.21.15.63:0/3935969200 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fed4809bf30 0x7fed48072a30 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:49.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.666+0000 7fed4dc0e640 1 --2- 172.21.15.63:0/3935969200 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fed4806dbf0 0x7fed480724f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:49.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.666+0000 7fed4dc0e640 1 -- 172.21.15.63:0/3935969200 >> 172.21.15.63:0/3935969200 conn(0x7fed480a6d80 msgr2=0x7fed4809f4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:49.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.666+0000 7fed4dc0e640 1 -- 172.21.15.63:0/3935969200 shutdown_connections 2024-04-26T04:18:49.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.666+0000 7fed4dc0e640 1 -- 172.21.15.63:0/3935969200 wait complete. 2024-04-26T04:18:49.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.674+0000 7f788ab4a640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-04-26T04:18:49.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.674+0000 7f788ab4a640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-04-26T04:18:49.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.674+0000 7f788ab4a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:49.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.674+0000 7f788ab4a640 1 -- 172.21.15.63:0/3268446327 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f7884072470 con 0x7f78840a47d0 2024-04-26T04:18:49.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.674+0000 7f78817fa640 1 -- 172.21.15.63:0/3268446327 <== mon.2 v2:172.21.15.63:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f78700bd370 con 0x7f78840a47d0 2024-04-26T04:18:49.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.674+0000 7f78817fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-04-26T04:18:49.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.674+0000 7f78817fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:18:49.682 INFO:teuthology.orchestra.run.smithi063.stdout:51539607557 2024-04-26T04:18:49.683 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.678+0000 7f788ab4a640 1 -- 172.21.15.63:0/3268446327 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f7868035ef0 msgr2=0x7f78680383b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:49.683 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.678+0000 7f788ab4a640 1 --2- 172.21.15.63:0/3268446327 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f7868035ef0 0x7f78680383b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7878009a50 tx=0x7f7878008040 comp rx=0 tx=0).stop 2024-04-26T04:18:49.684 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.678+0000 7f788ab4a640 10 monclient: shutdown 2024-04-26T04:18:49.684 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.678+0000 7f788ab4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:49.684 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.678+0000 7f788ab4a640 1 -- 172.21.15.63:0/3268446327 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f78840a47d0 msgr2=0x7f788412e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:49.684 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.678+0000 7f788ab4a640 1 --2- 172.21.15.63:0/3268446327 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f78840a47d0 0x7f788412e310 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f787001a0a0 tx=0x7f7870019210 comp rx=0 tx=0).stop 2024-04-26T04:18:49.684 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.678+0000 7f788ab4a640 1 -- 172.21.15.63:0/3268446327 shutdown_connections 2024-04-26T04:18:49.684 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.678+0000 7f788ab4a640 1 --2- 172.21.15.63:0/3268446327 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f7868035ef0 0x7f78680383b0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:49.685 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.678+0000 7f788ab4a640 1 --2- 172.21.15.63:0/3268446327 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f78840a4bd0 0x7f788406f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:49.685 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.678+0000 7f788ab4a640 1 --2- 172.21.15.63:0/3268446327 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f78840a47d0 0x7f788412e310 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:49.685 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.678+0000 7f788ab4a640 1 --2- 172.21.15.63:0/3268446327 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f78840a2cd0 0x7f788412ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:49.685 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.678+0000 7f788ab4a640 1 -- 172.21.15.63:0/3268446327 >> 172.21.15.63:0/3268446327 conn(0x7f78840a5f40 msgr2=0x7f7884130da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:49.685 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.678+0000 7f788ab4a640 1 -- 172.21.15.63:0/3268446327 shutdown_connections 2024-04-26T04:18:49.685 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.678+0000 7f788ab4a640 1 -- 172.21.15.63:0/3268446327 wait complete. 2024-04-26T04:18:49.688 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607556 got 51539607556 for osd.1 2024-04-26T04:18:49.688 DEBUG:teuthology.parallel:result is None 2024-04-26T04:18:49.698 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607557 got 51539607557 for osd.0 2024-04-26T04:18:49.698 DEBUG:teuthology.parallel:result is None 2024-04-26T04:18:49.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.726+0000 7ff4aca6d640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-04-26T04:18:49.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.726+0000 7ff4aca6d640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-04-26T04:18:49.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.726+0000 7ff4aca6d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:49.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.726+0000 7ff4aca6d640 1 -- 172.21.15.63:0/2035668620 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7ff4a813bf30 con 0x7ff4a8134f70 2024-04-26T04:18:49.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.726+0000 7ff496ffd640 1 -- 172.21.15.63:0/2035668620 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ff4a0012420 con 0x7ff4a8134f70 2024-04-26T04:18:49.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.726+0000 7ff496ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-04-26T04:18:49.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.726+0000 7ff496ffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:18:49.733 INFO:teuthology.orchestra.run.smithi063.stdout:47244640259 2024-04-26T04:18:49.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.730+0000 7ff4aca6d640 1 -- 172.21.15.63:0/2035668620 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7ff48803a320 msgr2=0x7ff48803c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:49.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.730+0000 7ff4aca6d640 1 --2- 172.21.15.63:0/2035668620 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7ff48803a320 0x7ff48803c7e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff498007a60 tx=0x7ff498008040 comp rx=0 tx=0).stop 2024-04-26T04:18:49.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.730+0000 7ff4aca6d640 10 monclient: shutdown 2024-04-26T04:18:49.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.730+0000 7ff4aca6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:49.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.730+0000 7ff4aca6d640 1 -- 172.21.15.63:0/2035668620 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff4a8134f70 msgr2=0x7ff4a8138850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:49.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.730+0000 7ff4aca6d640 1 --2- 172.21.15.63:0/2035668620 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff4a8134f70 0x7ff4a8138850 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff4a000c780 tx=0x7ff4a000cc40 comp rx=0 tx=0).stop 2024-04-26T04:18:49.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.730+0000 7ff4aca6d640 1 -- 172.21.15.63:0/2035668620 shutdown_connections 2024-04-26T04:18:49.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.730+0000 7ff4aca6d640 1 --2- 172.21.15.63:0/2035668620 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7ff48803a320 0x7ff48803c7e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:49.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.730+0000 7ff4aca6d640 1 --2- 172.21.15.63:0/2035668620 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff4a8134f70 0x7ff4a8138850 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:49.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.730+0000 7ff4aca6d640 1 --2- 172.21.15.63:0/2035668620 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff4a8134b90 0x7ff4a81cd4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:49.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.730+0000 7ff4aca6d640 1 --2- 172.21.15.63:0/2035668620 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff4a8133b10 0x7ff4a81ccd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:49.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.730+0000 7ff4aca6d640 1 -- 172.21.15.63:0/2035668620 >> 172.21.15.63:0/2035668620 conn(0x7ff4a80a7360 msgr2=0x7ff4a80a3df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:49.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.730+0000 7ff4aca6d640 1 -- 172.21.15.63:0/2035668620 shutdown_connections 2024-04-26T04:18:49.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.730+0000 7ff4aca6d640 1 -- 172.21.15.63:0/2035668620 wait complete. 2024-04-26T04:18:49.749 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.4 2024-04-26T04:18:49.749 DEBUG:teuthology.parallel:result is None 2024-04-26T04:18:49.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.750+0000 7ffb354b0640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-04-26T04:18:49.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.750+0000 7ffb354b0640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-04-26T04:18:49.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.750+0000 7ffb354b0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:49.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.750+0000 7ffb354b0640 1 -- 172.21.15.63:0/84734618 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7ffb300683d0 con 0x7ffb300a2cd0 2024-04-26T04:18:49.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.750+0000 7ffb177fe640 1 -- 172.21.15.63:0/84734618 <== mon.2 v2:172.21.15.63:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ffb20013dc0 con 0x7ffb300a2cd0 2024-04-26T04:18:49.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.750+0000 7ffb177fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-04-26T04:18:49.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.750+0000 7ffb177fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:18:49.756 INFO:teuthology.orchestra.run.smithi063.stdout:47244640259 2024-04-26T04:18:49.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.750+0000 7ffb354b0640 1 -- 172.21.15.63:0/84734618 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7ffb1c035f10 msgr2=0x7ffb1c0383d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:49.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.750+0000 7ffb354b0640 1 --2- 172.21.15.63:0/84734618 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7ffb1c035f10 0x7ffb1c0383d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ffb24007ca0 tx=0x7ffb24007bf0 comp rx=0 tx=0).stop 2024-04-26T04:18:49.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.750+0000 7ffb354b0640 10 monclient: shutdown 2024-04-26T04:18:49.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.750+0000 7ffb354b0640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:49.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.750+0000 7ffb354b0640 1 -- 172.21.15.63:0/84734618 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ffb300a2cd0 msgr2=0x7ffb301ceed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:49.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.750+0000 7ffb354b0640 1 --2- 172.21.15.63:0/84734618 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ffb300a2cd0 0x7ffb301ceed0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ffb2000e8d0 tx=0x7ffb2000ed90 comp rx=0 tx=0).stop 2024-04-26T04:18:49.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.754+0000 7ffb354b0640 1 -- 172.21.15.63:0/84734618 shutdown_connections 2024-04-26T04:18:49.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.754+0000 7ffb354b0640 1 --2- 172.21.15.63:0/84734618 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7ffb1c035f10 0x7ffb1c0383d0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:49.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.754+0000 7ffb354b0640 1 --2- 172.21.15.63:0/84734618 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffb300a4c00 0x7ffb3013ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:49.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.754+0000 7ffb354b0640 1 --2- 172.21.15.63:0/84734618 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffb300a47d0 0x7ffb301cf600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:49.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.754+0000 7ffb354b0640 1 --2- 172.21.15.63:0/84734618 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ffb300a2cd0 0x7ffb301ceed0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:49.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.754+0000 7ffb354b0640 1 -- 172.21.15.63:0/84734618 >> 172.21.15.63:0/84734618 conn(0x7ffb300a5f40 msgr2=0x7ffb300a2320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:49.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.754+0000 7ffb354b0640 1 -- 172.21.15.63:0/84734618 shutdown_connections 2024-04-26T04:18:49.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:49.754+0000 7ffb354b0640 1 -- 172.21.15.63:0/84734618 wait complete. 2024-04-26T04:18:49.772 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.5 2024-04-26T04:18:49.772 DEBUG:teuthology.parallel:result is None 2024-04-26T04:18:50.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.350+0000 7f621cff9640 1 -- 172.21.15.63:0/3835818408 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f62180c6070 con 0x7f62200a0bb0 2024-04-26T04:18:50.359 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.350+0000 7f621cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:50.359 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.350+0000 7f621cff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:18:50.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.510+0000 7f62262bd640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-04-26T04:18:50.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.510+0000 7f62262bd640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-04-26T04:18:50.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.510+0000 7f62262bd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-04-26T04:18:50.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.510+0000 7f62262bd640 1 -- 172.21.15.63:0/3835818408 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f6220062e10 con 0x7f62200a0bb0 2024-04-26T04:18:50.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.510+0000 7f621cff9640 1 -- 172.21.15.63:0/3835818408 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f62200a0f90 con 0x7f62200a0bb0 2024-04-26T04:18:50.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.510+0000 7f621cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-04-26T04:18:50.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.510+0000 7f621cff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:18:50.517 INFO:teuthology.orchestra.run.smithi063.stdout:51539607556 2024-04-26T04:18:50.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.514+0000 7f62262bd640 1 -- 172.21.15.63:0/3835818408 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f620c035ec0 msgr2=0x7f620c038380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:50.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.514+0000 7f62262bd640 1 --2- 172.21.15.63:0/3835818408 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f620c035ec0 0x7f620c038380 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6210007d60 tx=0x7f6210007cb0 comp rx=0 tx=0).stop 2024-04-26T04:18:50.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.514+0000 7f62262bd640 10 monclient: shutdown 2024-04-26T04:18:50.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.514+0000 7f62262bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:50.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.514+0000 7f62262bd640 1 -- 172.21.15.63:0/3835818408 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f62200a0bb0 msgr2=0x7f62201411c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:50.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.514+0000 7f62262bd640 1 --2- 172.21.15.63:0/3835818408 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f62200a0bb0 0x7f62201411c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f62200a11a0 tx=0x7f621801a870 comp rx=0 tx=0).stop 2024-04-26T04:18:50.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.514+0000 7f62262bd640 1 -- 172.21.15.63:0/3835818408 shutdown_connections 2024-04-26T04:18:50.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.514+0000 7f62262bd640 1 --2- 172.21.15.63:0/3835818408 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f620c035ec0 0x7f620c038380 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:50.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.514+0000 7f62262bd640 1 --2- 172.21.15.63:0/3835818408 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f62200a0bb0 0x7f62201411c0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:50.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.514+0000 7f62262bd640 1 --2- 172.21.15.63:0/3835818408 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f622009b400 0x7f6220140c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:50.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.514+0000 7f62262bd640 1 --2- 172.21.15.63:0/3835818408 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f622009aff0 0x7f6220144fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:50.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.514+0000 7f62262bd640 1 -- 172.21.15.63:0/3835818408 >> 172.21.15.63:0/3835818408 conn(0x7f62200a1e70 msgr2=0x7f622009b990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:50.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.514+0000 7f62262bd640 1 -- 172.21.15.63:0/3835818408 shutdown_connections 2024-04-26T04:18:50.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.514+0000 7f62262bd640 1 -- 172.21.15.63:0/3835818408 wait complete. 2024-04-26T04:18:50.535 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607556 got 51539607556 for osd.2 2024-04-26T04:18:50.536 DEBUG:teuthology.parallel:result is None 2024-04-26T04:18:50.536 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-26T04:18:50.536 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T04:18:50.654 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.646+0000 7fd3d27ae640 10 monclient: get_monmap_and_config 2024-04-26T04:18:50.654 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.646+0000 7fd3d27ae640 10 monclient: build_initial_monmap 2024-04-26T04:18:50.654 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.646+0000 7fd3d27ae640 10 monclient: monmap: 2024-04-26T04:18:50.654 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:50.654 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:50.654 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:50.653341+0000 2024-04-26T04:18:50.654 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:50.653341+0000 2024-04-26T04:18:50.655 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:50.655 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:50.655 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:18:50.655 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:18:50.655 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:18:50.655 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:18:50.655 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:18:50.655 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:50.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.650+0000 7fd3d27ae640 1 Processor -- start 2024-04-26T04:18:50.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.650+0000 7fd3d27ae640 1 -- start start 2024-04-26T04:18:50.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.650+0000 7fd3d27ae640 10 monclient: init 2024-04-26T04:18:50.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.650+0000 7fd3d27ae640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:50.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.650+0000 7fd3d27ae640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-04-26T04:18:50.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.650+0000 7fd3d27ae640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7fd3cc09d110 0x7fd3cc09aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:50.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.650+0000 7fd3d27ae640 10 monclient(hunting): picked mon.noname-a con 0x7fd3cc09d110 addr v2:172.21.15.63:3300/0 2024-04-26T04:18:50.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.650+0000 7fd3d27ae640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd3cc09d4f0 0x7fd3cc09d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:50.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.650+0000 7fd3d27ae640 10 monclient(hunting): picked mon.noname-c con 0x7fd3cc09d4f0 addr v2:172.21.15.186:3300/0 2024-04-26T04:18:50.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.650+0000 7fd3d27ae640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd3cc0a2ce0 addr v1:172.21.15.63:6789/0 2024-04-26T04:18:50.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.650+0000 7fd3d27ae640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3cc09ed60 con 0x7fd3cc0a2ce0 2024-04-26T04:18:50.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.650+0000 7fd3d27ae640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:50.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.650+0000 7fd3d27ae640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7fd3cc0a0550 con 0x7fd3cc09d110 2024-04-26T04:18:50.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.650+0000 7fd3d27ae640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:50.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.650+0000 7fd3d27ae640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fd3cc0a0a80 con 0x7fd3cc09d4f0 2024-04-26T04:18:50.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.650+0000 7fd3d27ae640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:50.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.650+0000 7fd3d27ae640 10 monclient(hunting): authenticate will time out at 962.360979s 2024-04-26T04:18:50.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.650+0000 7fd3cbfff640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7fd3cc09d110 0x7fd3cc09aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:50.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.650+0000 7fd3cbfff640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7fd3cc09d110 0x7fd3cc09aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:39246/0 (socket says 172.21.15.63:39246) 2024-04-26T04:18:50.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.650+0000 7fd3cbfff640 1 -- 172.21.15.63:0/1107348785 learned_addr learned my addr 172.21.15.63:0/1107348785 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:18:50.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.650+0000 7fd3cbfff640 10 monclient(hunting): get_auth_request con 0x7fd3cc09d110 auth_method 0 2024-04-26T04:18:50.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.650+0000 7fd3cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:50.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.650+0000 7fd3cbfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:50.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.650+0000 7fd3cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:50.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.650+0000 7fd3d0d4b640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7fd3cc0a2ce0 0x7fd3cc0a3110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:32788/0 (socket says 172.21.15.63:32788) 2024-04-26T04:18:50.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.650+0000 7fd3cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:50.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.650+0000 7fd3cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:50.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.650+0000 7fd3cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:50.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.650+0000 7fd3cbfff640 10 monclient(hunting): handle_auth_done global_id 4293 payload 274 2024-04-26T04:18:50.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.650+0000 7fd3cbfff640 1 -- 172.21.15.63:0/1107348785 >> v2:172.21.15.186:3300/0 conn(0x7fd3cc09d4f0 msgr2=0x7fd3cc09d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:50.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.650+0000 7fd3c3fff640 1 --2- 172.21.15.63:0/1107348785 >> v2:172.21.15.186:3300/0 conn(0x7fd3cc09d4f0 0x7fd3cc09d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:50.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.650+0000 7fd3cbfff640 1 --2- 172.21.15.63:0/1107348785 >> v2:172.21.15.186:3300/0 conn(0x7fd3cc09d4f0 0x7fd3cc09d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:50.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.650+0000 7fd3cbfff640 1 -- 172.21.15.63:0/1107348785 >> v1:172.21.15.63:6789/0 conn(0x7fd3cc0a2ce0 legacy=0x7fd3cc0a3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:50.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.650+0000 7fd3cbfff640 10 monclient: _finish_hunting 0 2024-04-26T04:18:50.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.650+0000 7fd3cbfff640 1 monclient: found mon.noname-a 2024-04-26T04:18:50.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.650+0000 7fd3cbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:50.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.650+0000 7fd3cbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:18:50.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.650+0000 7fd3cbfff640 1 -- 172.21.15.63:0/1107348785 --> v2:172.21.15.63:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3cc09f1a0 con 0x7fd3cc09d110 2024-04-26T04:18:50.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.650+0000 7fd3c3fff640 10 monclient: get_auth_request con 0x7fd3cc09d4f0 auth_method 0 2024-04-26T04:18:50.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.650+0000 7fd3c3fff640 1 --2- 172.21.15.63:0/1107348785 >> v2:172.21.15.186:3300/0 conn(0x7fd3cc09d4f0 0x7fd3cc09d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T04:18:50.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3cbfff640 1 -- 172.21.15.63:0/1107348785 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd3cc09d110 msgr2=0x7fd3cc09aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:50.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3cbfff640 1 --2- 172.21.15.63:0/1107348785 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd3cc09d110 0x7fd3cc09aa40 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc06bc50 tx=0x7fd3bc01c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47eedca085f33847 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:50.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3cb7fe640 1 -- 172.21.15.63:0/1107348785 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3bc0b0070 con 0x7fd3cc09d110 2024-04-26T04:18:50.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3cb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:50.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3cb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:18:50.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3cb7fe640 10 monclient: dump: 2024-04-26T04:18:50.665 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:50.665 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:50.665 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:50.665 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:50.665 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:50.665 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:50.665 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:50.666 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:50.666 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:50.666 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:50.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3cb7fe640 10 monclient: _finish_auth 0 2024-04-26T04:18:50.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3cb7fe640 10 monclient: _check_auth_tickets 2024-04-26T04:18:50.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3cb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:50.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3cb7fe640 1 -- 172.21.15.63:0/1107348785 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3bc0ab600 con 0x7fd3cc09d110 2024-04-26T04:18:50.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3d27ae640 5 monclient: authenticate success, global_id 4293 2024-04-26T04:18:50.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3d27ae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T04:18:50.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3cb7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:50.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3d27ae640 10 monclient: get_monmap_and_config success 2024-04-26T04:18:50.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3d27ae640 4 set_mon_vals no callback set 2024-04-26T04:18:50.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3cb7fe640 1 -- 172.21.15.63:0/1107348785 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3bc0ab9d0 con 0x7fd3cc09d110 2024-04-26T04:18:50.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3cb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:50.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3cb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:18:50.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3cb7fe640 10 monclient: dump: 2024-04-26T04:18:50.667 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:50.667 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:50.667 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:50.667 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:50.667 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:50.667 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:50.667 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:50.667 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:50.667 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:50.667 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:50.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3d27ae640 10 monclient: shutdown 2024-04-26T04:18:50.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3d27ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:50.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3d27ae640 1 -- 172.21.15.63:0/1107348785 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd3cc09d110 msgr2=0x7fd3cc09aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:50.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3d27ae640 1 --2- 172.21.15.63:0/1107348785 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd3cc09d110 0x7fd3cc09aa40 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc06bc50 tx=0x7fd3bc01c3a0 comp rx=0 tx=0).stop 2024-04-26T04:18:50.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3d27ae640 1 -- 172.21.15.63:0/1107348785 shutdown_connections 2024-04-26T04:18:50.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3d27ae640 1 --2- 172.21.15.63:0/1107348785 >> v2:172.21.15.186:3300/0 conn(0x7fd3cc09d4f0 0x7fd3cc09d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:50.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3d27ae640 1 --2- 172.21.15.63:0/1107348785 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd3cc09d110 0x7fd3cc09aa40 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:50.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3d27ae640 1 -- 172.21.15.63:0/1107348785 >> 172.21.15.63:0/1107348785 conn(0x7fd3cc0a7ab0 msgr2=0x7fd3cc0a3dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:50.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3d27ae640 1 -- 172.21.15.63:0/1107348785 shutdown_connections 2024-04-26T04:18:50.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3d27ae640 1 -- 172.21.15.63:0/1107348785 wait complete. 2024-04-26T04:18:50.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3d27ae640 10 monclient: build_initial_monmap 2024-04-26T04:18:50.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3d27ae640 10 monclient: monmap: 2024-04-26T04:18:50.668 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:50.668 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:18:50.668 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:18:50.669 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:18:50.669 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:50.669 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:50.669 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:18:50.669 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:18:50.669 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:18:50.669 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:50.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3d27ae640 1 Processor -- start 2024-04-26T04:18:50.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3d27ae640 1 -- start start 2024-04-26T04:18:50.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3d27ae640 10 monclient: init 2024-04-26T04:18:50.673 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3d27ae640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:50.673 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3d27ae640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T04:18:50.673 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3d27ae640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd3cc09d110 0x7fd3cc13df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:50.673 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3d27ae640 10 monclient(hunting): picked mon.noname-a con 0x7fd3cc09d110 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:18:50.673 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3d27ae640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3cc09d4f0 0x7fd3cc124950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:50.673 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3d27ae640 10 monclient(hunting): picked mon.noname-b con 0x7fd3cc09d4f0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:18:50.673 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3d27ae640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fd3cc0a2ce0 0x7fd3cc124e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:50.673 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3d27ae640 10 monclient(hunting): picked mon.noname-c con 0x7fd3cc0a2ce0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:50.673 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3d27ae640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:50.673 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3d27ae640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fd3cc141c90 con 0x7fd3cc09d110 2024-04-26T04:18:50.673 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3d27ae640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:50.673 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3d27ae640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd3cc141b10 con 0x7fd3cc09d4f0 2024-04-26T04:18:50.673 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3d27ae640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:50.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3d27ae640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7fd3cc141e10 con 0x7fd3cc0a2ce0 2024-04-26T04:18:50.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3d27ae640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:50.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3d27ae640 10 monclient(hunting): authenticate will time out at 962.367596s 2024-04-26T04:18:50.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3cbfff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd3cc09d110 0x7fd3cc13df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:50.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3cbfff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd3cc09d110 0x7fd3cc13df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:39258/0 (socket says 172.21.15.63:39258) 2024-04-26T04:18:50.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3cbfff640 1 -- 172.21.15.63:0/4066752205 learned_addr learned my addr 172.21.15.63:0/4066752205 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:18:50.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3d0d4b640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fd3cc0a2ce0 0x7fd3cc124e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:50.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3cbfff640 10 monclient(hunting): get_auth_request con 0x7fd3cc09d110 auth_method 0 2024-04-26T04:18:50.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:50.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3cbfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:50.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:50.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3d0d4b640 10 monclient(hunting): get_auth_request con 0x7fd3cc0a2ce0 auth_method 0 2024-04-26T04:18:50.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3d0d4b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:50.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3d0d4b640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:50.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3d0d4b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:50.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:50.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:50.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:50.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3d0d4b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:50.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3d0d4b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:50.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3d0d4b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:50.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3cbfff640 10 monclient(hunting): handle_auth_done global_id 4296 payload 824 2024-04-26T04:18:50.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3c3fff640 1 --2- 172.21.15.63:0/4066752205 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3cc09d4f0 0x7fd3cc124950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:50.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3cbfff640 1 -- 172.21.15.63:0/4066752205 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fd3cc0a2ce0 msgr2=0x7fd3cc124e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:50.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.654+0000 7fd3cbfff640 1 --2- 172.21.15.63:0/4066752205 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fd3cc0a2ce0 0x7fd3cc124e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:50.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.658+0000 7fd3cbfff640 1 -- 172.21.15.63:0/4066752205 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3cc09d4f0 msgr2=0x7fd3cc124950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:50.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.658+0000 7fd3cbfff640 1 --2- 172.21.15.63:0/4066752205 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3cc09d4f0 0x7fd3cc124950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:50.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.658+0000 7fd3cbfff640 10 monclient: _finish_hunting 0 2024-04-26T04:18:50.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.658+0000 7fd3cbfff640 1 monclient: found mon.noname-a 2024-04-26T04:18:50.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.658+0000 7fd3cbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:50.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.658+0000 7fd3cbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:18:50.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.658+0000 7fd3cbfff640 1 -- 172.21.15.63:0/4066752205 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3cc1cfde0 con 0x7fd3cc09d110 2024-04-26T04:18:50.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.658+0000 7fd3c3fff640 10 monclient: get_auth_request con 0x7fd3cc09d4f0 auth_method 0 2024-04-26T04:18:50.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.658+0000 7fd3c3fff640 1 --2- 172.21.15.63:0/4066752205 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3cc09d4f0 0x7fd3cc124950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T04:18:50.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.658+0000 7fd3cbfff640 1 -- 172.21.15.63:0/4066752205 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd3cc09d110 msgr2=0x7fd3cc13df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:50.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.658+0000 7fd3cbfff640 1 --2- 172.21.15.63:0/4066752205 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd3cc09d110 0x7fd3cc13df50 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd3bc018fb0 tx=0x7fd3bc019140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:50.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.658+0000 7fd3c97fa640 1 -- 172.21.15.63:0/4066752205 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3bc0b0070 con 0x7fd3cc09d110 2024-04-26T04:18:50.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.658+0000 7fd3c97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:50.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.658+0000 7fd3c97fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:18:50.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.658+0000 7fd3c97fa640 10 monclient: dump: 2024-04-26T04:18:50.676 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:50.676 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:50.676 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:50.677 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:50.677 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:50.677 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:50.677 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:50.677 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:50.677 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:50.677 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:50.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.658+0000 7fd3c97fa640 10 monclient: _finish_auth 0 2024-04-26T04:18:50.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.658+0000 7fd3c97fa640 10 monclient: _check_auth_tickets 2024-04-26T04:18:50.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.658+0000 7fd3c97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:50.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.658+0000 7fd3c97fa640 1 -- 172.21.15.63:0/4066752205 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3bc0b1040 con 0x7fd3cc09d110 2024-04-26T04:18:50.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.658+0000 7fd3c97fa640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:50.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.658+0000 7fd3c97fa640 1 -- 172.21.15.63:0/4066752205 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3bc0aaa00 con 0x7fd3cc09d110 2024-04-26T04:18:50.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.658+0000 7fd3c97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:50.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.658+0000 7fd3c97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:18:50.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.658+0000 7fd3c97fa640 10 monclient: dump: 2024-04-26T04:18:50.678 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:50.678 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:50.678 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:50.678 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:50.678 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:50.678 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:50.678 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:50.679 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:50.679 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:50.679 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:50.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.658+0000 7fd3caffd640 4 set_mon_vals no callback set 2024-04-26T04:18:50.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.658+0000 7fd3d27ae640 5 monclient: authenticate success, global_id 4296 2024-04-26T04:18:50.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.658+0000 7fd3d27ae640 10 monclient: _renew_subs 2024-04-26T04:18:50.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.658+0000 7fd3d27ae640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:50.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.658+0000 7fd3d27ae640 1 -- 172.21.15.63:0/4066752205 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3cc09df50 con 0x7fd3cc09d110 2024-04-26T04:18:50.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.658+0000 7fd3d27ae640 10 monclient: _renew_subs 2024-04-26T04:18:50.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.658+0000 7fd3d27ae640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:50.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.658+0000 7fd3d27ae640 1 -- 172.21.15.63:0/4066752205 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3cc1288e0 con 0x7fd3cc09d110 2024-04-26T04:18:50.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.658+0000 7fd3c97fa640 1 -- 172.21.15.63:0/4066752205 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fd3bc0aaba0 con 0x7fd3cc09d110 2024-04-26T04:18:50.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.658+0000 7fd3d27ae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:50.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.658+0000 7fd3d27ae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:50.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.658+0000 7fd3d27ae640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:50.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.658+0000 7fd3d27ae640 1 -- 172.21.15.63:0/4066752205 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3cc062e10 con 0x7fd3cc09d110 2024-04-26T04:18:50.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.658+0000 7fd3c97fa640 1 --2- 172.21.15.63:0/4066752205 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fd3ac035ec0 0x7fd3ac038380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:50.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.658+0000 7fd3c97fa640 1 -- 172.21.15.63:0/4066752205 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7fd3bc0e2c00 con 0x7fd3cc09d110 2024-04-26T04:18:50.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.662+0000 7fd3c3fff640 1 --2- 172.21.15.63:0/4066752205 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fd3ac035ec0 0x7fd3ac038380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:50.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.662+0000 7fd3c3fff640 10 monclient: get_auth_request con 0x7fd3ac035ec0 auth_method 0 2024-04-26T04:18:50.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.662+0000 7fd3c3fff640 1 --2- 172.21.15.63:0/4066752205 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fd3ac035ec0 0x7fd3ac038380 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc1261f0 tx=0x7fd3b4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:50.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.666+0000 7fd3c97fa640 1 -- 172.21.15.63:0/4066752205 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fd3cc09df50 con 0x7fd3cc09d110 2024-04-26T04:18:50.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.666+0000 7fd3c97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:50.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.666+0000 7fd3c97fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:18:50.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.814+0000 7fd3d27ae640 1 -- 172.21.15.63:0/4066752205 --> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd3cc0683f0 con 0x7fd3ac035ec0 2024-04-26T04:18:50.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.822+0000 7fd3c97fa640 1 -- 172.21.15.63:0/4066752205 <== mgr.4105 v2:172.21.15.186:6824/2583706639 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33807 (secure 0 0 0) 0x7fd3cc0683f0 con 0x7fd3ac035ec0 2024-04-26T04:18:50.825 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-26T04:18:50.825 INFO:teuthology.orchestra.run.smithi063.stderr:dumped all 2024-04-26T04:18:50.827 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.822+0000 7fd3d27ae640 1 -- 172.21.15.63:0/4066752205 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fd3ac035ec0 msgr2=0x7fd3ac038380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:50.827 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.822+0000 7fd3d27ae640 1 --2- 172.21.15.63:0/4066752205 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fd3ac035ec0 0x7fd3ac038380 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc1261f0 tx=0x7fd3b4008040 comp rx=0 tx=0).stop 2024-04-26T04:18:50.827 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.822+0000 7fd3d27ae640 10 monclient: shutdown 2024-04-26T04:18:50.827 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.822+0000 7fd3d27ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:50.827 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.822+0000 7fd3d27ae640 1 -- 172.21.15.63:0/4066752205 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd3cc09d110 msgr2=0x7fd3cc13df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:50.828 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.822+0000 7fd3d27ae640 1 --2- 172.21.15.63:0/4066752205 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd3cc09d110 0x7fd3cc13df50 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd3bc018fb0 tx=0x7fd3bc019140 comp rx=0 tx=0).stop 2024-04-26T04:18:50.828 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.822+0000 7fd3d27ae640 1 -- 172.21.15.63:0/4066752205 shutdown_connections 2024-04-26T04:18:50.828 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.822+0000 7fd3d27ae640 1 --2- 172.21.15.63:0/4066752205 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fd3ac035ec0 0x7fd3ac038380 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:50.828 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.822+0000 7fd3d27ae640 1 --2- 172.21.15.63:0/4066752205 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fd3cc0a2ce0 0x7fd3cc124e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:50.828 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.822+0000 7fd3d27ae640 1 --2- 172.21.15.63:0/4066752205 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3cc09d4f0 0x7fd3cc124950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:50.828 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.822+0000 7fd3d27ae640 1 --2- 172.21.15.63:0/4066752205 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd3cc09d110 0x7fd3cc13df50 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:50.828 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.822+0000 7fd3d27ae640 1 -- 172.21.15.63:0/4066752205 >> 172.21.15.63:0/4066752205 conn(0x7fd3cc0a7ab0 msgr2=0x7fd3cc09b0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:50.828 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.822+0000 7fd3d27ae640 1 -- 172.21.15.63:0/4066752205 shutdown_connections 2024-04-26T04:18:50.828 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.822+0000 7fd3d27ae640 1 -- 172.21.15.63:0/4066752205 wait complete. 2024-04-26T04:18:50.842 INFO:teuthology.orchestra.run.smithi063.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-04-26T04:18:49.446817+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":164372,"kb_used_data":1808,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566066668,"statfs":{"total":579820584960,"available":579652268032,"internally_reserved":0,"allocated":1851392,"data_stored":1284122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9567,"internal_metadata":165927585},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-26T04:18:47.497690+0000","last_change":"2024-04-26T04:18:47.131857+0000","last_active":"2024-04-26T04:18:47.497690+0000","last_peered":"2024-04-26T04:18:47.497690+0000","last_clean":"2024-04-26T04:18:47.497690+0000","last_became_active":"2024-04-26T04:18:47.131175+0000","last_became_peered":"2024-04-26T04:18:47.131175+0000","last_unstale":"2024-04-26T04:18:47.497690+0000","last_undegraded":"2024-04-26T04:18:47.497690+0000","last_fullsized":"2024-04-26T04:18:47.497690+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_clean_scrub_stamp":"2024-04-26T04:18:45.482773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:35:03.860574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-26T04:18:47.493565+0000","last_change":"2024-04-26T04:18:46.499700+0000","last_active":"2024-04-26T04:18:47.493565+0000","last_peered":"2024-04-26T04:18:47.493565+0000","last_clean":"2024-04-26T04:18:47.493565+0000","last_became_active":"2024-04-26T04:18:46.498983+0000","last_became_peered":"2024-04-26T04:18:46.498983+0000","last_unstale":"2024-04-26T04:18:47.493565+0000","last_undegraded":"2024-04-26T04:18:47.493565+0000","last_fullsized":"2024-04-26T04:18:47.493565+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_clean_scrub_stamp":"2024-04-26T04:18:45.482773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:51:56.740670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-26T04:18:47.497623+0000","last_change":"2024-04-26T04:18:46.507904+0000","last_active":"2024-04-26T04:18:47.497623+0000","last_peered":"2024-04-26T04:18:47.497623+0000","last_clean":"2024-04-26T04:18:47.497623+0000","last_became_active":"2024-04-26T04:18:46.507335+0000","last_became_peered":"2024-04-26T04:18:46.507335+0000","last_unstale":"2024-04-26T04:18:47.497623+0000","last_undegraded":"2024-04-26T04:18:47.497623+0000","last_fullsized":"2024-04-26T04:18:47.497623+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_clean_scrub_stamp":"2024-04-26T04:18:45.482773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:52:50.029369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-26T04:18:47.493398+0000","last_change":"2024-04-26T04:18:46.506682+0000","last_active":"2024-04-26T04:18:47.493398+0000","last_peered":"2024-04-26T04:18:47.493398+0000","last_clean":"2024-04-26T04:18:47.493398+0000","last_became_active":"2024-04-26T04:18:46.506209+0000","last_became_peered":"2024-04-26T04:18:46.506209+0000","last_unstale":"2024-04-26T04:18:47.493398+0000","last_undegraded":"2024-04-26T04:18:47.493398+0000","last_fullsized":"2024-04-26T04:18:47.493398+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_clean_scrub_stamp":"2024-04-26T04:18:45.482773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:20:00.475307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-26T04:18:46.506418+0000","last_change":"2024-04-26T04:18:46.504685+0000","last_active":"2024-04-26T04:18:46.506418+0000","last_peered":"2024-04-26T04:18:46.506418+0000","last_clean":"2024-04-26T04:18:46.506418+0000","last_became_active":"2024-04-26T04:18:46.504292+0000","last_became_peered":"2024-04-26T04:18:46.504292+0000","last_unstale":"2024-04-26T04:18:46.506418+0000","last_undegraded":"2024-04-26T04:18:46.506418+0000","last_fullsized":"2024-04-26T04:18:46.506418+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_clean_scrub_stamp":"2024-04-26T04:18:45.482773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:56:59.840987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-26T04:18:46.506175+0000","last_change":"2024-04-26T04:18:46.504010+0000","last_active":"2024-04-26T04:18:46.506175+0000","last_peered":"2024-04-26T04:18:46.506175+0000","last_clean":"2024-04-26T04:18:46.506175+0000","last_became_active":"2024-04-26T04:18:46.503343+0000","last_became_peered":"2024-04-26T04:18:46.503343+0000","last_unstale":"2024-04-26T04:18:46.506175+0000","last_undegraded":"2024-04-26T04:18:46.506175+0000","last_fullsized":"2024-04-26T04:18:46.506175+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_clean_scrub_stamp":"2024-04-26T04:18:45.482773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:43:27.096237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-26T04:18:47.497466+0000","last_change":"2024-04-26T04:18:46.503233+0000","last_active":"2024-04-26T04:18:47.497466+0000","last_peered":"2024-04-26T04:18:47.497466+0000","last_clean":"2024-04-26T04:18:47.497466+0000","last_became_active":"2024-04-26T04:18:46.502682+0000","last_became_peered":"2024-04-26T04:18:46.502682+0000","last_unstale":"2024-04-26T04:18:47.497466+0000","last_undegraded":"2024-04-26T04:18:47.497466+0000","last_fullsized":"2024-04-26T04:18:47.497466+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_clean_scrub_stamp":"2024-04-26T04:18:45.482773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:26:15.760818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-26T04:18:46.506086+0000","last_change":"2024-04-26T04:18:46.504844+0000","last_active":"2024-04-26T04:18:46.506086+0000","last_peered":"2024-04-26T04:18:46.506086+0000","last_clean":"2024-04-26T04:18:46.506086+0000","last_became_active":"2024-04-26T04:18:46.504332+0000","last_became_peered":"2024-04-26T04:18:46.504332+0000","last_unstale":"2024-04-26T04:18:46.506086+0000","last_undegraded":"2024-04-26T04:18:46.506086+0000","last_fullsized":"2024-04-26T04:18:46.506086+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_clean_scrub_stamp":"2024-04-26T04:18:45.482773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:02:57.619500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":164,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-26T04:18:47.497482+0000","last_change":"2024-04-26T04:18:45.505818+0000","last_active":"2024-04-26T04:18:47.497482+0000","last_peered":"2024-04-26T04:18:47.497482+0000","last_clean":"2024-04-26T04:18:47.497482+0000","last_became_active":"2024-04-26T04:18:45.504887+0000","last_became_peered":"2024-04-26T04:18:45.504887+0000","last_unstale":"2024-04-26T04:18:47.497482+0000","last_undegraded":"2024-04-26T04:18:47.497482+0000","last_fullsized":"2024-04-26T04:18:47.497482+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:44.470061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:44.470061+0000","last_clean_scrub_stamp":"2024-04-26T04:18:44.470061+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:42:56.629937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":12,"seq":51539607557,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27664,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344176,"statfs":{"total":96636764160,"available":96608436224,"internally_reserved":0,"allocated":663552,"data_stored":566639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":12,"seq":51539607556,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":131072,"data_stored":37711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607556,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":131072,"data_stored":37711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":131072,"data_stored":37711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640259,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27152,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344688,"statfs":{"total":96636764160,"available":96608960512,"internally_reserved":0,"allocated":131072,"data_stored":37711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640259,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28112,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343728,"statfs":{"total":96636764160,"available":96607977472,"internally_reserved":0,"allocated":663552,"data_stored":566639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T04:18:50.843 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T04:18:50.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.954+0000 7f0327efe640 10 monclient: get_monmap_and_config 2024-04-26T04:18:50.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.954+0000 7f0327efe640 10 monclient: build_initial_monmap 2024-04-26T04:18:50.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.954+0000 7f0327efe640 10 monclient: monmap: 2024-04-26T04:18:50.961 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:50.961 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:50.961 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:50.960972+0000 2024-04-26T04:18:50.961 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:50.960972+0000 2024-04-26T04:18:50.961 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:50.961 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:50.961 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:18:50.961 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:18:50.961 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:18:50.962 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:18:50.962 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:18:50.962 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:50.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.954+0000 7f0327efe640 1 Processor -- start 2024-04-26T04:18:50.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.954+0000 7f0327efe640 1 -- start start 2024-04-26T04:18:50.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.954+0000 7f0327efe640 10 monclient: init 2024-04-26T04:18:50.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.958+0000 7f0327efe640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:50.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.958+0000 7f0327efe640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-04-26T04:18:50.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.958+0000 7f0327efe640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f0320072450 0x7f0320133c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:50.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.958+0000 7f0327efe640 10 monclient(hunting): picked mon.noname-c con 0x7f0320072450 addr v2:172.21.15.186:3300/0 2024-04-26T04:18:50.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.958+0000 7f0327efe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0320072830 addr v1:172.21.15.63:6789/0 2024-04-26T04:18:50.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.958+0000 7f0327efe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0320072cb0 addr v1:172.21.15.186:6789/0 2024-04-26T04:18:50.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.958+0000 7f0327efe640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03200a2250 con 0x7f0320072830 2024-04-26T04:18:50.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.958+0000 7f0327efe640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f032006e660 con 0x7f0320072cb0 2024-04-26T04:18:50.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.958+0000 7f0327efe640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:50.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.958+0000 7f0327efe640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f032009f650 con 0x7f0320072450 2024-04-26T04:18:50.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.958+0000 7f0327efe640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:50.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.958+0000 7f0327efe640 10 monclient(hunting): authenticate will time out at 962.668614s 2024-04-26T04:18:50.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.958+0000 7f0325499640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f0320072830 0x7f03200a1430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:32798/0 (socket says 172.21.15.63:32798) 2024-04-26T04:18:50.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.958+0000 7f0325499640 1 -- 172.21.15.63:0/177546313 learned_addr learned my addr 172.21.15.63:0/177546313 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-04-26T04:18:50.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.958+0000 7f0325c9a640 1 --2- 172.21.15.63:0/177546313 >> v2:172.21.15.186:3300/0 conn(0x7f0320072450 0x7f0320133c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:50.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.958+0000 7f0325c9a640 10 monclient(hunting): get_auth_request con 0x7f0320072450 auth_method 0 2024-04-26T04:18:50.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.958+0000 7f0325c9a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:50.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.958+0000 7f0325c9a640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:50.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.958+0000 7f0325c9a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:50.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.958+0000 7f0325c9a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:50.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.958+0000 7f0325c9a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:50.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.958+0000 7f0325c9a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:50.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.958+0000 7f0325c9a640 10 monclient(hunting): handle_auth_done global_id 4282 payload 274 2024-04-26T04:18:50.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.958+0000 7f0325c9a640 1 -- 172.21.15.63:0/177546313 >> v1:172.21.15.186:6789/0 conn(0x7f0320072cb0 legacy=0x7f03200a1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:50.967 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.958+0000 7f0325c9a640 1 -- 172.21.15.63:0/177546313 >> v1:172.21.15.63:6789/0 conn(0x7f0320072830 legacy=0x7f03200a1430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:50.967 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.958+0000 7f0325c9a640 10 monclient: _finish_hunting 0 2024-04-26T04:18:50.967 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.958+0000 7f0325c9a640 1 monclient: found mon.noname-c 2024-04-26T04:18:50.967 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.958+0000 7f0325c9a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:50.967 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.958+0000 7f0325c9a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.186:3300/0 2024-04-26T04:18:50.967 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.958+0000 7f0325c9a640 1 -- 172.21.15.63:0/177546313 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f032006e940 con 0x7f0320072450 2024-04-26T04:18:50.967 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.962+0000 7f0325c9a640 1 -- 172.21.15.63:0/177546313 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0320072450 msgr2=0x7f0320133c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:50.967 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.962+0000 7f0325c9a640 1 --2- 172.21.15.63:0/177546313 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0320072450 0x7f0320133c70 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f031c0097c0 tx=0x7f031c01c2e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f9f60a4d28712d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:50.969 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.962+0000 7f0324c98640 1 -- 172.21.15.63:0/177546313 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f031c0b0070 con 0x7f0320072450 2024-04-26T04:18:50.969 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.962+0000 7f0324c98640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:50.969 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.962+0000 7f0324c98640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:18:50.969 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.962+0000 7f0324c98640 10 monclient: dump: 2024-04-26T04:18:50.969 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:50.969 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:50.969 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:50.970 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:50.970 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:50.970 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:50.970 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:50.970 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:50.970 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:50.971 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:50.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.962+0000 7f0324c98640 10 monclient: _finish_auth 0 2024-04-26T04:18:50.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.962+0000 7f0324c98640 10 monclient: _check_auth_tickets 2024-04-26T04:18:50.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.962+0000 7f0324c98640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:50.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.962+0000 7f0324c98640 1 -- 172.21.15.63:0/177546313 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f031c0ab600 con 0x7f0320072450 2024-04-26T04:18:50.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.962+0000 7f0324c98640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:50.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.962+0000 7f0324c98640 1 -- 172.21.15.63:0/177546313 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f031c0ab9d0 con 0x7f0320072450 2024-04-26T04:18:50.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.962+0000 7f0324c98640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:50.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.962+0000 7f0324c98640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T04:18:50.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.962+0000 7f0324c98640 10 monclient: dump: 2024-04-26T04:18:50.972 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:50.972 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:50.972 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:50.972 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:50.972 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:50.972 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:50.972 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:50.972 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:50.973 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:50.973 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:50.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.962+0000 7f0327efe640 5 monclient: authenticate success, global_id 4282 2024-04-26T04:18:50.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.962+0000 7f0327efe640 10 monclient: get_monmap_and_config success 2024-04-26T04:18:50.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.962+0000 7f0327efe640 4 set_mon_vals no callback set 2024-04-26T04:18:50.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.962+0000 7f0327efe640 10 monclient: shutdown 2024-04-26T04:18:50.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.962+0000 7f0327efe640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:50.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.962+0000 7f0327efe640 1 -- 172.21.15.63:0/177546313 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0320072450 msgr2=0x7f0320133c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:50.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.962+0000 7f0327efe640 1 --2- 172.21.15.63:0/177546313 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0320072450 0x7f0320133c70 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f031c0097c0 tx=0x7f031c01c2e0 comp rx=0 tx=0).stop 2024-04-26T04:18:50.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.962+0000 7f0327efe640 1 -- 172.21.15.63:0/177546313 shutdown_connections 2024-04-26T04:18:50.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.962+0000 7f0327efe640 1 --2- 172.21.15.63:0/177546313 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0320072450 0x7f0320133c70 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:50.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.962+0000 7f0327efe640 1 -- 172.21.15.63:0/177546313 >> 172.21.15.63:0/177546313 conn(0x7f03200a6290 msgr2=0x7f03200a2520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:50.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.962+0000 7f0327efe640 1 -- 172.21.15.63:0/177546313 shutdown_connections 2024-04-26T04:18:50.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.962+0000 7f0327efe640 1 -- 172.21.15.63:0/177546313 wait complete. 2024-04-26T04:18:50.980 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.966+0000 7f0327efe640 10 monclient: build_initial_monmap 2024-04-26T04:18:50.980 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.966+0000 7f0327efe640 10 monclient: monmap: 2024-04-26T04:18:50.980 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:50.980 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:18:50.980 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:18:50.980 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:18:50.981 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:50.981 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:50.981 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:18:50.981 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:18:50.981 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:18:50.981 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:50.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.966+0000 7f0327efe640 1 Processor -- start 2024-04-26T04:18:50.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.966+0000 7f0327efe640 1 -- start start 2024-04-26T04:18:50.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.966+0000 7f0327efe640 10 monclient: init 2024-04-26T04:18:50.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.966+0000 7f0327efe640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:50.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.966+0000 7f0327efe640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T04:18:50.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.966+0000 7f0327efe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0320072450 0x7f03201cedc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:50.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.966+0000 7f0327efe640 10 monclient(hunting): picked mon.noname-b con 0x7f0320072450 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:18:50.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.966+0000 7f0327efe640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f0320072830 0x7f03201304b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:50.982 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.966+0000 7f0327efe640 10 monclient(hunting): picked mon.noname-c con 0x7f0320072830 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:50.982 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.966+0000 7f0327efe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0320072cb0 0x7f03201309f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:50.982 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.966+0000 7f0327efe640 10 monclient(hunting): picked mon.noname-a con 0x7f0320072cb0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:18:50.982 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.966+0000 7f0327efe640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:50.982 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.966+0000 7f0327efe640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f03201429c0 con 0x7f0320072cb0 2024-04-26T04:18:50.982 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.966+0000 7f0327efe640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:50.982 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.966+0000 7f0327efe640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0320142840 con 0x7f0320072450 2024-04-26T04:18:50.982 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.966+0000 7f0327efe640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:50.982 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.966+0000 7f0327efe640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f0320142b40 con 0x7f0320072830 2024-04-26T04:18:50.982 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.966+0000 7f0327efe640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:50.982 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.966+0000 7f0327efe640 10 monclient(hunting): authenticate will time out at 962.677421s 2024-04-26T04:18:50.982 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.966+0000 7f032649b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0320072cb0 0x7f03201309f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:50.982 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.966+0000 7f0325499640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f0320072830 0x7f03201304b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:50.982 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.966+0000 7f032649b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0320072cb0 0x7f03201309f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:39266/0 (socket says 172.21.15.63:39266) 2024-04-26T04:18:50.983 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.966+0000 7f032649b640 1 -- 172.21.15.63:0/2755865544 learned_addr learned my addr 172.21.15.63:0/2755865544 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:18:50.983 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.966+0000 7f032649b640 10 monclient(hunting): get_auth_request con 0x7f0320072cb0 auth_method 0 2024-04-26T04:18:50.983 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.966+0000 7f032649b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:50.983 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.966+0000 7f032649b640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:50.983 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.966+0000 7f032649b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:50.983 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.966+0000 7f0325499640 10 monclient(hunting): get_auth_request con 0x7f0320072830 auth_method 0 2024-04-26T04:18:50.983 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.966+0000 7f0325499640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:50.983 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.966+0000 7f0325499640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:50.983 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.966+0000 7f0325499640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:50.983 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.966+0000 7f032649b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:50.983 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.966+0000 7f032649b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:50.983 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.966+0000 7f032649b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:50.983 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.966+0000 7f0325499640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:50.983 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.966+0000 7f0325499640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:50.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.966+0000 7f0325499640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:50.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.966+0000 7f0325c9a640 1 --2- 172.21.15.63:0/2755865544 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0320072450 0x7f03201cedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:50.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.966+0000 7f0325c9a640 10 monclient(hunting): get_auth_request con 0x7f0320072450 auth_method 0 2024-04-26T04:18:50.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.966+0000 7f0325c9a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:50.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.966+0000 7f0325c9a640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:50.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.966+0000 7f0325c9a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:50.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.966+0000 7f032649b640 10 monclient(hunting): handle_auth_done global_id 4299 payload 824 2024-04-26T04:18:50.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.966+0000 7f032649b640 1 -- 172.21.15.63:0/2755865544 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f0320072830 msgr2=0x7f03201304b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:50.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.966+0000 7f032649b640 1 --2- 172.21.15.63:0/2755865544 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f0320072830 0x7f03201304b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:50.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.966+0000 7f032649b640 1 -- 172.21.15.63:0/2755865544 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0320072450 msgr2=0x7f03201cedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:50.985 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.966+0000 7f032649b640 1 --2- 172.21.15.63:0/2755865544 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0320072450 0x7f03201cedc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:50.985 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.966+0000 7f032649b640 10 monclient: _finish_hunting 0 2024-04-26T04:18:50.985 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.966+0000 7f032649b640 1 monclient: found mon.noname-a 2024-04-26T04:18:50.985 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.966+0000 7f032649b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:50.985 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.966+0000 7f032649b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:18:50.985 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.966+0000 7f032649b640 1 -- 172.21.15.63:0/2755865544 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f032012ef80 con 0x7f0320072cb0 2024-04-26T04:18:50.985 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.970+0000 7f032649b640 1 -- 172.21.15.63:0/2755865544 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0320072cb0 msgr2=0x7f03201309f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:50.986 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.970+0000 7f032649b640 1 --2- 172.21.15.63:0/2755865544 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0320072cb0 0x7f03201309f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f031800c780 tx=0x7f031800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:50.986 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.970+0000 7f03127fc640 1 -- 172.21.15.63:0/2755865544 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f031800ac60 con 0x7f0320072cb0 2024-04-26T04:18:50.986 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.970+0000 7f03127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:50.986 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.970+0000 7f03127fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:18:50.986 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.970+0000 7f03127fc640 10 monclient: dump: 2024-04-26T04:18:50.986 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:50.986 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:50.986 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:50.986 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:50.986 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:50.986 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:50.986 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:50.987 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:50.987 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:50.987 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:50.987 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.970+0000 7f03127fc640 10 monclient: _finish_auth 0 2024-04-26T04:18:50.987 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.970+0000 7f03127fc640 10 monclient: _check_auth_tickets 2024-04-26T04:18:50.987 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.970+0000 7f03127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:50.987 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.970+0000 7f03127fc640 1 -- 172.21.15.63:0/2755865544 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f031800ae00 con 0x7f0320072cb0 2024-04-26T04:18:50.987 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.970+0000 7f0327efe640 5 monclient: authenticate success, global_id 4299 2024-04-26T04:18:50.987 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.970+0000 7f03127fc640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:50.987 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.970+0000 7f03127fc640 1 -- 172.21.15.63:0/2755865544 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0318004bb0 con 0x7f0320072cb0 2024-04-26T04:18:50.987 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.970+0000 7f03127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:50.987 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.970+0000 7f03127fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:18:50.988 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.970+0000 7f03127fc640 10 monclient: dump: 2024-04-26T04:18:50.988 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:50.988 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:50.988 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:50.988 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:50.988 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:50.988 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:50.988 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:50.988 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:50.988 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:50.988 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:50.988 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.970+0000 7f0327efe640 10 monclient: _renew_subs 2024-04-26T04:18:50.989 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.970+0000 7f0313fff640 4 set_mon_vals no callback set 2024-04-26T04:18:50.989 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.970+0000 7f0327efe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:50.989 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.970+0000 7f0327efe640 1 -- 172.21.15.63:0/2755865544 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f032009fd40 con 0x7f0320072cb0 2024-04-26T04:18:50.989 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.970+0000 7f0327efe640 10 monclient: _renew_subs 2024-04-26T04:18:50.989 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.970+0000 7f0327efe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:50.989 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.970+0000 7f0327efe640 1 -- 172.21.15.63:0/2755865544 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03201cf4e0 con 0x7f0320072cb0 2024-04-26T04:18:50.989 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.970+0000 7f0327efe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:50.989 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.970+0000 7f0327efe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:50.989 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.970+0000 7f0327efe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:50.989 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.970+0000 7f0327efe640 1 -- 172.21.15.63:0/2755865544 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0320062e10 con 0x7f0320072cb0 2024-04-26T04:18:50.989 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.970+0000 7f03127fc640 1 -- 172.21.15.63:0/2755865544 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f0318006b60 con 0x7f0320072cb0 2024-04-26T04:18:50.989 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.970+0000 7f03127fc640 1 --2- 172.21.15.63:0/2755865544 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f0304035ec0 0x7f0304038380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:50.989 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.970+0000 7f03127fc640 1 -- 172.21.15.63:0/2755865544 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7f0318048f40 con 0x7f0320072cb0 2024-04-26T04:18:50.990 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.970+0000 7f0325c9a640 1 --2- 172.21.15.63:0/2755865544 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f0304035ec0 0x7f0304038380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:50.990 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.970+0000 7f0325c9a640 10 monclient: get_auth_request con 0x7f0304035ec0 auth_method 0 2024-04-26T04:18:50.990 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.974+0000 7f0325c9a640 1 --2- 172.21.15.63:0/2755865544 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f0304035ec0 0x7f0304038380 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f031c0ab000 tx=0x7f031c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:50.990 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.974+0000 7f03127fc640 1 -- 172.21.15.63:0/2755865544 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f0318020070 con 0x7f0320072cb0 2024-04-26T04:18:50.990 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.974+0000 7f03127fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:50.990 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:50.974+0000 7f03127fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:18:51.134 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.130+0000 7f0327efe640 1 -- 172.21.15.63:0/2755865544 --> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0320072290 con 0x7f0304035ec0 2024-04-26T04:18:51.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.130+0000 7f03127fc640 1 -- 172.21.15.63:0/2755865544 <== mgr.4105 v2:172.21.15.186:6824/2583706639 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33807 (secure 0 0 0) 0x7f031c0ab560 con 0x7f0304035ec0 2024-04-26T04:18:51.138 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-26T04:18:51.138 INFO:teuthology.orchestra.run.smithi063.stderr:dumped all 2024-04-26T04:18:51.139 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.134+0000 7f0327efe640 1 -- 172.21.15.63:0/2755865544 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f0304035ec0 msgr2=0x7f0304038380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:51.139 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.134+0000 7f0327efe640 1 --2- 172.21.15.63:0/2755865544 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f0304035ec0 0x7f0304038380 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f031c0ab000 tx=0x7f031c008040 comp rx=0 tx=0).stop 2024-04-26T04:18:51.139 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.134+0000 7f0327efe640 10 monclient: shutdown 2024-04-26T04:18:51.140 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.134+0000 7f0327efe640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:51.140 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.134+0000 7f0327efe640 1 -- 172.21.15.63:0/2755865544 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0320072cb0 msgr2=0x7f03201309f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:51.140 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.134+0000 7f0327efe640 1 --2- 172.21.15.63:0/2755865544 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0320072cb0 0x7f03201309f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f031800c780 tx=0x7f031800cc40 comp rx=0 tx=0).stop 2024-04-26T04:18:51.140 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.134+0000 7f0327efe640 1 -- 172.21.15.63:0/2755865544 shutdown_connections 2024-04-26T04:18:51.140 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.134+0000 7f0327efe640 1 --2- 172.21.15.63:0/2755865544 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f0304035ec0 0x7f0304038380 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:51.140 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.134+0000 7f0327efe640 1 --2- 172.21.15.63:0/2755865544 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0320072cb0 0x7f03201309f0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:51.140 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.134+0000 7f0327efe640 1 --2- 172.21.15.63:0/2755865544 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f0320072830 0x7f03201304b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:51.140 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.134+0000 7f0327efe640 1 --2- 172.21.15.63:0/2755865544 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0320072450 0x7f03201cedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:51.140 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.134+0000 7f0327efe640 1 -- 172.21.15.63:0/2755865544 >> 172.21.15.63:0/2755865544 conn(0x7f03200a6290 msgr2=0x7f03200a3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:51.140 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.134+0000 7f0327efe640 1 -- 172.21.15.63:0/2755865544 shutdown_connections 2024-04-26T04:18:51.140 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.134+0000 7f0327efe640 1 -- 172.21.15.63:0/2755865544 wait complete. 2024-04-26T04:18:51.155 INFO:teuthology.orchestra.run.smithi063.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-04-26T04:18:49.446817+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":164372,"kb_used_data":1808,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566066668,"statfs":{"total":579820584960,"available":579652268032,"internally_reserved":0,"allocated":1851392,"data_stored":1284122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9567,"internal_metadata":165927585},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-26T04:18:47.497690+0000","last_change":"2024-04-26T04:18:47.131857+0000","last_active":"2024-04-26T04:18:47.497690+0000","last_peered":"2024-04-26T04:18:47.497690+0000","last_clean":"2024-04-26T04:18:47.497690+0000","last_became_active":"2024-04-26T04:18:47.131175+0000","last_became_peered":"2024-04-26T04:18:47.131175+0000","last_unstale":"2024-04-26T04:18:47.497690+0000","last_undegraded":"2024-04-26T04:18:47.497690+0000","last_fullsized":"2024-04-26T04:18:47.497690+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_clean_scrub_stamp":"2024-04-26T04:18:45.482773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:35:03.860574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-26T04:18:47.493565+0000","last_change":"2024-04-26T04:18:46.499700+0000","last_active":"2024-04-26T04:18:47.493565+0000","last_peered":"2024-04-26T04:18:47.493565+0000","last_clean":"2024-04-26T04:18:47.493565+0000","last_became_active":"2024-04-26T04:18:46.498983+0000","last_became_peered":"2024-04-26T04:18:46.498983+0000","last_unstale":"2024-04-26T04:18:47.493565+0000","last_undegraded":"2024-04-26T04:18:47.493565+0000","last_fullsized":"2024-04-26T04:18:47.493565+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_clean_scrub_stamp":"2024-04-26T04:18:45.482773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:51:56.740670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-26T04:18:47.497623+0000","last_change":"2024-04-26T04:18:46.507904+0000","last_active":"2024-04-26T04:18:47.497623+0000","last_peered":"2024-04-26T04:18:47.497623+0000","last_clean":"2024-04-26T04:18:47.497623+0000","last_became_active":"2024-04-26T04:18:46.507335+0000","last_became_peered":"2024-04-26T04:18:46.507335+0000","last_unstale":"2024-04-26T04:18:47.497623+0000","last_undegraded":"2024-04-26T04:18:47.497623+0000","last_fullsized":"2024-04-26T04:18:47.497623+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_clean_scrub_stamp":"2024-04-26T04:18:45.482773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:52:50.029369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-26T04:18:47.493398+0000","last_change":"2024-04-26T04:18:46.506682+0000","last_active":"2024-04-26T04:18:47.493398+0000","last_peered":"2024-04-26T04:18:47.493398+0000","last_clean":"2024-04-26T04:18:47.493398+0000","last_became_active":"2024-04-26T04:18:46.506209+0000","last_became_peered":"2024-04-26T04:18:46.506209+0000","last_unstale":"2024-04-26T04:18:47.493398+0000","last_undegraded":"2024-04-26T04:18:47.493398+0000","last_fullsized":"2024-04-26T04:18:47.493398+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_clean_scrub_stamp":"2024-04-26T04:18:45.482773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:20:00.475307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-26T04:18:46.506418+0000","last_change":"2024-04-26T04:18:46.504685+0000","last_active":"2024-04-26T04:18:46.506418+0000","last_peered":"2024-04-26T04:18:46.506418+0000","last_clean":"2024-04-26T04:18:46.506418+0000","last_became_active":"2024-04-26T04:18:46.504292+0000","last_became_peered":"2024-04-26T04:18:46.504292+0000","last_unstale":"2024-04-26T04:18:46.506418+0000","last_undegraded":"2024-04-26T04:18:46.506418+0000","last_fullsized":"2024-04-26T04:18:46.506418+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_clean_scrub_stamp":"2024-04-26T04:18:45.482773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:56:59.840987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-26T04:18:46.506175+0000","last_change":"2024-04-26T04:18:46.504010+0000","last_active":"2024-04-26T04:18:46.506175+0000","last_peered":"2024-04-26T04:18:46.506175+0000","last_clean":"2024-04-26T04:18:46.506175+0000","last_became_active":"2024-04-26T04:18:46.503343+0000","last_became_peered":"2024-04-26T04:18:46.503343+0000","last_unstale":"2024-04-26T04:18:46.506175+0000","last_undegraded":"2024-04-26T04:18:46.506175+0000","last_fullsized":"2024-04-26T04:18:46.506175+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_clean_scrub_stamp":"2024-04-26T04:18:45.482773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:43:27.096237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-26T04:18:47.497466+0000","last_change":"2024-04-26T04:18:46.503233+0000","last_active":"2024-04-26T04:18:47.497466+0000","last_peered":"2024-04-26T04:18:47.497466+0000","last_clean":"2024-04-26T04:18:47.497466+0000","last_became_active":"2024-04-26T04:18:46.502682+0000","last_became_peered":"2024-04-26T04:18:46.502682+0000","last_unstale":"2024-04-26T04:18:47.497466+0000","last_undegraded":"2024-04-26T04:18:47.497466+0000","last_fullsized":"2024-04-26T04:18:47.497466+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_clean_scrub_stamp":"2024-04-26T04:18:45.482773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:26:15.760818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-04-26T04:18:46.506086+0000","last_change":"2024-04-26T04:18:46.504844+0000","last_active":"2024-04-26T04:18:46.506086+0000","last_peered":"2024-04-26T04:18:46.506086+0000","last_clean":"2024-04-26T04:18:46.506086+0000","last_became_active":"2024-04-26T04:18:46.504332+0000","last_became_peered":"2024-04-26T04:18:46.504332+0000","last_unstale":"2024-04-26T04:18:46.506086+0000","last_undegraded":"2024-04-26T04:18:46.506086+0000","last_fullsized":"2024-04-26T04:18:46.506086+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_clean_scrub_stamp":"2024-04-26T04:18:45.482773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T14:02:57.619500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":164,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-26T04:18:47.497482+0000","last_change":"2024-04-26T04:18:45.505818+0000","last_active":"2024-04-26T04:18:47.497482+0000","last_peered":"2024-04-26T04:18:47.497482+0000","last_clean":"2024-04-26T04:18:47.497482+0000","last_became_active":"2024-04-26T04:18:45.504887+0000","last_became_peered":"2024-04-26T04:18:45.504887+0000","last_unstale":"2024-04-26T04:18:47.497482+0000","last_undegraded":"2024-04-26T04:18:47.497482+0000","last_fullsized":"2024-04-26T04:18:47.497482+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:44.470061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:44.470061+0000","last_clean_scrub_stamp":"2024-04-26T04:18:44.470061+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:42:56.629937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":12,"seq":51539607557,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27664,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344176,"statfs":{"total":96636764160,"available":96608436224,"internally_reserved":0,"allocated":663552,"data_stored":566639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":12,"seq":51539607556,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":131072,"data_stored":37711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607556,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":131072,"data_stored":37711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":131072,"data_stored":37711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640259,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27152,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344688,"statfs":{"total":96636764160,"available":96608960512,"internally_reserved":0,"allocated":131072,"data_stored":37711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640259,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28112,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343728,"statfs":{"total":96636764160,"available":96607977472,"internally_reserved":0,"allocated":663552,"data_stored":566639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T04:18:51.156 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-26T04:18:51.156 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-26T04:18:51.156 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-26T04:18:51.156 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-26T04:18:51.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.270+0000 7fa5fb847640 10 monclient: get_monmap_and_config 2024-04-26T04:18:51.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.270+0000 7fa5fb847640 10 monclient: build_initial_monmap 2024-04-26T04:18:51.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.270+0000 7fa5fb847640 10 monclient: monmap: 2024-04-26T04:18:51.275 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:51.275 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:51.275 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:51.274752+0000 2024-04-26T04:18:51.276 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:51.274752+0000 2024-04-26T04:18:51.276 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:51.276 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:51.276 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:18:51.276 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:18:51.276 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:18:51.276 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:18:51.276 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:18:51.276 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:51.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.270+0000 7fa5fb847640 1 Processor -- start 2024-04-26T04:18:51.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.270+0000 7fa5fb847640 1 -- start start 2024-04-26T04:18:51.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.270+0000 7fa5fb847640 10 monclient: init 2024-04-26T04:18:51.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.270+0000 7fa5fb847640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:51.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.270+0000 7fa5fb847640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-04-26T04:18:51.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.270+0000 7fa5fb847640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa5f40a2cd0 addr v1:172.21.15.186:6789/0 2024-04-26T04:18:51.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.270+0000 7fa5fb847640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fa5f40a30b0 0x7fa5f413c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:51.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.270+0000 7fa5fb847640 10 monclient(hunting): picked mon.noname-b con 0x7fa5f40a30b0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:51.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.270+0000 7fa5fb847640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa5f413c540 addr v1:172.21.15.63:6789/0 2024-04-26T04:18:51.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.270+0000 7fa5fb847640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5f40a4ee0 con 0x7fa5f413c540 2024-04-26T04:18:51.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.270+0000 7fa5fb847640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5f40a0fb0 con 0x7fa5f40a2cd0 2024-04-26T04:18:51.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.270+0000 7fa5fb847640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:51.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.270+0000 7fa5fb847640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7fa5f40a5380 con 0x7fa5f40a30b0 2024-04-26T04:18:51.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.270+0000 7fa5fb847640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:51.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.270+0000 7fa5fb847640 10 monclient(hunting): authenticate will time out at 962.982367s 2024-04-26T04:18:51.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.270+0000 7fa5f8de2640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fa5f40a30b0 0x7fa5f413c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:51.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.270+0000 7fa5f8de2640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fa5f40a30b0 0x7fa5f413c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:36904/0 (socket says 172.21.15.63:36904) 2024-04-26T04:18:51.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.270+0000 7fa5f8de2640 1 -- 172.21.15.63:0/798494251 learned_addr learned my addr 172.21.15.63:0/798494251 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:18:51.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.270+0000 7fa5f8de2640 10 monclient(hunting): get_auth_request con 0x7fa5f40a30b0 auth_method 0 2024-04-26T04:18:51.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.270+0000 7fa5f8de2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:51.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.270+0000 7fa5f8de2640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:51.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.270+0000 7fa5f8de2640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:51.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.270+0000 7fa5f8de2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:51.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.270+0000 7fa5f8de2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:51.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.270+0000 7fa5f8de2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:51.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.270+0000 7fa5f8de2640 10 monclient(hunting): handle_auth_done global_id 4283 payload 274 2024-04-26T04:18:51.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.270+0000 7fa5f8de2640 1 -- 172.21.15.63:0/798494251 >> v1:172.21.15.186:6789/0 conn(0x7fa5f40a2cd0 legacy=0x7fa5f40a47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:51.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.270+0000 7fa5f8de2640 1 -- 172.21.15.63:0/798494251 >> v1:172.21.15.63:6789/0 conn(0x7fa5f413c540 legacy=0x7fa5f413e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:51.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.270+0000 7fa5f8de2640 10 monclient: _finish_hunting 0 2024-04-26T04:18:51.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.270+0000 7fa5f8de2640 1 monclient: found mon.noname-b 2024-04-26T04:18:51.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.270+0000 7fa5f8de2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:51.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.270+0000 7fa5f8de2640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.63:3301/0 2024-04-26T04:18:51.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.270+0000 7fa5f8de2640 1 -- 172.21.15.63:0/798494251 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5f40a1290 con 0x7fa5f40a30b0 2024-04-26T04:18:51.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.274+0000 7fa5f8de2640 1 -- 172.21.15.63:0/798494251 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fa5f40a30b0 msgr2=0x7fa5f413c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:51.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.274+0000 7fa5f8de2640 1 --2- 172.21.15.63:0/798494251 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fa5f40a30b0 0x7fa5f413c000 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa5e40098a0 tx=0x7fa5e401c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=991b3dceb283336e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:51.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.274+0000 7fa5ebfff640 1 -- 172.21.15.63:0/798494251 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5e40b0070 con 0x7fa5f40a30b0 2024-04-26T04:18:51.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.274+0000 7fa5ebfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:51.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.274+0000 7fa5ebfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T04:18:51.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.274+0000 7fa5ebfff640 10 monclient: dump: 2024-04-26T04:18:51.289 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:51.289 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:51.289 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:51.289 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:51.290 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:51.290 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:51.290 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:51.290 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:51.290 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:51.290 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:51.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.274+0000 7fa5ebfff640 10 monclient: _finish_auth 0 2024-04-26T04:18:51.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.274+0000 7fa5ebfff640 10 monclient: _check_auth_tickets 2024-04-26T04:18:51.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.274+0000 7fa5ebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:51.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.274+0000 7fa5ebfff640 1 -- 172.21.15.63:0/798494251 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5e40ab600 con 0x7fa5f40a30b0 2024-04-26T04:18:51.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.274+0000 7fa5fb847640 5 monclient: authenticate success, global_id 4283 2024-04-26T04:18:51.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.274+0000 7fa5fb847640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T04:18:51.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.274+0000 7fa5ebfff640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:51.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.274+0000 7fa5fb847640 10 monclient: get_monmap_and_config success 2024-04-26T04:18:51.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.274+0000 7fa5fb847640 4 set_mon_vals no callback set 2024-04-26T04:18:51.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.274+0000 7fa5ebfff640 1 -- 172.21.15.63:0/798494251 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5e40ab9d0 con 0x7fa5f40a30b0 2024-04-26T04:18:51.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.274+0000 7fa5ebfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:51.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.274+0000 7fa5ebfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:18:51.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.274+0000 7fa5ebfff640 10 monclient: dump: 2024-04-26T04:18:51.291 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:51.291 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:51.291 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:51.291 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:51.291 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:51.291 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:51.291 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:51.291 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:51.291 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:51.292 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:51.292 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.274+0000 7fa5fb847640 10 monclient: shutdown 2024-04-26T04:18:51.292 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.274+0000 7fa5fb847640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:51.292 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.274+0000 7fa5fb847640 1 -- 172.21.15.63:0/798494251 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fa5f40a30b0 msgr2=0x7fa5f413c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:51.292 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.274+0000 7fa5fb847640 1 --2- 172.21.15.63:0/798494251 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fa5f40a30b0 0x7fa5f413c000 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa5e40098a0 tx=0x7fa5e401c450 comp rx=0 tx=0).stop 2024-04-26T04:18:51.292 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.274+0000 7fa5fb847640 1 -- 172.21.15.63:0/798494251 shutdown_connections 2024-04-26T04:18:51.292 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.274+0000 7fa5fb847640 1 --2- 172.21.15.63:0/798494251 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fa5f40a30b0 0x7fa5f413c000 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:51.292 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.274+0000 7fa5fb847640 1 -- 172.21.15.63:0/798494251 >> 172.21.15.63:0/798494251 conn(0x7fa5f40a5f40 msgr2=0x7fa5f40a6320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:51.292 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.274+0000 7fa5fb847640 1 -- 172.21.15.63:0/798494251 shutdown_connections 2024-04-26T04:18:51.292 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.274+0000 7fa5fb847640 1 -- 172.21.15.63:0/798494251 wait complete. 2024-04-26T04:18:51.292 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.278+0000 7fa5fb847640 10 monclient: build_initial_monmap 2024-04-26T04:18:51.292 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.278+0000 7fa5fb847640 10 monclient: monmap: 2024-04-26T04:18:51.292 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:51.292 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:18:51.292 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:18:51.293 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:18:51.293 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:51.293 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:51.293 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:18:51.293 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:18:51.293 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:18:51.293 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:51.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.278+0000 7fa5fb847640 1 Processor -- start 2024-04-26T04:18:51.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.278+0000 7fa5fb847640 1 -- start start 2024-04-26T04:18:51.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.278+0000 7fa5fb847640 10 monclient: init 2024-04-26T04:18:51.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.278+0000 7fa5fb847640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:51.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.278+0000 7fa5fb847640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T04:18:51.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.278+0000 7fa5fb847640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa5f40a2cd0 0x7fa5f41cf590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:51.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.278+0000 7fa5fb847640 10 monclient(hunting): picked mon.noname-a con 0x7fa5f40a2cd0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:18:51.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.278+0000 7fa5fb847640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5f40a30b0 0x7fa5f406ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:51.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.278+0000 7fa5fb847640 10 monclient(hunting): picked mon.noname-b con 0x7fa5f40a30b0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:18:51.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.278+0000 7fa5fb847640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fa5f413c540 0x7fa5f406f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:51.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.278+0000 7fa5fb847640 10 monclient(hunting): picked mon.noname-c con 0x7fa5f413c540 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:51.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.278+0000 7fa5fb847640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:51.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.278+0000 7fa5fb847640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fa5f4142c00 con 0x7fa5f40a2cd0 2024-04-26T04:18:51.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.278+0000 7fa5fb847640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:51.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.278+0000 7fa5fb847640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa5f4142a80 con 0x7fa5f40a30b0 2024-04-26T04:18:51.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.278+0000 7fa5fb847640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:51.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.278+0000 7fa5fb847640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7fa5f4142d80 con 0x7fa5f413c540 2024-04-26T04:18:51.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.278+0000 7fa5fb847640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:51.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.278+0000 7fa5fb847640 10 monclient(hunting): authenticate will time out at 962.989166s 2024-04-26T04:18:51.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.278+0000 7fa5f95e3640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa5f40a2cd0 0x7fa5f41cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:51.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.278+0000 7fa5f9de4640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fa5f413c540 0x7fa5f406f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:51.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.278+0000 7fa5f95e3640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa5f40a2cd0 0x7fa5f41cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:39270/0 (socket says 172.21.15.63:39270) 2024-04-26T04:18:51.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.278+0000 7fa5f95e3640 1 -- 172.21.15.63:0/4289145111 learned_addr learned my addr 172.21.15.63:0/4289145111 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:18:51.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.278+0000 7fa5f95e3640 10 monclient(hunting): get_auth_request con 0x7fa5f40a2cd0 auth_method 0 2024-04-26T04:18:51.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.278+0000 7fa5f95e3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:51.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.278+0000 7fa5f95e3640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:51.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.278+0000 7fa5f95e3640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:51.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.278+0000 7fa5f9de4640 10 monclient(hunting): get_auth_request con 0x7fa5f413c540 auth_method 0 2024-04-26T04:18:51.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.278+0000 7fa5f9de4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:51.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.278+0000 7fa5f9de4640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:51.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.278+0000 7fa5f9de4640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:51.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.278+0000 7fa5f9de4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:51.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.278+0000 7fa5f9de4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:51.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.278+0000 7fa5f9de4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:51.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.278+0000 7fa5f95e3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:51.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.278+0000 7fa5f95e3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:51.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.278+0000 7fa5f95e3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:51.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.278+0000 7fa5f8de2640 1 --2- 172.21.15.63:0/4289145111 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5f40a30b0 0x7fa5f406ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:51.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.278+0000 7fa5f9de4640 10 monclient(hunting): handle_auth_done global_id 4286 payload 824 2024-04-26T04:18:51.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.278+0000 7fa5f9de4640 1 -- 172.21.15.63:0/4289145111 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5f40a30b0 msgr2=0x7fa5f406ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:51.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.278+0000 7fa5f9de4640 1 --2- 172.21.15.63:0/4289145111 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5f40a30b0 0x7fa5f406ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:51.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.278+0000 7fa5f9de4640 1 -- 172.21.15.63:0/4289145111 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa5f40a2cd0 msgr2=0x7fa5f41cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:51.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.278+0000 7fa5f9de4640 1 --2- 172.21.15.63:0/4289145111 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa5f40a2cd0 0x7fa5f41cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:51.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.278+0000 7fa5f9de4640 10 monclient: _finish_hunting 0 2024-04-26T04:18:51.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.278+0000 7fa5f9de4640 1 monclient: found mon.noname-c 2024-04-26T04:18:51.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.278+0000 7fa5f9de4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:51.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.278+0000 7fa5f9de4640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.63:3301/0 2024-04-26T04:18:51.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.278+0000 7fa5f9de4640 1 -- 172.21.15.63:0/4289145111 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5f40a37d0 con 0x7fa5f413c540 2024-04-26T04:18:51.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.278+0000 7fa5f95e3640 1 --2- 172.21.15.63:0/4289145111 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa5f40a2cd0 0x7fa5f41cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T04:18:51.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.278+0000 7fa5f8de2640 10 monclient: get_auth_request con 0x7fa5f40a30b0 auth_method 0 2024-04-26T04:18:51.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.278+0000 7fa5f8de2640 1 --2- 172.21.15.63:0/4289145111 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5f40a30b0 0x7fa5f406ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T04:18:51.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.278+0000 7fa5f9de4640 1 -- 172.21.15.63:0/4289145111 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fa5f413c540 msgr2=0x7fa5f406f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:51.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.278+0000 7fa5f9de4640 1 --2- 172.21.15.63:0/4289145111 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fa5f413c540 0x7fa5f406f470 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa5ec00c750 tx=0x7fa5ec00cc10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:51.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.282+0000 7fa5e9ffb640 1 -- 172.21.15.63:0/4289145111 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5ec016020 con 0x7fa5f413c540 2024-04-26T04:18:51.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.282+0000 7fa5e9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:51.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.282+0000 7fa5e9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:18:51.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.282+0000 7fa5e9ffb640 10 monclient: dump: 2024-04-26T04:18:51.300 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:51.300 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:51.300 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:51.300 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:51.300 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:51.300 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:51.300 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:51.300 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:51.300 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:51.300 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:51.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.282+0000 7fa5e9ffb640 10 monclient: _finish_auth 0 2024-04-26T04:18:51.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.282+0000 7fa5e9ffb640 10 monclient: _check_auth_tickets 2024-04-26T04:18:51.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.282+0000 7fa5e9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:51.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.282+0000 7fa5e9ffb640 1 -- 172.21.15.63:0/4289145111 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5ec00aa60 con 0x7fa5f413c540 2024-04-26T04:18:51.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.282+0000 7fa5fb847640 5 monclient: authenticate success, global_id 4286 2024-04-26T04:18:51.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.282+0000 7fa5fb847640 10 monclient: _renew_subs 2024-04-26T04:18:51.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.282+0000 7fa5fb847640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:51.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.282+0000 7fa5fb847640 1 -- 172.21.15.63:0/4289145111 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5f4072b40 con 0x7fa5f413c540 2024-04-26T04:18:51.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.282+0000 7fa5e9ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:51.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.282+0000 7fa5e9ffb640 1 -- 172.21.15.63:0/4289145111 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5ec00ae50 con 0x7fa5f413c540 2024-04-26T04:18:51.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.282+0000 7fa5e9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:51.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.282+0000 7fa5e9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:18:51.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.282+0000 7fa5e9ffb640 10 monclient: dump: 2024-04-26T04:18:51.301 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:51.301 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:51.301 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:51.301 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:51.301 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:51.301 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:51.302 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:51.302 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:51.302 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:51.302 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:51.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.282+0000 7fa5eb7fe640 4 set_mon_vals no callback set 2024-04-26T04:18:51.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.282+0000 7fa5fb847640 10 monclient: _renew_subs 2024-04-26T04:18:51.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.282+0000 7fa5fb847640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:51.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.282+0000 7fa5fb847640 1 -- 172.21.15.63:0/4289145111 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5f41c7e20 con 0x7fa5f413c540 2024-04-26T04:18:51.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.282+0000 7fa5fb847640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:51.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.282+0000 7fa5fb847640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:51.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.282+0000 7fa5fb847640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:51.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.282+0000 7fa5fb847640 1 -- 172.21.15.63:0/4289145111 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5f4062e10 con 0x7fa5f413c540 2024-04-26T04:18:51.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.282+0000 7fa5e9ffb640 1 -- 172.21.15.63:0/4289145111 <== mon.2 v2:172.21.15.63:3301/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fa5ec019020 con 0x7fa5f413c540 2024-04-26T04:18:51.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.282+0000 7fa5e9ffb640 1 --2- 172.21.15.63:0/4289145111 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fa5d8036090 0x7fa5d8038550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:51.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.282+0000 7fa5e9ffb640 1 -- 172.21.15.63:0/4289145111 <== mon.2 v2:172.21.15.63:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7fa5ec048eb0 con 0x7fa5f413c540 2024-04-26T04:18:51.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.282+0000 7fa5f95e3640 1 --2- 172.21.15.63:0/4289145111 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fa5d8036090 0x7fa5d8038550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:51.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.282+0000 7fa5f95e3640 10 monclient: get_auth_request con 0x7fa5d8036090 auth_method 0 2024-04-26T04:18:51.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.286+0000 7fa5f95e3640 1 --2- 172.21.15.63:0/4289145111 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fa5d8036090 0x7fa5d8038550 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa5f0004470 tx=0x7fa5f0003700 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:51.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.286+0000 7fa5e9ffb640 1 -- 172.21.15.63:0/4289145111 <== mon.2 v2:172.21.15.63:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fa5ec020020 con 0x7fa5f413c540 2024-04-26T04:18:51.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.286+0000 7fa5e9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:51.304 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.286+0000 7fa5e9ffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:18:51.481 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.474+0000 7fa5fb847640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2024-04-26T04:18:51.482 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.474+0000 7fa5fb847640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2024-04-26T04:18:51.482 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.474+0000 7fa5fb847640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:51.482 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.474+0000 7fa5fb847640 1 -- 172.21.15.63:0/4289145111 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fa5f4071410 con 0x7fa5f413c540 2024-04-26T04:18:51.482 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.474+0000 7fa5e9ffb640 1 -- 172.21.15.63:0/4289145111 <== mon.2 v2:172.21.15.63:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fa5f40a37d0 con 0x7fa5f413c540 2024-04-26T04:18:51.483 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.474+0000 7fa5e9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2024-04-26T04:18:51.483 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.474+0000 7fa5e9ffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:18:51.484 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-26T04:18:51.484 INFO:teuthology.orchestra.run.smithi063.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-26T04:18:51.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.478+0000 7fa5fb847640 1 -- 172.21.15.63:0/4289145111 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fa5d8036090 msgr2=0x7fa5d8038550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:51.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.478+0000 7fa5fb847640 1 --2- 172.21.15.63:0/4289145111 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fa5d8036090 0x7fa5d8038550 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa5f0004470 tx=0x7fa5f0003700 comp rx=0 tx=0).stop 2024-04-26T04:18:51.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.478+0000 7fa5fb847640 10 monclient: shutdown 2024-04-26T04:18:51.486 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.478+0000 7fa5fb847640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:51.486 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.478+0000 7fa5fb847640 1 -- 172.21.15.63:0/4289145111 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fa5f413c540 msgr2=0x7fa5f406f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:51.486 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.478+0000 7fa5fb847640 1 --2- 172.21.15.63:0/4289145111 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fa5f413c540 0x7fa5f406f470 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa5ec00c750 tx=0x7fa5ec00cc10 comp rx=0 tx=0).stop 2024-04-26T04:18:51.486 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.478+0000 7fa5fb847640 1 -- 172.21.15.63:0/4289145111 shutdown_connections 2024-04-26T04:18:51.486 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.478+0000 7fa5fb847640 1 --2- 172.21.15.63:0/4289145111 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fa5d8036090 0x7fa5d8038550 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:51.486 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.478+0000 7fa5fb847640 1 --2- 172.21.15.63:0/4289145111 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fa5f413c540 0x7fa5f406f470 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:51.486 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.478+0000 7fa5fb847640 1 --2- 172.21.15.63:0/4289145111 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5f40a30b0 0x7fa5f406ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:51.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.478+0000 7fa5fb847640 1 --2- 172.21.15.63:0/4289145111 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa5f40a2cd0 0x7fa5f41cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:51.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.478+0000 7fa5fb847640 1 -- 172.21.15.63:0/4289145111 >> 172.21.15.63:0/4289145111 conn(0x7fa5f40a5f40 msgr2=0x7fa5f40a7340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:51.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.478+0000 7fa5fb847640 1 -- 172.21.15.63:0/4289145111 shutdown_connections 2024-04-26T04:18:51.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.478+0000 7fa5fb847640 1 -- 172.21.15.63:0/4289145111 wait complete. 2024-04-26T04:18:51.498 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-26T04:18:51.499 INFO:teuthology.run_tasks:Running task mon_thrash... 2024-04-26T04:18:51.513 INFO:tasks.mon_thrash:Beginning mon_thrash... 2024-04-26T04:18:51.513 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T04:18:51.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.630+0000 7f47d6ed7640 10 monclient: get_monmap_and_config 2024-04-26T04:18:51.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.630+0000 7f47d6ed7640 10 monclient: build_initial_monmap 2024-04-26T04:18:51.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.630+0000 7f47d6ed7640 10 monclient: monmap: 2024-04-26T04:18:51.637 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:51.637 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:51.637 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:51.636653+0000 2024-04-26T04:18:51.637 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:51.636653+0000 2024-04-26T04:18:51.637 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:51.637 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:51.637 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:18:51.638 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:18:51.638 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:18:51.638 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:18:51.638 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:18:51.638 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:51.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.630+0000 7f47d6ed7640 1 Processor -- start 2024-04-26T04:18:51.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.630+0000 7f47d6ed7640 1 -- start start 2024-04-26T04:18:51.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.630+0000 7f47d6ed7640 10 monclient: init 2024-04-26T04:18:51.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.634+0000 7f47d6ed7640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:51.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.634+0000 7f47d6ed7640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-04-26T04:18:51.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.634+0000 7f47d6ed7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f47d0072450 addr v1:172.21.15.186:6789/0 2024-04-26T04:18:51.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.634+0000 7f47d6ed7640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f47d0133c70 0x7f47d013bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:51.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.634+0000 7f47d6ed7640 10 monclient(hunting): picked mon.noname-c con 0x7f47d0133c70 addr v2:172.21.15.186:3300/0 2024-04-26T04:18:51.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.634+0000 7f47d6ed7640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f47d013c530 0x7f47d013e920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:51.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.634+0000 7f47d6ed7640 10 monclient(hunting): picked mon.noname-a con 0x7f47d013c530 addr v2:172.21.15.63:3300/0 2024-04-26T04:18:51.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.634+0000 7f47d6ed7640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47d01340f0 con 0x7f47d0072450 2024-04-26T04:18:51.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.634+0000 7f47d6ed7640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:51.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.634+0000 7f47d6ed7640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7f47d009f650 con 0x7f47d013c530 2024-04-26T04:18:51.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.634+0000 7f47d6ed7640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:51.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.634+0000 7f47d6ed7640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f47d0072f40 con 0x7f47d0133c70 2024-04-26T04:18:51.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.634+0000 7f47d6ed7640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:51.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.634+0000 7f47d6ed7640 10 monclient(hunting): authenticate will time out at 963.344495s 2024-04-26T04:18:51.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.634+0000 7f47d5474640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f47d013c530 0x7f47d013e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:51.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.634+0000 7f47d5474640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f47d013c530 0x7f47d013e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:39278/0 (socket says 172.21.15.63:39278) 2024-04-26T04:18:51.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.634+0000 7f47d5474640 1 -- 172.21.15.63:0/2431193699 learned_addr learned my addr 172.21.15.63:0/2431193699 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:18:51.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.634+0000 7f47d5474640 10 monclient(hunting): get_auth_request con 0x7f47d013c530 auth_method 0 2024-04-26T04:18:51.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.634+0000 7f47d5474640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:51.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.634+0000 7f47d5474640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:51.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.634+0000 7f47d5474640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:51.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.634+0000 7f47d5474640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:51.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.634+0000 7f47d5474640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:51.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.634+0000 7f47d5474640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:51.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.634+0000 7f47cffff640 1 --2- 172.21.15.63:0/2431193699 >> v2:172.21.15.186:3300/0 conn(0x7f47d0133c70 0x7f47d013bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:51.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.634+0000 7f47cffff640 10 monclient(hunting): get_auth_request con 0x7f47d0133c70 auth_method 0 2024-04-26T04:18:51.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.634+0000 7f47cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:51.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.634+0000 7f47cffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:51.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.634+0000 7f47cffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:51.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.634+0000 7f47d5474640 10 monclient(hunting): handle_auth_done global_id 4305 payload 274 2024-04-26T04:18:51.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.634+0000 7f47d5474640 1 -- 172.21.15.63:0/2431193699 >> v2:172.21.15.186:3300/0 conn(0x7f47d0133c70 msgr2=0x7f47d013bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:51.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.634+0000 7f47d5474640 1 --2- 172.21.15.63:0/2431193699 >> v2:172.21.15.186:3300/0 conn(0x7f47d0133c70 0x7f47d013bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:51.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.634+0000 7f47d5474640 1 -- 172.21.15.63:0/2431193699 >> v1:172.21.15.186:6789/0 conn(0x7f47d0072450 legacy=0x7f47d0072830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:51.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.634+0000 7f47d5474640 10 monclient: _finish_hunting 0 2024-04-26T04:18:51.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.634+0000 7f47d5474640 1 monclient: found mon.noname-a 2024-04-26T04:18:51.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.634+0000 7f47d5474640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:51.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.634+0000 7f47d5474640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:18:51.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.634+0000 7f47d5474640 1 -- 172.21.15.63:0/2431193699 --> v2:172.21.15.63:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47d006e660 con 0x7f47d013c530 2024-04-26T04:18:51.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.638+0000 7f47d5474640 1 -- 172.21.15.63:0/2431193699 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f47d013c530 msgr2=0x7f47d013e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:51.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.638+0000 7f47d5474640 1 --2- 172.21.15.63:0/2431193699 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f47d013c530 0x7f47d013e920 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f47d006e2f0 tx=0x7f47c001c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bae9434764ddd615 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:51.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.638+0000 7f47cf7fe640 1 -- 172.21.15.63:0/2431193699 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47c00ab600 con 0x7f47d013c530 2024-04-26T04:18:51.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.638+0000 7f47cf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:51.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.638+0000 7f47cf7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:18:51.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.638+0000 7f47cf7fe640 10 monclient: dump: 2024-04-26T04:18:51.651 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:51.651 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:51.651 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:51.652 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:51.652 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:51.652 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:51.652 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:51.652 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:51.652 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:51.652 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:51.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.638+0000 7f47cf7fe640 10 monclient: _finish_auth 0 2024-04-26T04:18:51.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.638+0000 7f47cf7fe640 10 monclient: _check_auth_tickets 2024-04-26T04:18:51.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.638+0000 7f47cf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:51.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.638+0000 7f47cf7fe640 1 -- 172.21.15.63:0/2431193699 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47c00b0070 con 0x7f47d013c530 2024-04-26T04:18:51.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.638+0000 7f47d6ed7640 5 monclient: authenticate success, global_id 4305 2024-04-26T04:18:51.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.638+0000 7f47d6ed7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T04:18:51.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.638+0000 7f47cf7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:51.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.638+0000 7f47d6ed7640 10 monclient: get_monmap_and_config success 2024-04-26T04:18:51.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.638+0000 7f47cf7fe640 1 -- 172.21.15.63:0/2431193699 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47c00ab9c0 con 0x7f47d013c530 2024-04-26T04:18:51.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.638+0000 7f47d6ed7640 4 set_mon_vals no callback set 2024-04-26T04:18:51.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.638+0000 7f47cf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:51.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.638+0000 7f47cf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:18:51.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.638+0000 7f47cf7fe640 10 monclient: dump: 2024-04-26T04:18:51.654 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:51.654 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:51.654 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:51.654 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:51.654 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:51.654 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:51.654 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:51.654 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:51.654 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:51.654 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:51.654 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.638+0000 7f47d6ed7640 10 monclient: shutdown 2024-04-26T04:18:51.654 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.638+0000 7f47d6ed7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:51.654 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.638+0000 7f47d6ed7640 1 -- 172.21.15.63:0/2431193699 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f47d013c530 msgr2=0x7f47d013e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:51.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.638+0000 7f47d6ed7640 1 --2- 172.21.15.63:0/2431193699 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f47d013c530 0x7f47d013e920 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f47d006e2f0 tx=0x7f47c001c400 comp rx=0 tx=0).stop 2024-04-26T04:18:51.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.638+0000 7f47d6ed7640 1 -- 172.21.15.63:0/2431193699 shutdown_connections 2024-04-26T04:18:51.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.638+0000 7f47d6ed7640 1 --2- 172.21.15.63:0/2431193699 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f47d013c530 0x7f47d013e920 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:51.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.638+0000 7f47d6ed7640 1 --2- 172.21.15.63:0/2431193699 >> v2:172.21.15.186:3300/0 conn(0x7f47d0133c70 0x7f47d013bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:51.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.638+0000 7f47d6ed7640 1 -- 172.21.15.63:0/2431193699 >> 172.21.15.63:0/2431193699 conn(0x7f47d00a6290 msgr2=0x7f47d00a2520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:51.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.638+0000 7f47d6ed7640 1 -- 172.21.15.63:0/2431193699 shutdown_connections 2024-04-26T04:18:51.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.638+0000 7f47d6ed7640 1 -- 172.21.15.63:0/2431193699 wait complete. 2024-04-26T04:18:51.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.638+0000 7f47d6ed7640 10 monclient: build_initial_monmap 2024-04-26T04:18:51.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.638+0000 7f47d6ed7640 10 monclient: monmap: 2024-04-26T04:18:51.655 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:51.655 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:18:51.655 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:18:51.655 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:18:51.655 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:51.656 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:51.656 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:18:51.656 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:18:51.656 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:18:51.656 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:51.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.642+0000 7f47d6ed7640 1 Processor -- start 2024-04-26T04:18:51.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.642+0000 7f47d6ed7640 1 -- start start 2024-04-26T04:18:51.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.642+0000 7f47d6ed7640 10 monclient: init 2024-04-26T04:18:51.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.642+0000 7f47d6ed7640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:51.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.642+0000 7f47d6ed7640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T04:18:51.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.642+0000 7f47d6ed7640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f47d0072450 0x7f47d013ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:51.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.642+0000 7f47d6ed7640 10 monclient(hunting): picked mon.noname-c con 0x7f47d0072450 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:51.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.642+0000 7f47d6ed7640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f47d0133c70 0x7f47d0130500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:51.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.642+0000 7f47d6ed7640 10 monclient(hunting): picked mon.noname-b con 0x7f47d0133c70 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:18:51.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.642+0000 7f47d6ed7640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f47d013c530 0x7f47d0130a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:51.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.642+0000 7f47d6ed7640 10 monclient(hunting): picked mon.noname-a con 0x7f47d013c530 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:18:51.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.642+0000 7f47d6ed7640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:51.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.642+0000 7f47d6ed7640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f47d006c290 con 0x7f47d013c530 2024-04-26T04:18:51.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.642+0000 7f47d6ed7640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:51.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.642+0000 7f47d6ed7640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f47d006c110 con 0x7f47d0133c70 2024-04-26T04:18:51.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.642+0000 7f47d6ed7640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:51.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.642+0000 7f47d6ed7640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f47d0141680 con 0x7f47d0072450 2024-04-26T04:18:51.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.642+0000 7f47d6ed7640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:51.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.642+0000 7f47d6ed7640 10 monclient(hunting): authenticate will time out at 963.352559s 2024-04-26T04:18:51.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.642+0000 7f47d5474640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f47d013c530 0x7f47d0130a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:51.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.642+0000 7f47d5474640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f47d013c530 0x7f47d0130a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:39292/0 (socket says 172.21.15.63:39292) 2024-04-26T04:18:51.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.642+0000 7f47d5474640 1 -- 172.21.15.63:0/1589608174 learned_addr learned my addr 172.21.15.63:0/1589608174 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:18:51.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.642+0000 7f47d5474640 10 monclient(hunting): get_auth_request con 0x7f47d013c530 auth_method 0 2024-04-26T04:18:51.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.642+0000 7f47d5474640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:51.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.642+0000 7f47d5474640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:51.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.642+0000 7f47d5474640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:51.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.642+0000 7f47d4c73640 1 --2- 172.21.15.63:0/1589608174 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f47d0072450 0x7f47d013ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:51.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.642+0000 7f47d4c73640 10 monclient(hunting): get_auth_request con 0x7f47d0072450 auth_method 0 2024-04-26T04:18:51.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.642+0000 7f47d4c73640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:51.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.642+0000 7f47d4c73640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:51.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.642+0000 7f47d4c73640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:51.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.642+0000 7f47d5474640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:51.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.642+0000 7f47d5474640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:51.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.642+0000 7f47d5474640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:51.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.642+0000 7f47d4c73640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:51.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.642+0000 7f47d4c73640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:51.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.642+0000 7f47d4c73640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:51.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.642+0000 7f47cffff640 1 --2- 172.21.15.63:0/1589608174 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f47d0133c70 0x7f47d0130500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:51.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.642+0000 7f47cffff640 10 monclient(hunting): get_auth_request con 0x7f47d0133c70 auth_method 0 2024-04-26T04:18:51.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.642+0000 7f47cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:51.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.642+0000 7f47cffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:51.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.642+0000 7f47cffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:51.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.642+0000 7f47d5474640 10 monclient(hunting): handle_auth_done global_id 4308 payload 824 2024-04-26T04:18:51.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.642+0000 7f47d5474640 1 -- 172.21.15.63:0/1589608174 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f47d0072450 msgr2=0x7f47d013ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:51.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.642+0000 7f47d5474640 1 --2- 172.21.15.63:0/1589608174 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f47d0072450 0x7f47d013ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:51.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.642+0000 7f47d5474640 1 -- 172.21.15.63:0/1589608174 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f47d0133c70 msgr2=0x7f47d0130500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:51.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.642+0000 7f47d5474640 1 --2- 172.21.15.63:0/1589608174 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f47d0133c70 0x7f47d0130500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:51.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.642+0000 7f47d5474640 10 monclient: _finish_hunting 0 2024-04-26T04:18:51.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.642+0000 7f47d5474640 1 monclient: found mon.noname-a 2024-04-26T04:18:51.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.642+0000 7f47d5474640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:51.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.642+0000 7f47d5474640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:18:51.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.642+0000 7f47d5474640 1 -- 172.21.15.63:0/1589608174 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47d0133770 con 0x7f47d013c530 2024-04-26T04:18:51.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.642+0000 7f47d5474640 1 -- 172.21.15.63:0/1589608174 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f47d013c530 msgr2=0x7f47d0130a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:51.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.642+0000 7f47d5474640 1 --2- 172.21.15.63:0/1589608174 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f47d013c530 0x7f47d0130a40 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f47c00ad040 tx=0x7f47c001ac40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:51.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.642+0000 7f47cd7fa640 1 -- 172.21.15.63:0/1589608174 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47c00b4800 con 0x7f47d013c530 2024-04-26T04:18:51.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.642+0000 7f47cd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:51.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.642+0000 7f47cd7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:18:51.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.642+0000 7f47cd7fa640 10 monclient: dump: 2024-04-26T04:18:51.660 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:51.660 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:51.660 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:51.660 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:51.660 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:51.660 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:51.660 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:51.660 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:51.660 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:51.660 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:51.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.642+0000 7f47cd7fa640 10 monclient: _finish_auth 0 2024-04-26T04:18:51.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.642+0000 7f47cd7fa640 10 monclient: _check_auth_tickets 2024-04-26T04:18:51.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.642+0000 7f47cd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:51.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.642+0000 7f47cd7fa640 1 -- 172.21.15.63:0/1589608174 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47c00b49c0 con 0x7f47d013c530 2024-04-26T04:18:51.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.642+0000 7f47cd7fa640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:51.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.642+0000 7f47cd7fa640 1 -- 172.21.15.63:0/1589608174 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47c00b4db0 con 0x7f47d013c530 2024-04-26T04:18:51.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.642+0000 7f47cd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:51.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.642+0000 7f47cd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:18:51.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.642+0000 7f47cd7fa640 10 monclient: dump: 2024-04-26T04:18:51.661 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:51.661 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:51.661 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:51.661 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:51.661 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:51.661 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:51.661 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:51.661 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:51.662 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:51.662 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:51.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.642+0000 7f47ceffd640 4 set_mon_vals no callback set 2024-04-26T04:18:51.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.646+0000 7f47d6ed7640 5 monclient: authenticate success, global_id 4308 2024-04-26T04:18:51.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.646+0000 7f47d6ed7640 10 monclient: _renew_subs 2024-04-26T04:18:51.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.646+0000 7f47d6ed7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:51.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.646+0000 7f47d6ed7640 1 -- 172.21.15.63:0/1589608174 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47d0138170 con 0x7f47d013c530 2024-04-26T04:18:51.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.646+0000 7f47d6ed7640 10 monclient: _renew_subs 2024-04-26T04:18:51.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.646+0000 7f47d6ed7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:51.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.646+0000 7f47d6ed7640 1 -- 172.21.15.63:0/1589608174 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47d013a1f0 con 0x7f47d013c530 2024-04-26T04:18:51.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.646+0000 7f47d6ed7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:51.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.646+0000 7f47cd7fa640 1 -- 172.21.15.63:0/1589608174 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f47c00aa070 con 0x7f47d013c530 2024-04-26T04:18:51.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.646+0000 7f47d6ed7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:51.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.646+0000 7f47d6ed7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:51.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.646+0000 7f47d6ed7640 1 -- 172.21.15.63:0/1589608174 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47d0062e10 con 0x7f47d013c530 2024-04-26T04:18:51.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.646+0000 7f47cd7fa640 1 --2- 172.21.15.63:0/1589608174 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f47bc035f40 0x7f47bc038400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:51.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.646+0000 7f47cd7fa640 1 -- 172.21.15.63:0/1589608174 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7f47c00e3150 con 0x7f47d013c530 2024-04-26T04:18:51.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.646+0000 7f47d4c73640 1 --2- 172.21.15.63:0/1589608174 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f47bc035f40 0x7f47bc038400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:51.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.646+0000 7f47d4c73640 10 monclient: get_auth_request con 0x7f47bc035f40 auth_method 0 2024-04-26T04:18:51.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.650+0000 7f47d4c73640 1 --2- 172.21.15.63:0/1589608174 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f47bc035f40 0x7f47bc038400 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f47c8005680 tx=0x7f47c8003dd0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:51.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.650+0000 7f47cd7fa640 1 -- 172.21.15.63:0/1589608174 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f47c00160b0 con 0x7f47d013c530 2024-04-26T04:18:51.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.650+0000 7f47cd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:51.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.650+0000 7f47cd7fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:18:51.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.810+0000 7f47d6ed7640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-26T04:18:51.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.810+0000 7f47d6ed7640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-26T04:18:51.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.810+0000 7f47d6ed7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:51.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.810+0000 7f47d6ed7640 1 -- 172.21.15.63:0/1589608174 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f47d0062e10 con 0x7f47d013c530 2024-04-26T04:18:51.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.810+0000 7f47cd7fa640 1 -- 172.21.15.63:0/1589608174 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) ==== 74+0+12245 (secure 0 0 0) 0x7f47c00abe20 con 0x7f47d013c530 2024-04-26T04:18:51.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.810+0000 7f47cd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-26T04:18:51.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.810+0000 7f47cd7fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:18:51.817 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-26T04:18:51.817 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":15,"fsid":"cd1c6a0c-4e4b-4014-89c0-2e535479ef39","created":"2024-04-26T04:18:34.126231+0000","modified":"2024-04-26T04:18:47.487705+0000","last_up_change":"2024-04-26T04:18:44.470061+0000","last_in_change":"2024-04-26T04:18:36.862750+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T04:18:43.521296+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-26T04:18:45.211226+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d37141b-8282-4c62-864a-84a75ebf8f6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":2758152700},{"type":"v1","addr":"172.21.15.63:6801","nonce":2758152700}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":2758152700},{"type":"v1","addr":"172.21.15.63:6803","nonce":2758152700}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":2758152700},{"type":"v1","addr":"172.21.15.63:6807","nonce":2758152700}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":2758152700},{"type":"v1","addr":"172.21.15.63:6805","nonce":2758152700}]},"public_addr":"172.21.15.63:6801/2758152700","cluster_addr":"172.21.15.63:6803/2758152700","heartbeat_back_addr":"172.21.15.63:6807/2758152700","heartbeat_front_addr":"172.21.15.63:6805/2758152700","state":["exists","up"]},{"osd":1,"uuid":"7220e45a-72bb-4d8e-93c2-32005fa2c85e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":3596874741},{"type":"v1","addr":"172.21.15.63:6817","nonce":3596874741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":3596874741},{"type":"v1","addr":"172.21.15.63:6819","nonce":3596874741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":3596874741},{"type":"v1","addr":"172.21.15.63:6823","nonce":3596874741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":3596874741},{"type":"v1","addr":"172.21.15.63:6821","nonce":3596874741}]},"public_addr":"172.21.15.63:6817/3596874741","cluster_addr":"172.21.15.63:6819/3596874741","heartbeat_back_addr":"172.21.15.63:6823/3596874741","heartbeat_front_addr":"172.21.15.63:6821/3596874741","state":["exists","up"]},{"osd":2,"uuid":"913de4d8-31af-4394-a0ee-194cc31f822f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":413829082},{"type":"v1","addr":"172.21.15.63:6809","nonce":413829082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":413829082},{"type":"v1","addr":"172.21.15.63:6811","nonce":413829082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":413829082},{"type":"v1","addr":"172.21.15.63:6815","nonce":413829082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":413829082},{"type":"v1","addr":"172.21.15.63:6813","nonce":413829082}]},"public_addr":"172.21.15.63:6809/413829082","cluster_addr":"172.21.15.63:6811/413829082","heartbeat_back_addr":"172.21.15.63:6815/413829082","heartbeat_front_addr":"172.21.15.63:6813/413829082","state":["exists","up"]},{"osd":3,"uuid":"c492772c-ccf5-4b85-9056-6a5ee9686c0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6807","nonce":4227496981},{"type":"v1","addr":"172.21.15.186:6808","nonce":4227496981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4227496981},{"type":"v1","addr":"172.21.15.186:6811","nonce":4227496981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4227496981},{"type":"v1","addr":"172.21.15.186:6815","nonce":4227496981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4227496981},{"type":"v1","addr":"172.21.15.186:6813","nonce":4227496981}]},"public_addr":"172.21.15.186:6808/4227496981","cluster_addr":"172.21.15.186:6811/4227496981","heartbeat_back_addr":"172.21.15.186:6815/4227496981","heartbeat_front_addr":"172.21.15.186:6813/4227496981","state":["exists","up"]},{"osd":4,"uuid":"24f067f0-554a-4dd2-a26c-7bda3d0978f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":3746743567},{"type":"v1","addr":"172.21.15.186:6801","nonce":3746743567}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":3746743567},{"type":"v1","addr":"172.21.15.186:6803","nonce":3746743567}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":3746743567},{"type":"v1","addr":"172.21.15.186:6809","nonce":3746743567}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":3746743567},{"type":"v1","addr":"172.21.15.186:6805","nonce":3746743567}]},"public_addr":"172.21.15.186:6801/3746743567","cluster_addr":"172.21.15.186:6803/3746743567","heartbeat_back_addr":"172.21.15.186:6809/3746743567","heartbeat_front_addr":"172.21.15.186:6805/3746743567","state":["exists","up"]},{"osd":5,"uuid":"70454d09-e50c-4f65-83c5-0f9bb5c0f8f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":3109931567},{"type":"v1","addr":"172.21.15.186:6817","nonce":3109931567}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":3109931567},{"type":"v1","addr":"172.21.15.186:6819","nonce":3109931567}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":3109931567},{"type":"v1","addr":"172.21.15.186:6823","nonce":3109931567}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":3109931567},{"type":"v1","addr":"172.21.15.186:6821","nonce":3109931567}]},"public_addr":"172.21.15.186:6817/3109931567","cluster_addr":"172.21.15.186:6819/3109931567","heartbeat_back_addr":"172.21.15.186:6823/3109931567","heartbeat_front_addr":"172.21.15.186:6821/3109931567","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T04:18:42.278922+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T04:18:42.387279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T04:18:42.394303+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T04:18:42.367922+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T04:18:42.357914+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T04:18:42.415215+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T04:18:51.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.814+0000 7f47d6ed7640 1 -- 172.21.15.63:0/1589608174 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f47bc035f40 msgr2=0x7f47bc038400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:51.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.814+0000 7f47d6ed7640 1 --2- 172.21.15.63:0/1589608174 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f47bc035f40 0x7f47bc038400 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f47c8005680 tx=0x7f47c8003dd0 comp rx=0 tx=0).stop 2024-04-26T04:18:51.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.814+0000 7f47d6ed7640 10 monclient: shutdown 2024-04-26T04:18:51.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.814+0000 7f47d6ed7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:51.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.814+0000 7f47d6ed7640 1 -- 172.21.15.63:0/1589608174 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f47d013c530 msgr2=0x7f47d0130a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:51.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.814+0000 7f47d6ed7640 1 --2- 172.21.15.63:0/1589608174 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f47d013c530 0x7f47d0130a40 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f47c00ad040 tx=0x7f47c001ac40 comp rx=0 tx=0).stop 2024-04-26T04:18:51.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.814+0000 7f47d6ed7640 1 -- 172.21.15.63:0/1589608174 shutdown_connections 2024-04-26T04:18:51.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.814+0000 7f47d6ed7640 1 --2- 172.21.15.63:0/1589608174 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f47bc035f40 0x7f47bc038400 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:51.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.814+0000 7f47d6ed7640 1 --2- 172.21.15.63:0/1589608174 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f47d013c530 0x7f47d0130a40 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:51.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.814+0000 7f47d6ed7640 1 --2- 172.21.15.63:0/1589608174 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f47d0133c70 0x7f47d0130500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:51.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.814+0000 7f47d6ed7640 1 --2- 172.21.15.63:0/1589608174 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f47d0072450 0x7f47d013ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:51.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.814+0000 7f47d6ed7640 1 -- 172.21.15.63:0/1589608174 >> 172.21.15.63:0/1589608174 conn(0x7f47d00a6290 msgr2=0x7f47d0131690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:51.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.814+0000 7f47d6ed7640 1 -- 172.21.15.63:0/1589608174 shutdown_connections 2024-04-26T04:18:51.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.814+0000 7f47d6ed7640 1 -- 172.21.15.63:0/1589608174 wait complete. 2024-04-26T04:18:51.837 INFO:tasks.mon_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-26T04:18:43.521296+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.059999942779541, 'score_stable': 6.059999942779541, 'optimal_score': 0.33000001311302185, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-04-26T04:18:45.211226+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-26T04:18:51.837 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-26T04:18:51.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.950+0000 7f881668b640 10 monclient: get_monmap_and_config 2024-04-26T04:18:51.958 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.950+0000 7f881668b640 10 monclient: build_initial_monmap 2024-04-26T04:18:51.958 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.950+0000 7f881668b640 10 monclient: monmap: 2024-04-26T04:18:51.958 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:51.958 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:51.958 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:51.957724+0000 2024-04-26T04:18:51.958 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:51.957724+0000 2024-04-26T04:18:51.958 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:51.958 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:51.958 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:18:51.958 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:18:51.958 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:18:51.958 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:18:51.959 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:18:51.959 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:51.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.954+0000 7f881668b640 1 Processor -- start 2024-04-26T04:18:51.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.954+0000 7f881668b640 1 -- start start 2024-04-26T04:18:51.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.954+0000 7f881668b640 10 monclient: init 2024-04-26T04:18:51.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.954+0000 7f881668b640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:51.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.954+0000 7f881668b640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-04-26T04:18:51.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.954+0000 7f881668b640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f88100a2a90 0x7f88100a2e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:51.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.954+0000 7f881668b640 10 monclient(hunting): picked mon.noname-a con 0x7f88100a2a90 addr v2:172.21.15.63:3300/0 2024-04-26T04:18:51.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.954+0000 7f881668b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f88100a1ec0 addr v1:172.21.15.186:6789/0 2024-04-26T04:18:51.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.954+0000 7f881668b640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f88100a22d0 0x7f88100a45c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:51.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.954+0000 7f881668b640 10 monclient(hunting): picked mon.noname-b con 0x7f88100a22d0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:51.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.954+0000 7f881668b640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88100a0480 con 0x7f88100a1ec0 2024-04-26T04:18:51.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.954+0000 7f881668b640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:51.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.954+0000 7f881668b640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7f88100a0f40 con 0x7f88100a2a90 2024-04-26T04:18:51.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.954+0000 7f881668b640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:51.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.954+0000 7f881668b640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f88100a0c60 con 0x7f88100a22d0 2024-04-26T04:18:51.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.954+0000 7f881668b640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:51.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.954+0000 7f881668b640 10 monclient(hunting): authenticate will time out at 963.665384s 2024-04-26T04:18:51.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.954+0000 7f880ffff640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f88100a2a90 0x7f88100a2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:51.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.954+0000 7f8814c28640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f88100a22d0 0x7f88100a45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:51.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.954+0000 7f880ffff640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f88100a2a90 0x7f88100a2e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:39294/0 (socket says 172.21.15.63:39294) 2024-04-26T04:18:51.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.954+0000 7f880ffff640 1 -- 172.21.15.63:0/1629900197 learned_addr learned my addr 172.21.15.63:0/1629900197 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:18:51.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.954+0000 7f880ffff640 10 monclient(hunting): get_auth_request con 0x7f88100a2a90 auth_method 0 2024-04-26T04:18:51.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.954+0000 7f880ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:51.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.954+0000 7f880ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:51.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.954+0000 7f880ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:51.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.954+0000 7f8814c28640 10 monclient(hunting): get_auth_request con 0x7f88100a22d0 auth_method 0 2024-04-26T04:18:51.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.954+0000 7f8814c28640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:51.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.954+0000 7f8814c28640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:51.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.954+0000 7f8814c28640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:51.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.954+0000 7f880ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:51.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.954+0000 7f880ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:51.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.954+0000 7f880ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:51.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.954+0000 7f8814c28640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:51.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.954+0000 7f8814c28640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:51.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.954+0000 7f8814c28640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:51.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.954+0000 7f880ffff640 10 monclient(hunting): handle_auth_done global_id 4311 payload 274 2024-04-26T04:18:51.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.954+0000 7f880ffff640 1 -- 172.21.15.63:0/1629900197 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f88100a22d0 msgr2=0x7f88100a45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:51.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.954+0000 7f880ffff640 1 --2- 172.21.15.63:0/1629900197 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f88100a22d0 0x7f88100a45c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:51.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.954+0000 7f880ffff640 1 -- 172.21.15.63:0/1629900197 >> v1:172.21.15.186:6789/0 conn(0x7f88100a1ec0 legacy=0x7f88100a3eb0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-26T04:18:51.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.954+0000 7f880ffff640 10 monclient: _finish_hunting 0 2024-04-26T04:18:51.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.954+0000 7f880ffff640 1 monclient: found mon.noname-a 2024-04-26T04:18:51.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.954+0000 7f880ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:51.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.954+0000 7f880ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:18:51.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.954+0000 7f880ffff640 1 -- 172.21.15.63:0/1629900197 --> v2:172.21.15.63:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88100a1220 con 0x7f88100a2a90 2024-04-26T04:18:51.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.954+0000 7f8814c28640 1 --2- 172.21.15.63:0/1629900197 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f88100a22d0 0x7f88100a45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T04:18:51.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.958+0000 7f880ffff640 1 -- 172.21.15.63:0/1629900197 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f88100a2a90 msgr2=0x7f88100a2e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:51.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.958+0000 7f880ffff640 1 --2- 172.21.15.63:0/1629900197 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f88100a2a90 0x7f88100a2e70 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f88000098a0 tx=0x7f880001c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a3d8fe0d66c7914 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:51.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.958+0000 7f880f7fe640 1 -- 172.21.15.63:0/1629900197 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88000b1070 con 0x7f88100a2a90 2024-04-26T04:18:51.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.958+0000 7f880f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:51.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.958+0000 7f880f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:18:51.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.958+0000 7f880f7fe640 10 monclient: dump: 2024-04-26T04:18:51.972 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:51.972 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:51.972 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:51.972 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:51.972 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:51.972 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:51.973 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:51.973 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:51.973 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:51.973 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:51.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.958+0000 7f880f7fe640 10 monclient: _finish_auth 0 2024-04-26T04:18:51.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.958+0000 7f880f7fe640 10 monclient: _check_auth_tickets 2024-04-26T04:18:51.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.958+0000 7f880f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:51.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.958+0000 7f881668b640 5 monclient: authenticate success, global_id 4311 2024-04-26T04:18:51.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.958+0000 7f881668b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T04:18:51.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.958+0000 7f880f7fe640 1 -- 172.21.15.63:0/1629900197 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88000ac600 con 0x7f88100a2a90 2024-04-26T04:18:51.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.958+0000 7f880f7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:51.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.958+0000 7f881668b640 10 monclient: get_monmap_and_config success 2024-04-26T04:18:51.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.958+0000 7f881668b640 4 set_mon_vals no callback set 2024-04-26T04:18:51.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.958+0000 7f880f7fe640 1 -- 172.21.15.63:0/1629900197 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88000ac9d0 con 0x7f88100a2a90 2024-04-26T04:18:51.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.958+0000 7f880f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:51.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.958+0000 7f880f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:18:51.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.958+0000 7f880f7fe640 10 monclient: dump: 2024-04-26T04:18:51.974 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:51.974 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:51.974 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:51.974 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:51.974 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:51.974 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:51.975 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:51.975 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:51.975 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:51.975 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:51.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.958+0000 7f881668b640 10 monclient: shutdown 2024-04-26T04:18:51.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.958+0000 7f881668b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:51.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.958+0000 7f881668b640 1 -- 172.21.15.63:0/1629900197 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f88100a2a90 msgr2=0x7f88100a2e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:51.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.958+0000 7f881668b640 1 --2- 172.21.15.63:0/1629900197 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f88100a2a90 0x7f88100a2e70 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f88000098a0 tx=0x7f880001c3c0 comp rx=0 tx=0).stop 2024-04-26T04:18:51.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.958+0000 7f881668b640 1 -- 172.21.15.63:0/1629900197 shutdown_connections 2024-04-26T04:18:51.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.958+0000 7f881668b640 1 --2- 172.21.15.63:0/1629900197 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f88100a22d0 0x7f88100a45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:51.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.958+0000 7f881668b640 1 --2- 172.21.15.63:0/1629900197 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f88100a2a90 0x7f88100a2e70 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:51.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.958+0000 7f881668b640 1 -- 172.21.15.63:0/1629900197 >> 172.21.15.63:0/1629900197 conn(0x7f88100a5fb0 msgr2=0x7f88100a63c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:51.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.958+0000 7f881668b640 1 -- 172.21.15.63:0/1629900197 shutdown_connections 2024-04-26T04:18:51.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.958+0000 7f881668b640 1 -- 172.21.15.63:0/1629900197 wait complete. 2024-04-26T04:18:51.976 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.958+0000 7f881668b640 10 monclient: build_initial_monmap 2024-04-26T04:18:51.976 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.958+0000 7f881668b640 10 monclient: monmap: 2024-04-26T04:18:51.976 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:51.976 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:18:51.976 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:18:51.976 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:18:51.976 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:51.976 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:51.976 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:18:51.976 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:18:51.977 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:18:51.977 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:51.977 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.958+0000 7f881668b640 1 Processor -- start 2024-04-26T04:18:51.977 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f881668b640 1 -- start start 2024-04-26T04:18:51.977 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f881668b640 10 monclient: init 2024-04-26T04:18:51.977 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f881668b640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:51.977 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f881668b640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T04:18:51.977 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f881668b640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f88100a1ec0 0x7f88101cecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:51.977 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f881668b640 10 monclient(hunting): picked mon.noname-c con 0x7f88100a1ec0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:51.978 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f881668b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f88100a22d0 0x7f881013de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:51.978 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f881668b640 10 monclient(hunting): picked mon.noname-b con 0x7f88100a22d0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:18:51.978 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f881668b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f88100a2a90 0x7f88101248f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:51.978 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f881668b640 10 monclient(hunting): picked mon.noname-a con 0x7f88100a2a90 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:18:51.978 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f881668b640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:51.978 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f881668b640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f8810141870 con 0x7f88100a2a90 2024-04-26T04:18:51.978 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f881668b640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:51.978 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f881668b640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f88101416f0 con 0x7f88100a22d0 2024-04-26T04:18:51.978 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f881668b640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:51.978 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f881668b640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f88101419f0 con 0x7f88100a1ec0 2024-04-26T04:18:51.978 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f881668b640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:51.978 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f881668b640 10 monclient(hunting): authenticate will time out at 963.672631s 2024-04-26T04:18:51.978 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f880ffff640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f88100a1ec0 0x7f88101cecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:51.978 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f880ffff640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f88100a1ec0 0x7f88101cecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:36946/0 (socket says 172.21.15.63:36946) 2024-04-26T04:18:51.979 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f880ffff640 1 -- 172.21.15.63:0/2503650903 learned_addr learned my addr 172.21.15.63:0/2503650903 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:18:51.979 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f880ffff640 10 monclient(hunting): get_auth_request con 0x7f88100a1ec0 auth_method 0 2024-04-26T04:18:51.979 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f880ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:51.979 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f8814c28640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f88100a2a90 0x7f88101248f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:51.979 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f880ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:51.979 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f880ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:51.979 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f8814c28640 10 monclient(hunting): get_auth_request con 0x7f88100a2a90 auth_method 0 2024-04-26T04:18:51.979 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f8814c28640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:51.979 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f8814c28640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:51.979 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f8814c28640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:51.979 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f880ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:51.979 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f880ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:51.979 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f880ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:51.979 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f8814c28640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:51.980 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f8814c28640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:51.980 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f8814c28640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:51.980 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f8807fff640 1 --2- 172.21.15.63:0/2503650903 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f88100a22d0 0x7f881013de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:51.980 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f880ffff640 10 monclient(hunting): handle_auth_done global_id 4295 payload 824 2024-04-26T04:18:51.980 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f880ffff640 1 -- 172.21.15.63:0/2503650903 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f88100a22d0 msgr2=0x7f881013de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:51.980 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f880ffff640 1 --2- 172.21.15.63:0/2503650903 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f88100a22d0 0x7f881013de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:51.980 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f880ffff640 1 -- 172.21.15.63:0/2503650903 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f88100a2a90 msgr2=0x7f88101248f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:51.980 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f880ffff640 1 --2- 172.21.15.63:0/2503650903 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f88100a2a90 0x7f88101248f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:51.980 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f880ffff640 10 monclient: _finish_hunting 0 2024-04-26T04:18:51.980 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f880ffff640 1 monclient: found mon.noname-c 2024-04-26T04:18:51.980 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f880ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:51.980 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f880ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.63:3301/0 2024-04-26T04:18:51.980 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f880ffff640 1 -- 172.21.15.63:0/2503650903 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8810126540 con 0x7f88100a1ec0 2024-04-26T04:18:51.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f8807fff640 10 monclient: get_auth_request con 0x7f88100a22d0 auth_method 0 2024-04-26T04:18:51.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f8807fff640 1 --2- 172.21.15.63:0/2503650903 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f88100a22d0 0x7f881013de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T04:18:51.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f880ffff640 1 -- 172.21.15.63:0/2503650903 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f88100a1ec0 msgr2=0x7f88101cecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:51.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f880ffff640 1 --2- 172.21.15.63:0/2503650903 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f88100a1ec0 0x7f88101cecc0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f880001cc60 tx=0x7f880001ac00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:51.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f880d7fa640 1 -- 172.21.15.63:0/2503650903 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88000bb020 con 0x7f88100a1ec0 2024-04-26T04:18:51.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f880d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:51.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f880d7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:18:51.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f880d7fa640 10 monclient: dump: 2024-04-26T04:18:51.981 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:51.981 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:51.981 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:51.981 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:51.981 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:51.981 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:51.981 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:51.982 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:51.982 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:51.982 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:51.982 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f880d7fa640 10 monclient: _finish_auth 0 2024-04-26T04:18:51.982 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f880d7fa640 10 monclient: _check_auth_tickets 2024-04-26T04:18:51.982 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f880d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:51.982 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f880d7fa640 1 -- 172.21.15.63:0/2503650903 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88000b7630 con 0x7f88100a1ec0 2024-04-26T04:18:51.982 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f880d7fa640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:51.982 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f880d7fa640 1 -- 172.21.15.63:0/2503650903 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88000b1040 con 0x7f88100a1ec0 2024-04-26T04:18:51.982 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f881668b640 5 monclient: authenticate success, global_id 4295 2024-04-26T04:18:51.982 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f881668b640 10 monclient: _renew_subs 2024-04-26T04:18:51.982 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f881668b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:51.982 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f881668b640 1 -- 172.21.15.63:0/2503650903 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88100a3720 con 0x7f88100a1ec0 2024-04-26T04:18:51.982 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f880d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:51.983 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f880effd640 4 set_mon_vals no callback set 2024-04-26T04:18:51.983 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f880d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:18:51.983 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f880d7fa640 10 monclient: dump: 2024-04-26T04:18:51.983 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:51.983 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:51.983 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:51.983 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:51.983 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:51.983 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:51.983 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:51.983 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:51.983 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:51.983 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:51.983 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f881668b640 10 monclient: _renew_subs 2024-04-26T04:18:51.983 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f881668b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:51.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.962+0000 7f881668b640 1 -- 172.21.15.63:0/2503650903 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f88101cfb40 con 0x7f88100a1ec0 2024-04-26T04:18:51.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.966+0000 7f881668b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:51.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.966+0000 7f881668b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:51.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.966+0000 7f881668b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:51.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.966+0000 7f881668b640 1 -- 172.21.15.63:0/2503650903 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8810062e10 con 0x7f88100a1ec0 2024-04-26T04:18:51.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.966+0000 7f880d7fa640 1 -- 172.21.15.63:0/2503650903 <== mon.2 v2:172.21.15.63:3301/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f88000be020 con 0x7f88100a1ec0 2024-04-26T04:18:51.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.966+0000 7f880d7fa640 1 --2- 172.21.15.63:0/2503650903 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f87f0035ec0 0x7f87f0038380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:51.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.966+0000 7f880d7fa640 1 -- 172.21.15.63:0/2503650903 <== mon.2 v2:172.21.15.63:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7f88000e2e40 con 0x7f88100a1ec0 2024-04-26T04:18:51.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.966+0000 7f8807fff640 1 --2- 172.21.15.63:0/2503650903 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f87f0035ec0 0x7f87f0038380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:51.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.966+0000 7f8807fff640 10 monclient: get_auth_request con 0x7f87f0035ec0 auth_method 0 2024-04-26T04:18:51.985 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.970+0000 7f8807fff640 1 --2- 172.21.15.63:0/2503650903 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f87f0035ec0 0x7f87f0038380 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8810126140 tx=0x7f87f8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:51.985 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.970+0000 7f880d7fa640 1 -- 172.21.15.63:0/2503650903 <== mon.2 v2:172.21.15.63:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f88000c5030 con 0x7f88100a1ec0 2024-04-26T04:18:51.985 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.970+0000 7f880d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:51.985 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:51.970+0000 7f880d7fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:18:52.139 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.134+0000 7f881668b640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-04-26T04:18:52.139 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.134+0000 7f881668b640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-04-26T04:18:52.139 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.134+0000 7f881668b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:52.140 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.134+0000 7f881668b640 1 -- 172.21.15.63:0/2503650903 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f88100683c0 con 0x7f88100a1ec0 2024-04-26T04:18:52.140 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.134+0000 7f880d7fa640 1 -- 172.21.15.63:0/2503650903 <== mon.2 v2:172.21.15.63:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v15) ==== 93+0+10 (secure 0 0 0) 0x7f88000160d0 con 0x7f88100a1ec0 2024-04-26T04:18:52.140 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.134+0000 7f880d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-04-26T04:18:52.140 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.134+0000 7f880d7fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:18:52.140 INFO:teuthology.orchestra.run.smithi063.stdout:pg_num: 1 2024-04-26T04:18:52.142 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.138+0000 7f881668b640 1 -- 172.21.15.63:0/2503650903 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f87f0035ec0 msgr2=0x7f87f0038380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:52.142 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.138+0000 7f881668b640 1 --2- 172.21.15.63:0/2503650903 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f87f0035ec0 0x7f87f0038380 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8810126140 tx=0x7f87f8008040 comp rx=0 tx=0).stop 2024-04-26T04:18:52.142 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.138+0000 7f881668b640 10 monclient: shutdown 2024-04-26T04:18:52.142 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.138+0000 7f881668b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:52.142 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.138+0000 7f881668b640 1 -- 172.21.15.63:0/2503650903 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f88100a1ec0 msgr2=0x7f88101cecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:52.142 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.138+0000 7f881668b640 1 --2- 172.21.15.63:0/2503650903 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f88100a1ec0 0x7f88101cecc0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f880001cc60 tx=0x7f880001ac00 comp rx=0 tx=0).stop 2024-04-26T04:18:52.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.138+0000 7f881668b640 1 -- 172.21.15.63:0/2503650903 shutdown_connections 2024-04-26T04:18:52.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.138+0000 7f881668b640 1 --2- 172.21.15.63:0/2503650903 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f87f0035ec0 0x7f87f0038380 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:52.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.138+0000 7f881668b640 1 --2- 172.21.15.63:0/2503650903 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f88100a2a90 0x7f88101248f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:52.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.138+0000 7f881668b640 1 --2- 172.21.15.63:0/2503650903 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f88100a22d0 0x7f881013de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:52.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.138+0000 7f881668b640 1 --2- 172.21.15.63:0/2503650903 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f88100a1ec0 0x7f88101cecc0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:52.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.138+0000 7f881668b640 1 -- 172.21.15.63:0/2503650903 >> 172.21.15.63:0/2503650903 conn(0x7f88100a5fb0 msgr2=0x7f8810070e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:52.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.138+0000 7f881668b640 1 -- 172.21.15.63:0/2503650903 shutdown_connections 2024-04-26T04:18:52.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.138+0000 7f881668b640 1 -- 172.21.15.63:0/2503650903 wait complete. 2024-04-26T04:18:52.159 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2024-04-26T04:18:52.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.278+0000 7f7c75275640 10 monclient: get_monmap_and_config 2024-04-26T04:18:52.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.278+0000 7f7c75275640 10 monclient: build_initial_monmap 2024-04-26T04:18:52.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.278+0000 7f7c75275640 10 monclient: monmap: 2024-04-26T04:18:52.286 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:52.286 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:52.286 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:52.285737+0000 2024-04-26T04:18:52.286 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:52.285737+0000 2024-04-26T04:18:52.286 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:52.286 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:52.286 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:18:52.286 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:18:52.287 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:18:52.287 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:18:52.287 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:18:52.287 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:52.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.282+0000 7f7c75275640 1 Processor -- start 2024-04-26T04:18:52.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.282+0000 7f7c75275640 1 -- start start 2024-04-26T04:18:52.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.282+0000 7f7c75275640 10 monclient: init 2024-04-26T04:18:52.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.282+0000 7f7c75275640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:52.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.282+0000 7f7c75275640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-04-26T04:18:52.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.282+0000 7f7c75275640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7c70133dc0 addr v1:172.21.15.186:6789/0 2024-04-26T04:18:52.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.282+0000 7f7c75275640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f7c70072610 0x7f7c70139d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:52.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.282+0000 7f7c75275640 10 monclient(hunting): picked mon.noname-a con 0x7f7c70072610 addr v2:172.21.15.63:3300/0 2024-04-26T04:18:52.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.282+0000 7f7c75275640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f7c700a1bc0 0x7f7c7013c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:52.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.282+0000 7f7c75275640 10 monclient(hunting): picked mon.noname-b con 0x7f7c700a1bc0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:52.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.282+0000 7f7c75275640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c70071ca0 con 0x7f7c70133dc0 2024-04-26T04:18:52.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.282+0000 7f7c75275640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:52.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.282+0000 7f7c75275640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7f7c700a31b0 con 0x7f7c70072610 2024-04-26T04:18:52.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.282+0000 7f7c75275640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:52.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.282+0000 7f7c75275640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f7c70071390 con 0x7f7c700a1bc0 2024-04-26T04:18:52.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.282+0000 7f7c75275640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:52.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.282+0000 7f7c75275640 10 monclient(hunting): authenticate will time out at 963.993751s 2024-04-26T04:18:52.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.282+0000 7f7c6e7fc640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f7c70072610 0x7f7c70139d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:52.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.282+0000 7f7c6f7fe640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f7c700a1bc0 0x7f7c7013c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:52.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.282+0000 7f7c6e7fc640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f7c70072610 0x7f7c70139d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:39312/0 (socket says 172.21.15.63:39312) 2024-04-26T04:18:52.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.282+0000 7f7c6e7fc640 1 -- 172.21.15.63:0/1180331335 learned_addr learned my addr 172.21.15.63:0/1180331335 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:18:52.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.282+0000 7f7c6e7fc640 10 monclient(hunting): get_auth_request con 0x7f7c70072610 auth_method 0 2024-04-26T04:18:52.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.282+0000 7f7c6e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:52.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.282+0000 7f7c6e7fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:52.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.282+0000 7f7c6e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:52.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.282+0000 7f7c6f7fe640 10 monclient(hunting): get_auth_request con 0x7f7c700a1bc0 auth_method 0 2024-04-26T04:18:52.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.282+0000 7f7c6f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:52.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.282+0000 7f7c6f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:52.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.282+0000 7f7c6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:52.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.282+0000 7f7c6e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:52.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.282+0000 7f7c6e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:52.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.282+0000 7f7c6e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:52.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.282+0000 7f7c6f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:52.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.282+0000 7f7c6f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:52.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.282+0000 7f7c6f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:52.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.282+0000 7f7c6e7fc640 10 monclient(hunting): handle_auth_done global_id 4317 payload 274 2024-04-26T04:18:52.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.282+0000 7f7c6e7fc640 1 -- 172.21.15.63:0/1180331335 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f7c700a1bc0 msgr2=0x7f7c7013c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:52.292 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.282+0000 7f7c6e7fc640 1 --2- 172.21.15.63:0/1180331335 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f7c700a1bc0 0x7f7c7013c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:52.292 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.282+0000 7f7c6e7fc640 1 -- 172.21.15.63:0/1180331335 >> v1:172.21.15.186:6789/0 conn(0x7f7c70133dc0 legacy=0x7f7c701341a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:52.292 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.282+0000 7f7c6e7fc640 10 monclient: _finish_hunting 0 2024-04-26T04:18:52.292 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.282+0000 7f7c6e7fc640 1 monclient: found mon.noname-a 2024-04-26T04:18:52.292 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.282+0000 7f7c6e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:52.292 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.282+0000 7f7c6e7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:18:52.292 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.282+0000 7f7c6e7fc640 1 -- 172.21.15.63:0/1180331335 --> v2:172.21.15.63:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c700a3600 con 0x7f7c70072610 2024-04-26T04:18:52.292 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.282+0000 7f7c6f7fe640 1 --2- 172.21.15.63:0/1180331335 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f7c700a1bc0 0x7f7c7013c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T04:18:52.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.286+0000 7f7c6e7fc640 1 -- 172.21.15.63:0/1180331335 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c70072610 msgr2=0x7f7c70139d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:52.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.286+0000 7f7c6e7fc640 1 --2- 172.21.15.63:0/1180331335 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c70072610 0x7f7c70139d10 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7c64009870 tx=0x7f7c6401c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0f3842a4309bd14 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:52.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.286+0000 7f7c6dffb640 1 -- 172.21.15.63:0/1180331335 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c640b0070 con 0x7f7c70072610 2024-04-26T04:18:52.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.286+0000 7f7c6dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:52.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.286+0000 7f7c6dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:18:52.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.286+0000 7f7c6dffb640 10 monclient: dump: 2024-04-26T04:18:52.294 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:52.294 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:52.294 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:52.294 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:52.294 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:52.294 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:52.294 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:52.294 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:52.295 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:52.295 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:52.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.286+0000 7f7c6dffb640 10 monclient: _finish_auth 0 2024-04-26T04:18:52.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.286+0000 7f7c6dffb640 10 monclient: _check_auth_tickets 2024-04-26T04:18:52.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.286+0000 7f7c6dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:52.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.286+0000 7f7c6dffb640 1 -- 172.21.15.63:0/1180331335 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c640ab600 con 0x7f7c70072610 2024-04-26T04:18:52.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.286+0000 7f7c6dffb640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:52.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.286+0000 7f7c6dffb640 1 -- 172.21.15.63:0/1180331335 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c640ab9d0 con 0x7f7c70072610 2024-04-26T04:18:52.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.286+0000 7f7c75275640 5 monclient: authenticate success, global_id 4317 2024-04-26T04:18:52.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.286+0000 7f7c75275640 10 monclient: get_monmap_and_config success 2024-04-26T04:18:52.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.286+0000 7f7c75275640 4 set_mon_vals no callback set 2024-04-26T04:18:52.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.286+0000 7f7c6dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:52.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.286+0000 7f7c6dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:18:52.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.286+0000 7f7c6dffb640 10 monclient: dump: 2024-04-26T04:18:52.295 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:52.296 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:52.296 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:52.296 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:52.296 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:52.296 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:52.296 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:52.296 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:52.296 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:52.296 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:52.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.286+0000 7f7c75275640 10 monclient: shutdown 2024-04-26T04:18:52.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.286+0000 7f7c75275640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:52.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.286+0000 7f7c75275640 1 -- 172.21.15.63:0/1180331335 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c70072610 msgr2=0x7f7c70139d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:52.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.286+0000 7f7c75275640 1 --2- 172.21.15.63:0/1180331335 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c70072610 0x7f7c70139d10 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7c64009870 tx=0x7f7c6401c3e0 comp rx=0 tx=0).stop 2024-04-26T04:18:52.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.286+0000 7f7c75275640 1 -- 172.21.15.63:0/1180331335 shutdown_connections 2024-04-26T04:18:52.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.286+0000 7f7c75275640 1 --2- 172.21.15.63:0/1180331335 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f7c700a1bc0 0x7f7c7013c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:52.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.286+0000 7f7c75275640 1 --2- 172.21.15.63:0/1180331335 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c70072610 0x7f7c70139d10 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:52.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.286+0000 7f7c75275640 1 -- 172.21.15.63:0/1180331335 >> 172.21.15.63:0/1180331335 conn(0x7f7c700a7390 msgr2=0x7f7c700a77a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:52.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.286+0000 7f7c75275640 1 -- 172.21.15.63:0/1180331335 shutdown_connections 2024-04-26T04:18:52.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.286+0000 7f7c75275640 1 -- 172.21.15.63:0/1180331335 wait complete. 2024-04-26T04:18:52.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.290+0000 7f7c75275640 10 monclient: build_initial_monmap 2024-04-26T04:18:52.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.290+0000 7f7c75275640 10 monclient: monmap: 2024-04-26T04:18:52.297 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:52.297 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:18:52.297 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:18:52.297 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:18:52.297 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:52.297 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:52.297 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:18:52.297 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:18:52.297 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:18:52.298 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:52.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.290+0000 7f7c75275640 1 Processor -- start 2024-04-26T04:18:52.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.290+0000 7f7c75275640 1 -- start start 2024-04-26T04:18:52.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.290+0000 7f7c75275640 10 monclient: init 2024-04-26T04:18:52.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.290+0000 7f7c75275640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:52.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.290+0000 7f7c75275640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T04:18:52.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.290+0000 7f7c75275640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c70072610 0x7f7c701333c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:52.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.290+0000 7f7c75275640 10 monclient(hunting): picked mon.noname-a con 0x7f7c70072610 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:18:52.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.290+0000 7f7c75275640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f7c700a1bc0 0x7f7c7013af10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:52.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.290+0000 7f7c75275640 10 monclient(hunting): picked mon.noname-c con 0x7f7c700a1bc0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:52.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.290+0000 7f7c75275640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c70133dc0 0x7f7c7006e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:52.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.290+0000 7f7c75275640 10 monclient(hunting): picked mon.noname-b con 0x7f7c70133dc0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:18:52.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.290+0000 7f7c75275640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:52.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.290+0000 7f7c75275640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f7c7013f350 con 0x7f7c70072610 2024-04-26T04:18:52.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.290+0000 7f7c75275640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:52.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.290+0000 7f7c75275640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7c7013f1d0 con 0x7f7c70133dc0 2024-04-26T04:18:52.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.290+0000 7f7c75275640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:52.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.290+0000 7f7c75275640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f7c7013f4d0 con 0x7f7c700a1bc0 2024-04-26T04:18:52.304 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.290+0000 7f7c75275640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:52.304 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.290+0000 7f7c75275640 10 monclient(hunting): authenticate will time out at 964.001381s 2024-04-26T04:18:52.304 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.290+0000 7f7c6e7fc640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f7c700a1bc0 0x7f7c7013af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:52.304 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.290+0000 7f7c6effd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c70072610 0x7f7c701333c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:52.304 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.290+0000 7f7c6e7fc640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f7c700a1bc0 0x7f7c7013af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:36952/0 (socket says 172.21.15.63:36952) 2024-04-26T04:18:52.304 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.290+0000 7f7c6e7fc640 1 -- 172.21.15.63:0/3033863402 learned_addr learned my addr 172.21.15.63:0/3033863402 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:18:52.304 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.290+0000 7f7c6e7fc640 10 monclient(hunting): get_auth_request con 0x7f7c700a1bc0 auth_method 0 2024-04-26T04:18:52.304 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.290+0000 7f7c6e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:52.304 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.290+0000 7f7c6e7fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:52.304 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.290+0000 7f7c6e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:52.304 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.290+0000 7f7c6effd640 10 monclient(hunting): get_auth_request con 0x7f7c70072610 auth_method 0 2024-04-26T04:18:52.304 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.290+0000 7f7c6effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:52.304 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.290+0000 7f7c6effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:52.304 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.290+0000 7f7c6effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:52.305 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.290+0000 7f7c6e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:52.305 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.290+0000 7f7c6e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:52.305 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.290+0000 7f7c6e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:52.305 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.290+0000 7f7c6effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:52.305 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.290+0000 7f7c6effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:52.305 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.290+0000 7f7c6f7fe640 1 --2- 172.21.15.63:0/3033863402 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c70133dc0 0x7f7c7006e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:52.305 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.290+0000 7f7c6effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:52.305 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.290+0000 7f7c6f7fe640 10 monclient(hunting): get_auth_request con 0x7f7c70133dc0 auth_method 0 2024-04-26T04:18:52.305 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.290+0000 7f7c6f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:52.305 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.290+0000 7f7c6f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:52.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.290+0000 7f7c6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:52.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.290+0000 7f7c6e7fc640 10 monclient(hunting): handle_auth_done global_id 4301 payload 824 2024-04-26T04:18:52.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.290+0000 7f7c6e7fc640 1 -- 172.21.15.63:0/3033863402 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c70133dc0 msgr2=0x7f7c7006e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:52.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.290+0000 7f7c6e7fc640 1 --2- 172.21.15.63:0/3033863402 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c70133dc0 0x7f7c7006e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:52.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.290+0000 7f7c6e7fc640 1 -- 172.21.15.63:0/3033863402 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c70072610 msgr2=0x7f7c701333c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:52.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.290+0000 7f7c6e7fc640 1 --2- 172.21.15.63:0/3033863402 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c70072610 0x7f7c701333c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:52.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.290+0000 7f7c6e7fc640 10 monclient: _finish_hunting 0 2024-04-26T04:18:52.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.290+0000 7f7c6e7fc640 1 monclient: found mon.noname-c 2024-04-26T04:18:52.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.290+0000 7f7c6e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:52.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.290+0000 7f7c6e7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.63:3301/0 2024-04-26T04:18:52.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.290+0000 7f7c6e7fc640 1 -- 172.21.15.63:0/3033863402 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c7006feb0 con 0x7f7c700a1bc0 2024-04-26T04:18:52.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.294+0000 7f7c6e7fc640 1 -- 172.21.15.63:0/3033863402 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f7c700a1bc0 msgr2=0x7f7c7013af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:52.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.294+0000 7f7c6e7fc640 1 --2- 172.21.15.63:0/3033863402 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f7c700a1bc0 0x7f7c7013af10 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7c6401cc80 tx=0x7f7c64019180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:52.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.294+0000 7f7c577fe640 1 -- 172.21.15.63:0/3033863402 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c640b9020 con 0x7f7c700a1bc0 2024-04-26T04:18:52.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.294+0000 7f7c577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:52.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.294+0000 7f7c577fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:18:52.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.294+0000 7f7c577fe640 10 monclient: dump: 2024-04-26T04:18:52.307 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:52.307 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:52.307 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:52.307 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:52.307 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:52.307 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:52.307 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:52.308 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:52.308 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:52.308 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:52.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.294+0000 7f7c577fe640 10 monclient: _finish_auth 0 2024-04-26T04:18:52.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.294+0000 7f7c577fe640 10 monclient: _check_auth_tickets 2024-04-26T04:18:52.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.294+0000 7f7c577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:52.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.294+0000 7f7c577fe640 1 -- 172.21.15.63:0/3033863402 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c640b1040 con 0x7f7c700a1bc0 2024-04-26T04:18:52.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.294+0000 7f7c577fe640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:52.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.294+0000 7f7c577fe640 1 -- 172.21.15.63:0/3033863402 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c640b0050 con 0x7f7c700a1bc0 2024-04-26T04:18:52.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.294+0000 7f7c75275640 5 monclient: authenticate success, global_id 4301 2024-04-26T04:18:52.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.294+0000 7f7c75275640 10 monclient: _renew_subs 2024-04-26T04:18:52.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.294+0000 7f7c75275640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:52.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.294+0000 7f7c75275640 1 -- 172.21.15.63:0/3033863402 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c70138c10 con 0x7f7c700a1bc0 2024-04-26T04:18:52.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.294+0000 7f7c6cff9640 4 set_mon_vals no callback set 2024-04-26T04:18:52.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.294+0000 7f7c577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:52.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.294+0000 7f7c577fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:18:52.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.294+0000 7f7c577fe640 10 monclient: dump: 2024-04-26T04:18:52.309 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:52.309 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:52.309 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:52.309 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:52.309 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:52.309 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:52.309 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:52.309 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:52.309 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:52.309 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:52.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.294+0000 7f7c75275640 10 monclient: _renew_subs 2024-04-26T04:18:52.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.294+0000 7f7c75275640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:52.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.294+0000 7f7c75275640 1 -- 172.21.15.63:0/3033863402 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c7013b450 con 0x7f7c700a1bc0 2024-04-26T04:18:52.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.294+0000 7f7c75275640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:52.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.294+0000 7f7c75275640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:52.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.294+0000 7f7c75275640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:52.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.294+0000 7f7c75275640 1 -- 172.21.15.63:0/3033863402 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c70062e10 con 0x7f7c700a1bc0 2024-04-26T04:18:52.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.294+0000 7f7c577fe640 1 -- 172.21.15.63:0/3033863402 <== mon.2 v2:172.21.15.63:3301/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f7c640bc020 con 0x7f7c700a1bc0 2024-04-26T04:18:52.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.294+0000 7f7c577fe640 1 --2- 172.21.15.63:0/3033863402 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f7c5c035ec0 0x7f7c5c038380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:52.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.294+0000 7f7c577fe640 1 -- 172.21.15.63:0/3033863402 <== mon.2 v2:172.21.15.63:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7f7c640e2b60 con 0x7f7c700a1bc0 2024-04-26T04:18:52.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.294+0000 7f7c6effd640 1 --2- 172.21.15.63:0/3033863402 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f7c5c035ec0 0x7f7c5c038380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:52.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.294+0000 7f7c6effd640 10 monclient: get_auth_request con 0x7f7c5c035ec0 auth_method 0 2024-04-26T04:18:52.311 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.298+0000 7f7c6effd640 1 --2- 172.21.15.63:0/3033863402 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f7c5c035ec0 0x7f7c5c038380 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7c60007fd0 tx=0x7f7c60007e40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:52.311 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.298+0000 7f7c577fe640 1 -- 172.21.15.63:0/3033863402 <== mon.2 v2:172.21.15.63:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f7c640c3030 con 0x7f7c700a1bc0 2024-04-26T04:18:52.311 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.298+0000 7f7c577fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:52.311 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.298+0000 7f7c577fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:18:52.463 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.458+0000 7f7c75275640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-04-26T04:18:52.463 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.458+0000 7f7c75275640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-04-26T04:18:52.463 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.458+0000 7f7c75275640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:52.463 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.458+0000 7f7c75275640 1 -- 172.21.15.63:0/3033863402 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) -- 0x7f7c70068460 con 0x7f7c700a1bc0 2024-04-26T04:18:52.464 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.458+0000 7f7c577fe640 1 -- 172.21.15.63:0/3033863402 <== mon.2 v2:172.21.15.63:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v15) ==== 92+0+10 (secure 0 0 0) 0x7f7c640160f0 con 0x7f7c700a1bc0 2024-04-26T04:18:52.464 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.458+0000 7f7c577fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-04-26T04:18:52.464 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.458+0000 7f7c577fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:18:52.464 INFO:teuthology.orchestra.run.smithi063.stdout:pg_num: 8 2024-04-26T04:18:52.466 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.462+0000 7f7c75275640 1 -- 172.21.15.63:0/3033863402 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f7c5c035ec0 msgr2=0x7f7c5c038380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:52.466 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.462+0000 7f7c75275640 1 --2- 172.21.15.63:0/3033863402 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f7c5c035ec0 0x7f7c5c038380 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7c60007fd0 tx=0x7f7c60007e40 comp rx=0 tx=0).stop 2024-04-26T04:18:52.466 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.462+0000 7f7c75275640 10 monclient: shutdown 2024-04-26T04:18:52.466 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.462+0000 7f7c75275640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:52.466 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.462+0000 7f7c75275640 1 -- 172.21.15.63:0/3033863402 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f7c700a1bc0 msgr2=0x7f7c7013af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:52.466 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.462+0000 7f7c75275640 1 --2- 172.21.15.63:0/3033863402 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f7c700a1bc0 0x7f7c7013af10 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7c6401cc80 tx=0x7f7c64019180 comp rx=0 tx=0).stop 2024-04-26T04:18:52.466 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.462+0000 7f7c75275640 1 -- 172.21.15.63:0/3033863402 shutdown_connections 2024-04-26T04:18:52.466 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.462+0000 7f7c75275640 1 --2- 172.21.15.63:0/3033863402 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f7c5c035ec0 0x7f7c5c038380 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:52.466 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.462+0000 7f7c75275640 1 --2- 172.21.15.63:0/3033863402 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c70133dc0 0x7f7c7006e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:52.466 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.462+0000 7f7c75275640 1 --2- 172.21.15.63:0/3033863402 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f7c700a1bc0 0x7f7c7013af10 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:52.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.462+0000 7f7c75275640 1 --2- 172.21.15.63:0/3033863402 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c70072610 0x7f7c701333c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:52.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.462+0000 7f7c75275640 1 -- 172.21.15.63:0/3033863402 >> 172.21.15.63:0/3033863402 conn(0x7f7c700a7390 msgr2=0x7f7c7009e5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:52.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.462+0000 7f7c75275640 1 -- 172.21.15.63:0/3033863402 shutdown_connections 2024-04-26T04:18:52.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.462+0000 7f7c75275640 1 -- 172.21.15.63:0/3033863402 wait complete. 2024-04-26T04:18:52.481 INFO:tasks.mon_thrash.ceph_manager:waiting for clean 2024-04-26T04:18:52.482 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T04:18:52.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.594+0000 7f77c3fd1640 10 monclient: get_monmap_and_config 2024-04-26T04:18:52.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.594+0000 7f77c3fd1640 10 monclient: build_initial_monmap 2024-04-26T04:18:52.606 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.594+0000 7f77c3fd1640 10 monclient: monmap: 2024-04-26T04:18:52.606 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:52.606 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:52.606 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:52.601394+0000 2024-04-26T04:18:52.606 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:52.601394+0000 2024-04-26T04:18:52.606 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:52.606 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:52.606 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:18:52.606 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:18:52.606 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:18:52.606 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:18:52.606 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:18:52.606 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:52.606 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.598+0000 7f77c3fd1640 1 Processor -- start 2024-04-26T04:18:52.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.598+0000 7f77c3fd1640 1 -- start start 2024-04-26T04:18:52.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.598+0000 7f77c3fd1640 10 monclient: init 2024-04-26T04:18:52.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.598+0000 7f77c3fd1640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:52.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.598+0000 7f77c3fd1640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-04-26T04:18:52.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.598+0000 7f77c3fd1640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f77bc135ef0 0x7f77bc09bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:52.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.598+0000 7f77c3fd1640 10 monclient(hunting): picked mon.noname-a con 0x7f77bc135ef0 addr v2:172.21.15.63:3300/0 2024-04-26T04:18:52.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.598+0000 7f77c3fd1640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f77bc1362d0 0x7f77bc072af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:52.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.598+0000 7f77c3fd1640 10 monclient(hunting): picked mon.noname-c con 0x7f77bc1362d0 addr v2:172.21.15.186:3300/0 2024-04-26T04:18:52.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.598+0000 7f77c3fd1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f77bc09fcb0 addr v1:172.21.15.186:6789/0 2024-04-26T04:18:52.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.598+0000 7f77c3fd1640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77bc0a0d60 con 0x7f77bc09fcb0 2024-04-26T04:18:52.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.598+0000 7f77c3fd1640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:52.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.598+0000 7f77c3fd1640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7f77bc0a1040 con 0x7f77bc135ef0 2024-04-26T04:18:52.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.598+0000 7f77c3fd1640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:52.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.598+0000 7f77c3fd1640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f77bc073030 con 0x7f77bc1362d0 2024-04-26T04:18:52.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.598+0000 7f77c3fd1640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:52.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.598+0000 7f77c3fd1640 10 monclient(hunting): authenticate will time out at 964.309128s 2024-04-26T04:18:52.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.598+0000 7f77c1d6d640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f77bc135ef0 0x7f77bc09bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:52.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.598+0000 7f77c1d6d640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f77bc135ef0 0x7f77bc09bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:39334/0 (socket says 172.21.15.63:39334) 2024-04-26T04:18:52.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.598+0000 7f77c1d6d640 1 -- 172.21.15.63:0/3534931193 learned_addr learned my addr 172.21.15.63:0/3534931193 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:18:52.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.598+0000 7f77c1d6d640 10 monclient(hunting): get_auth_request con 0x7f77bc135ef0 auth_method 0 2024-04-26T04:18:52.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.598+0000 7f77c1d6d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:52.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.598+0000 7f77c1d6d640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:52.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.598+0000 7f77c1d6d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:52.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.598+0000 7f77c1d6d640 1 -- 172.21.15.63:0/3534931193 >> v2:172.21.15.63:3300/0 conn(0x7f77bc135ef0 msgr2=0x7f77bc09bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T04:18:52.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.598+0000 7f77c1d6d640 1 -- 172.21.15.63:0/3534931193 >> v2:172.21.15.63:3300/0 conn(0x7f77bc135ef0 msgr2=0x7f77bc09bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:18:52.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.598+0000 7f77c1d6d640 1 --2- 172.21.15.63:0/3534931193 >> v2:172.21.15.63:3300/0 conn(0x7f77bc135ef0 0x7f77bc09bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:18:52.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.598+0000 7f77c1d6d640 1 --2- 172.21.15.63:0/3534931193 >> v2:172.21.15.63:3300/0 conn(0x7f77bc135ef0 0x7f77bc09bcb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:18:52.609 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.598+0000 7f77c156c640 1 --2- 172.21.15.63:0/3534931193 >> v2:172.21.15.186:3300/0 conn(0x7f77bc1362d0 0x7f77bc072af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:52.609 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.598+0000 7f77c156c640 10 monclient(hunting): get_auth_request con 0x7f77bc1362d0 auth_method 0 2024-04-26T04:18:52.609 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.598+0000 7f77c156c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:52.609 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.598+0000 7f77c156c640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:52.609 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.598+0000 7f77c156c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:52.609 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.598+0000 7f77c156c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:52.609 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.598+0000 7f77c156c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:52.609 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.598+0000 7f77c156c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:52.609 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.598+0000 7f77c156c640 10 monclient(hunting): handle_auth_done global_id 4297 payload 274 2024-04-26T04:18:52.609 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.598+0000 7f77c156c640 1 -- 172.21.15.63:0/3534931193 >> v2:172.21.15.63:3300/0 conn(0x7f77bc135ef0 msgr2=0x7f77bc09bcb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T04:18:52.609 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.598+0000 7f77c156c640 1 --2- 172.21.15.63:0/3534931193 >> v2:172.21.15.63:3300/0 conn(0x7f77bc135ef0 0x7f77bc09bcb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:52.609 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.598+0000 7f77c156c640 1 -- 172.21.15.63:0/3534931193 >> v1:172.21.15.186:6789/0 conn(0x7f77bc09fcb0 legacy=0x7f77bc0a00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:52.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.598+0000 7f77c156c640 10 monclient: _finish_hunting 0 2024-04-26T04:18:52.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.598+0000 7f77c156c640 1 monclient: found mon.noname-c 2024-04-26T04:18:52.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.598+0000 7f77c156c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:52.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.598+0000 7f77c156c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.186:3300/0 2024-04-26T04:18:52.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.598+0000 7f77c156c640 1 -- 172.21.15.63:0/3534931193 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77bc1366e0 con 0x7f77bc1362d0 2024-04-26T04:18:52.617 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.602+0000 7f77c156c640 1 -- 172.21.15.63:0/3534931193 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f77bc1362d0 msgr2=0x7f77bc072af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:52.617 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.602+0000 7f77c156c640 1 --2- 172.21.15.63:0/3534931193 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f77bc1362d0 0x7f77bc072af0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f77bc0a1ce0 tx=0x7f77ac01c300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e4b3d243a0970ebd server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:52.617 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.602+0000 7f77c0d6b640 1 -- 172.21.15.63:0/3534931193 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77ac0b0070 con 0x7f77bc1362d0 2024-04-26T04:18:52.617 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.602+0000 7f77c0d6b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:52.618 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.602+0000 7f77c0d6b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:18:52.618 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.602+0000 7f77c0d6b640 10 monclient: dump: 2024-04-26T04:18:52.618 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:52.618 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:52.618 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:52.618 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:52.618 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:52.618 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:52.618 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:52.618 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:52.618 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:52.618 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:52.618 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.602+0000 7f77c0d6b640 10 monclient: _finish_auth 0 2024-04-26T04:18:52.618 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.602+0000 7f77c0d6b640 10 monclient: _check_auth_tickets 2024-04-26T04:18:52.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.602+0000 7f77c0d6b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:52.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.602+0000 7f77c0d6b640 1 -- 172.21.15.63:0/3534931193 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77ac0a9710 con 0x7f77bc1362d0 2024-04-26T04:18:52.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.602+0000 7f77c3fd1640 5 monclient: authenticate success, global_id 4297 2024-04-26T04:18:52.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.602+0000 7f77c3fd1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T04:18:52.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.602+0000 7f77c0d6b640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:52.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.602+0000 7f77c3fd1640 10 monclient: get_monmap_and_config success 2024-04-26T04:18:52.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.602+0000 7f77c3fd1640 4 set_mon_vals no callback set 2024-04-26T04:18:52.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.602+0000 7f77c0d6b640 1 -- 172.21.15.63:0/3534931193 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77ac0a9ae0 con 0x7f77bc1362d0 2024-04-26T04:18:52.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.602+0000 7f77c0d6b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:52.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.602+0000 7f77c0d6b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T04:18:52.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.602+0000 7f77c0d6b640 10 monclient: dump: 2024-04-26T04:18:52.619 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:52.619 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:52.619 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:52.620 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:52.620 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:52.620 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:52.620 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:52.620 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:52.620 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:52.620 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:52.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.602+0000 7f77c3fd1640 10 monclient: shutdown 2024-04-26T04:18:52.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.602+0000 7f77c3fd1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:52.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.602+0000 7f77c3fd1640 1 -- 172.21.15.63:0/3534931193 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f77bc1362d0 msgr2=0x7f77bc072af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:52.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.602+0000 7f77c3fd1640 1 --2- 172.21.15.63:0/3534931193 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f77bc1362d0 0x7f77bc072af0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f77bc0a1ce0 tx=0x7f77ac01c300 comp rx=0 tx=0).stop 2024-04-26T04:18:52.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.602+0000 7f77c3fd1640 1 -- 172.21.15.63:0/3534931193 shutdown_connections 2024-04-26T04:18:52.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.602+0000 7f77c3fd1640 1 --2- 172.21.15.63:0/3534931193 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f77bc1362d0 0x7f77bc072af0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:52.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.602+0000 7f77c3fd1640 1 --2- 172.21.15.63:0/3534931193 >> v2:172.21.15.63:3300/0 conn(0x7f77bc135ef0 0x7f77bc09bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:52.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.602+0000 7f77c3fd1640 1 -- 172.21.15.63:0/3534931193 >> 172.21.15.63:0/3534931193 conn(0x7f77bc0a1db0 msgr2=0x7f77bc0a2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:52.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.602+0000 7f77c3fd1640 1 -- 172.21.15.63:0/3534931193 shutdown_connections 2024-04-26T04:18:52.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.606+0000 7f77c3fd1640 1 -- 172.21.15.63:0/3534931193 wait complete. 2024-04-26T04:18:52.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.606+0000 7f77c3fd1640 10 monclient: build_initial_monmap 2024-04-26T04:18:52.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.606+0000 7f77c3fd1640 10 monclient: monmap: 2024-04-26T04:18:52.621 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:52.621 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:18:52.621 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:18:52.621 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:18:52.621 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:52.621 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:52.621 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:18:52.621 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:18:52.621 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:18:52.621 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:52.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.606+0000 7f77c3fd1640 1 Processor -- start 2024-04-26T04:18:52.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.606+0000 7f77c3fd1640 1 -- start start 2024-04-26T04:18:52.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.606+0000 7f77c3fd1640 10 monclient: init 2024-04-26T04:18:52.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.606+0000 7f77c3fd1640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:52.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.606+0000 7f77c3fd1640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T04:18:52.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.606+0000 7f77c3fd1640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f77bc09fcb0 0x7f77bc1cef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:52.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.606+0000 7f77c3fd1640 10 monclient(hunting): picked mon.noname-c con 0x7f77bc09fcb0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:52.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.606+0000 7f77c3fd1640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f77bc135ef0 0x7f77bc135070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:52.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.606+0000 7f77c3fd1640 10 monclient(hunting): picked mon.noname-a con 0x7f77bc135ef0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:18:52.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.606+0000 7f77c3fd1640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f77bc1362d0 0x7f77bc1cba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:52.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.606+0000 7f77c3fd1640 10 monclient(hunting): picked mon.noname-b con 0x7f77bc1362d0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:18:52.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.606+0000 7f77c3fd1640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:52.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.606+0000 7f77c3fd1640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f77bc1418d0 con 0x7f77bc135ef0 2024-04-26T04:18:52.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.606+0000 7f77c3fd1640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:52.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.606+0000 7f77c3fd1640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f77bc0731b0 con 0x7f77bc1362d0 2024-04-26T04:18:52.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.606+0000 7f77c3fd1640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:52.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.606+0000 7f77c3fd1640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f77bc141a50 con 0x7f77bc09fcb0 2024-04-26T04:18:52.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.606+0000 7f77c3fd1640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:52.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.606+0000 7f77c3fd1640 10 monclient(hunting): authenticate will time out at 964.317449s 2024-04-26T04:18:52.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.606+0000 7f77c156c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f77bc135ef0 0x7f77bc135070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:52.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.606+0000 7f77c1d6d640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f77bc09fcb0 0x7f77bc1cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:52.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.606+0000 7f77c156c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f77bc135ef0 0x7f77bc135070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:39336/0 (socket says 172.21.15.63:39336) 2024-04-26T04:18:52.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.606+0000 7f77c156c640 1 -- 172.21.15.63:0/2054657208 learned_addr learned my addr 172.21.15.63:0/2054657208 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:18:52.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.606+0000 7f77c156c640 10 monclient(hunting): get_auth_request con 0x7f77bc135ef0 auth_method 0 2024-04-26T04:18:52.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.606+0000 7f77c156c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:52.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.606+0000 7f77c156c640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:52.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.606+0000 7f77c156c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:52.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.606+0000 7f77c1d6d640 10 monclient(hunting): get_auth_request con 0x7f77bc09fcb0 auth_method 0 2024-04-26T04:18:52.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.606+0000 7f77c1d6d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:52.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.606+0000 7f77c1d6d640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:52.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.606+0000 7f77c1d6d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:52.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.606+0000 7f77c156c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:52.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.606+0000 7f77c156c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:52.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.606+0000 7f77c156c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:52.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.606+0000 7f77c1d6d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:52.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.606+0000 7f77c1d6d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:52.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.606+0000 7f77c1d6d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:52.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.606+0000 7f77c256e640 1 --2- 172.21.15.63:0/2054657208 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f77bc1362d0 0x7f77bc1cba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:52.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.606+0000 7f77c256e640 10 monclient(hunting): get_auth_request con 0x7f77bc1362d0 auth_method 0 2024-04-26T04:18:52.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.606+0000 7f77c256e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:52.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.606+0000 7f77c256e640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:52.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.606+0000 7f77c256e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:52.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.606+0000 7f77c1d6d640 10 monclient(hunting): handle_auth_done global_id 4304 payload 824 2024-04-26T04:18:52.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.606+0000 7f77c1d6d640 1 -- 172.21.15.63:0/2054657208 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f77bc1362d0 msgr2=0x7f77bc1cba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:52.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.606+0000 7f77c1d6d640 1 --2- 172.21.15.63:0/2054657208 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f77bc1362d0 0x7f77bc1cba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:52.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.606+0000 7f77c1d6d640 1 -- 172.21.15.63:0/2054657208 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f77bc135ef0 msgr2=0x7f77bc135070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:52.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.606+0000 7f77c1d6d640 1 --2- 172.21.15.63:0/2054657208 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f77bc135ef0 0x7f77bc135070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:52.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.606+0000 7f77c1d6d640 10 monclient: _finish_hunting 0 2024-04-26T04:18:52.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.606+0000 7f77c1d6d640 1 monclient: found mon.noname-c 2024-04-26T04:18:52.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.606+0000 7f77c1d6d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:52.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.606+0000 7f77c1d6d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.63:3301/0 2024-04-26T04:18:52.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.606+0000 7f77c1d6d640 1 -- 172.21.15.63:0/2054657208 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77bc09eb70 con 0x7f77bc09fcb0 2024-04-26T04:18:52.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.606+0000 7f77c156c640 1 --2- 172.21.15.63:0/2054657208 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f77bc135ef0 0x7f77bc135070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T04:18:52.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.610+0000 7f77c1d6d640 1 -- 172.21.15.63:0/2054657208 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f77bc09fcb0 msgr2=0x7f77bc1cef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:52.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.610+0000 7f77c1d6d640 1 --2- 172.21.15.63:0/2054657208 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f77bc09fcb0 0x7f77bc1cef70 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f77b800e810 tx=0x7f77b800ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:52.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.610+0000 7f77b27fc640 1 -- 172.21.15.63:0/2054657208 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77b8017070 con 0x7f77bc09fcb0 2024-04-26T04:18:52.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.610+0000 7f77b27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:52.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.610+0000 7f77b27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:18:52.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.610+0000 7f77b27fc640 10 monclient: dump: 2024-04-26T04:18:52.626 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:52.626 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:52.626 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:52.626 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:52.626 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:52.626 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:52.626 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:52.626 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:52.626 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:52.626 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:52.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.610+0000 7f77b27fc640 10 monclient: _finish_auth 0 2024-04-26T04:18:52.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.610+0000 7f77b27fc640 10 monclient: _check_auth_tickets 2024-04-26T04:18:52.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.610+0000 7f77b27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:52.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.610+0000 7f77c3fd1640 5 monclient: authenticate success, global_id 4304 2024-04-26T04:18:52.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.610+0000 7f77c3fd1640 10 monclient: _renew_subs 2024-04-26T04:18:52.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.610+0000 7f77c3fd1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:52.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.610+0000 7f77c3fd1640 1 -- 172.21.15.63:0/2054657208 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77bc09e0b0 con 0x7f77bc09fcb0 2024-04-26T04:18:52.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.610+0000 7f77b27fc640 1 -- 172.21.15.63:0/2054657208 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77b8004260 con 0x7f77bc09fcb0 2024-04-26T04:18:52.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.610+0000 7f77b27fc640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:52.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.610+0000 7f77b27fc640 1 -- 172.21.15.63:0/2054657208 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77b800f040 con 0x7f77bc09fcb0 2024-04-26T04:18:52.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.610+0000 7f77b27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:52.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.610+0000 7f77b27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:18:52.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.610+0000 7f77b27fc640 10 monclient: dump: 2024-04-26T04:18:52.627 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:52.627 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:52.627 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:52.627 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:52.627 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:52.628 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:52.628 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:52.628 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:52.628 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:52.628 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:52.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.610+0000 7f77b37fe640 4 set_mon_vals no callback set 2024-04-26T04:18:52.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.610+0000 7f77c3fd1640 10 monclient: _renew_subs 2024-04-26T04:18:52.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.610+0000 7f77c3fd1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:52.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.610+0000 7f77c3fd1640 1 -- 172.21.15.63:0/2054657208 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f77bc11fa20 con 0x7f77bc09fcb0 2024-04-26T04:18:52.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.610+0000 7f77c3fd1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:52.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.610+0000 7f77c3fd1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:52.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.610+0000 7f77c3fd1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:52.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.610+0000 7f77c3fd1640 1 -- 172.21.15.63:0/2054657208 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77bc062e10 con 0x7f77bc09fcb0 2024-04-26T04:18:52.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.610+0000 7f77b27fc640 1 -- 172.21.15.63:0/2054657208 <== mon.2 v2:172.21.15.63:3301/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f77b800c6d0 con 0x7f77bc09fcb0 2024-04-26T04:18:52.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.610+0000 7f77b27fc640 1 --2- 172.21.15.63:0/2054657208 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f77a0035ec0 0x7f77a0038380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:52.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.610+0000 7f77b27fc640 1 -- 172.21.15.63:0/2054657208 <== mon.2 v2:172.21.15.63:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7f77b8048870 con 0x7f77bc09fcb0 2024-04-26T04:18:52.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.610+0000 7f77c156c640 1 --2- 172.21.15.63:0/2054657208 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f77a0035ec0 0x7f77a0038380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:52.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.610+0000 7f77c156c640 10 monclient: get_auth_request con 0x7f77a0035ec0 auth_method 0 2024-04-26T04:18:52.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.614+0000 7f77c156c640 1 --2- 172.21.15.63:0/2054657208 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f77a0035ec0 0x7f77a0038380 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f77ac0ae040 tx=0x7f77ac0a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:52.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.614+0000 7f77b27fc640 1 -- 172.21.15.63:0/2054657208 <== mon.2 v2:172.21.15.63:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f77b8023350 con 0x7f77bc09fcb0 2024-04-26T04:18:52.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.614+0000 7f77b27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:52.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.614+0000 7f77b27fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:18:52.777 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.770+0000 7f77c3fd1640 1 -- 172.21.15.63:0/2054657208 --> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f77bc11f5f0 con 0x7f77a0035ec0 2024-04-26T04:18:52.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.774+0000 7f77b27fc640 1 -- 172.21.15.63:0/2054657208 <== mgr.4105 v2:172.21.15.186:6824/2583706639 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33770 (secure 0 0 0) 0x7f77ac0a96f0 con 0x7f77a0035ec0 2024-04-26T04:18:52.781 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-26T04:18:52.781 INFO:teuthology.orchestra.run.smithi063.stderr:dumped all 2024-04-26T04:18:52.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.778+0000 7f77c3fd1640 1 -- 172.21.15.63:0/2054657208 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f77a0035ec0 msgr2=0x7f77a0038380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:52.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.778+0000 7f77c3fd1640 1 --2- 172.21.15.63:0/2054657208 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f77a0035ec0 0x7f77a0038380 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f77ac0ae040 tx=0x7f77ac0a6000 comp rx=0 tx=0).stop 2024-04-26T04:18:52.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.778+0000 7f77c3fd1640 10 monclient: shutdown 2024-04-26T04:18:52.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.778+0000 7f77c3fd1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:52.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.778+0000 7f77c3fd1640 1 -- 172.21.15.63:0/2054657208 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f77bc09fcb0 msgr2=0x7f77bc1cef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:52.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.778+0000 7f77c3fd1640 1 --2- 172.21.15.63:0/2054657208 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f77bc09fcb0 0x7f77bc1cef70 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f77b800e810 tx=0x7f77b800ecd0 comp rx=0 tx=0).stop 2024-04-26T04:18:52.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.778+0000 7f77c3fd1640 1 -- 172.21.15.63:0/2054657208 shutdown_connections 2024-04-26T04:18:52.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.778+0000 7f77c3fd1640 1 --2- 172.21.15.63:0/2054657208 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f77a0035ec0 0x7f77a0038380 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:52.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.778+0000 7f77c3fd1640 1 --2- 172.21.15.63:0/2054657208 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f77bc1362d0 0x7f77bc1cba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:52.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.778+0000 7f77c3fd1640 1 --2- 172.21.15.63:0/2054657208 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f77bc135ef0 0x7f77bc135070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:52.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.778+0000 7f77c3fd1640 1 --2- 172.21.15.63:0/2054657208 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f77bc09fcb0 0x7f77bc1cef70 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:52.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.778+0000 7f77c3fd1640 1 -- 172.21.15.63:0/2054657208 >> 172.21.15.63:0/2054657208 conn(0x7f77bc0a1db0 msgr2=0x7f77bc09c6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:52.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.778+0000 7f77c3fd1640 1 -- 172.21.15.63:0/2054657208 shutdown_connections 2024-04-26T04:18:52.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.778+0000 7f77c3fd1640 1 -- 172.21.15.63:0/2054657208 wait complete. 2024-04-26T04:18:52.800 INFO:teuthology.orchestra.run.smithi063.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2024-04-26T04:18:51.447541+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":164460,"kb_used_data":1880,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566066580,"statfs":{"total":579820584960,"available":579652177920,"internally_reserved":0,"allocated":1925120,"data_stored":1313426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9567,"internal_metadata":165927585},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000724"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-26T04:18:47.497690+0000","last_change":"2024-04-26T04:18:47.131857+0000","last_active":"2024-04-26T04:18:47.497690+0000","last_peered":"2024-04-26T04:18:47.497690+0000","last_clean":"2024-04-26T04:18:47.497690+0000","last_became_active":"2024-04-26T04:18:47.131175+0000","last_became_peered":"2024-04-26T04:18:47.131175+0000","last_unstale":"2024-04-26T04:18:47.497690+0000","last_undegraded":"2024-04-26T04:18:47.497690+0000","last_fullsized":"2024-04-26T04:18:47.497690+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_clean_scrub_stamp":"2024-04-26T04:18:45.482773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:35:03.860574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-26T04:18:47.493565+0000","last_change":"2024-04-26T04:18:46.499700+0000","last_active":"2024-04-26T04:18:47.493565+0000","last_peered":"2024-04-26T04:18:47.493565+0000","last_clean":"2024-04-26T04:18:47.493565+0000","last_became_active":"2024-04-26T04:18:46.498983+0000","last_became_peered":"2024-04-26T04:18:46.498983+0000","last_unstale":"2024-04-26T04:18:47.493565+0000","last_undegraded":"2024-04-26T04:18:47.493565+0000","last_fullsized":"2024-04-26T04:18:47.493565+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_clean_scrub_stamp":"2024-04-26T04:18:45.482773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:51:56.740670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-26T04:18:47.497623+0000","last_change":"2024-04-26T04:18:46.507904+0000","last_active":"2024-04-26T04:18:47.497623+0000","last_peered":"2024-04-26T04:18:47.497623+0000","last_clean":"2024-04-26T04:18:47.497623+0000","last_became_active":"2024-04-26T04:18:46.507335+0000","last_became_peered":"2024-04-26T04:18:46.507335+0000","last_unstale":"2024-04-26T04:18:47.497623+0000","last_undegraded":"2024-04-26T04:18:47.497623+0000","last_fullsized":"2024-04-26T04:18:47.497623+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_clean_scrub_stamp":"2024-04-26T04:18:45.482773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:52:50.029369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-26T04:18:47.493398+0000","last_change":"2024-04-26T04:18:46.506682+0000","last_active":"2024-04-26T04:18:47.493398+0000","last_peered":"2024-04-26T04:18:47.493398+0000","last_clean":"2024-04-26T04:18:47.493398+0000","last_became_active":"2024-04-26T04:18:46.506209+0000","last_became_peered":"2024-04-26T04:18:46.506209+0000","last_unstale":"2024-04-26T04:18:47.493398+0000","last_undegraded":"2024-04-26T04:18:47.493398+0000","last_fullsized":"2024-04-26T04:18:47.493398+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_clean_scrub_stamp":"2024-04-26T04:18:45.482773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:20:00.475307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-26T04:18:47.836289+0000","last_change":"2024-04-26T04:18:46.504685+0000","last_active":"2024-04-26T04:18:47.836289+0000","last_peered":"2024-04-26T04:18:47.836289+0000","last_clean":"2024-04-26T04:18:47.836289+0000","last_became_active":"2024-04-26T04:18:46.504292+0000","last_became_peered":"2024-04-26T04:18:46.504292+0000","last_unstale":"2024-04-26T04:18:47.836289+0000","last_undegraded":"2024-04-26T04:18:47.836289+0000","last_fullsized":"2024-04-26T04:18:47.836289+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_clean_scrub_stamp":"2024-04-26T04:18:45.482773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:42:29.110482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-26T04:18:47.832700+0000","last_change":"2024-04-26T04:18:46.504010+0000","last_active":"2024-04-26T04:18:47.832700+0000","last_peered":"2024-04-26T04:18:47.832700+0000","last_clean":"2024-04-26T04:18:47.832700+0000","last_became_active":"2024-04-26T04:18:46.503343+0000","last_became_peered":"2024-04-26T04:18:46.503343+0000","last_unstale":"2024-04-26T04:18:47.832700+0000","last_undegraded":"2024-04-26T04:18:47.832700+0000","last_fullsized":"2024-04-26T04:18:47.832700+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_clean_scrub_stamp":"2024-04-26T04:18:45.482773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:19:38.324198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-26T04:18:47.497466+0000","last_change":"2024-04-26T04:18:46.503233+0000","last_active":"2024-04-26T04:18:47.497466+0000","last_peered":"2024-04-26T04:18:47.497466+0000","last_clean":"2024-04-26T04:18:47.497466+0000","last_became_active":"2024-04-26T04:18:46.502682+0000","last_became_peered":"2024-04-26T04:18:46.502682+0000","last_unstale":"2024-04-26T04:18:47.497466+0000","last_undegraded":"2024-04-26T04:18:47.497466+0000","last_fullsized":"2024-04-26T04:18:47.497466+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_clean_scrub_stamp":"2024-04-26T04:18:45.482773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:26:15.760818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-26T04:18:47.836255+0000","last_change":"2024-04-26T04:18:46.504844+0000","last_active":"2024-04-26T04:18:47.836255+0000","last_peered":"2024-04-26T04:18:47.836255+0000","last_clean":"2024-04-26T04:18:47.836255+0000","last_became_active":"2024-04-26T04:18:46.504332+0000","last_became_peered":"2024-04-26T04:18:46.504332+0000","last_unstale":"2024-04-26T04:18:47.836255+0000","last_undegraded":"2024-04-26T04:18:47.836255+0000","last_fullsized":"2024-04-26T04:18:47.836255+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_clean_scrub_stamp":"2024-04-26T04:18:45.482773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:13:26.538440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":164,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-26T04:18:47.497482+0000","last_change":"2024-04-26T04:18:45.505818+0000","last_active":"2024-04-26T04:18:47.497482+0000","last_peered":"2024-04-26T04:18:47.497482+0000","last_clean":"2024-04-26T04:18:47.497482+0000","last_became_active":"2024-04-26T04:18:45.504887+0000","last_became_peered":"2024-04-26T04:18:45.504887+0000","last_unstale":"2024-04-26T04:18:47.497482+0000","last_undegraded":"2024-04-26T04:18:47.497482+0000","last_fullsized":"2024-04-26T04:18:47.497482+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:44.470061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:44.470061+0000","last_clean_scrub_stamp":"2024-04-26T04:18:44.470061+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:42:56.629937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":12,"seq":51539607558,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27680,"kb_used_data":660,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344160,"statfs":{"total":96636764160,"available":96608419840,"internally_reserved":0,"allocated":675840,"data_stored":571523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":12,"seq":51539607557,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":143360,"data_stored":42595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607557,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":143360,"data_stored":42595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":143360,"data_stored":42595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640260,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":143360,"data_stored":42595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28128,"kb_used_data":660,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343712,"statfs":{"total":96636764160,"available":96607961088,"internally_reserved":0,"allocated":675840,"data_stored":571523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T04:18:52.801 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T04:18:52.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.918+0000 7f171ef5a640 10 monclient: get_monmap_and_config 2024-04-26T04:18:52.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.918+0000 7f171ef5a640 10 monclient: build_initial_monmap 2024-04-26T04:18:52.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.918+0000 7f171ef5a640 10 monclient: monmap: 2024-04-26T04:18:52.925 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:52.925 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:52.925 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:52.924538+0000 2024-04-26T04:18:52.925 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:52.924538+0000 2024-04-26T04:18:52.925 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:52.925 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:52.925 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:18:52.925 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:18:52.925 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:18:52.925 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:18:52.925 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:18:52.925 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:52.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.918+0000 7f171ef5a640 1 Processor -- start 2024-04-26T04:18:52.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.918+0000 7f171ef5a640 1 -- start start 2024-04-26T04:18:52.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.918+0000 7f171ef5a640 10 monclient: init 2024-04-26T04:18:52.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.922+0000 7f171ef5a640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:52.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.922+0000 7f171ef5a640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-04-26T04:18:52.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.922+0000 7f171ef5a640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f171809b3b0 0x7f171809e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:52.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.922+0000 7f171ef5a640 10 monclient(hunting): picked mon.noname-a con 0x7f171809b3b0 addr v2:172.21.15.63:3300/0 2024-04-26T04:18:52.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.922+0000 7f171ef5a640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f17180a0bb0 0x7f17180a0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:52.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.922+0000 7f171ef5a640 10 monclient(hunting): picked mon.noname-c con 0x7f17180a0bb0 addr v2:172.21.15.186:3300/0 2024-04-26T04:18:52.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.922+0000 7f171ef5a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f17180a1500 addr v1:172.21.15.63:6789/0 2024-04-26T04:18:52.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.922+0000 7f171ef5a640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f171809dde0 con 0x7f17180a1500 2024-04-26T04:18:52.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.922+0000 7f171ef5a640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:52.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.922+0000 7f171ef5a640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7f171809c100 con 0x7f171809b3b0 2024-04-26T04:18:52.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.922+0000 7f171ef5a640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:52.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.922+0000 7f171ef5a640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f171809b790 con 0x7f17180a0bb0 2024-04-26T04:18:52.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.922+0000 7f171ef5a640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:52.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.922+0000 7f171ef5a640 10 monclient(hunting): authenticate will time out at 964.632338s 2024-04-26T04:18:52.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.922+0000 7f171ccf6640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f171809b3b0 0x7f171809e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:52.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.922+0000 7f171d4f7640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f17180a1500 0x7f171813b340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:32810/0 (socket says 172.21.15.63:32810) 2024-04-26T04:18:52.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.922+0000 7f171d4f7640 1 -- 172.21.15.63:0/4095746348 learned_addr learned my addr 172.21.15.63:0/4095746348 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-04-26T04:18:52.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.922+0000 7f171ccf6640 10 monclient(hunting): get_auth_request con 0x7f171809b3b0 auth_method 0 2024-04-26T04:18:52.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.922+0000 7f171ccf6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:52.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.922+0000 7f171ccf6640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:52.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.922+0000 7f171ccf6640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:52.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.922+0000 7f171ccf6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:52.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.922+0000 7f171ccf6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:52.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.922+0000 7f171ccf6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:52.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.922+0000 7f171ccf6640 10 monclient(hunting): handle_auth_done global_id 4326 payload 274 2024-04-26T04:18:52.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.922+0000 7f1717fff640 1 --2- 172.21.15.63:0/4095746348 >> v2:172.21.15.186:3300/0 conn(0x7f17180a0bb0 0x7f17180a0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:52.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.922+0000 7f171ccf6640 1 -- 172.21.15.63:0/4095746348 >> v2:172.21.15.186:3300/0 conn(0x7f17180a0bb0 msgr2=0x7f17180a0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:52.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.922+0000 7f171ccf6640 1 --2- 172.21.15.63:0/4095746348 >> v2:172.21.15.186:3300/0 conn(0x7f17180a0bb0 0x7f17180a0fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:52.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.922+0000 7f171ccf6640 1 -- 172.21.15.63:0/4095746348 >> v1:172.21.15.63:6789/0 conn(0x7f17180a1500 legacy=0x7f171813b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:52.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.922+0000 7f171ccf6640 10 monclient: _finish_hunting 0 2024-04-26T04:18:52.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.922+0000 7f171ccf6640 1 monclient: found mon.noname-a 2024-04-26T04:18:52.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.922+0000 7f171ccf6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:52.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.922+0000 7f171ccf6640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:18:52.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.922+0000 7f171ccf6640 1 -- 172.21.15.63:0/4095746348 --> v2:172.21.15.63:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f171809c9b0 con 0x7f171809b3b0 2024-04-26T04:18:52.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.926+0000 7f171ccf6640 1 -- 172.21.15.63:0/4095746348 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f171809b3b0 msgr2=0x7f171809e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:52.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.926+0000 7f171ccf6640 1 --2- 172.21.15.63:0/4095746348 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f171809b3b0 0x7f171809e4e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f171806d8e0 tx=0x7f170801c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c0ab691f686d0cd server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:52.931 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.926+0000 7f17177fe640 1 -- 172.21.15.63:0/4095746348 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17080ad070 con 0x7f171809b3b0 2024-04-26T04:18:52.932 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.926+0000 7f17177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:52.932 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.926+0000 7f17177fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:18:52.932 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.926+0000 7f17177fe640 10 monclient: dump: 2024-04-26T04:18:52.932 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:52.932 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:52.932 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:52.932 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:52.932 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:52.932 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:52.932 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:52.932 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:52.932 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:52.932 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:52.932 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.926+0000 7f17177fe640 10 monclient: _finish_auth 0 2024-04-26T04:18:52.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.926+0000 7f17177fe640 10 monclient: _check_auth_tickets 2024-04-26T04:18:52.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.926+0000 7f17177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:52.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.926+0000 7f17177fe640 1 -- 172.21.15.63:0/4095746348 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17080a9750 con 0x7f171809b3b0 2024-04-26T04:18:52.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.926+0000 7f17177fe640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:52.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.926+0000 7f17177fe640 1 -- 172.21.15.63:0/4095746348 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17080a9b20 con 0x7f171809b3b0 2024-04-26T04:18:52.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.926+0000 7f171ef5a640 5 monclient: authenticate success, global_id 4326 2024-04-26T04:18:52.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.926+0000 7f171ef5a640 10 monclient: get_monmap_and_config success 2024-04-26T04:18:52.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.926+0000 7f171ef5a640 4 set_mon_vals no callback set 2024-04-26T04:18:52.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.926+0000 7f17177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:52.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.926+0000 7f17177fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:18:52.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.926+0000 7f17177fe640 10 monclient: dump: 2024-04-26T04:18:52.933 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:52.933 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:52.934 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:52.934 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:52.934 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:52.934 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:52.934 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:52.934 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:52.934 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:52.934 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:52.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.926+0000 7f171ef5a640 10 monclient: shutdown 2024-04-26T04:18:52.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.926+0000 7f171ef5a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:52.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.926+0000 7f171ef5a640 1 -- 172.21.15.63:0/4095746348 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f171809b3b0 msgr2=0x7f171809e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:52.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.926+0000 7f171ef5a640 1 --2- 172.21.15.63:0/4095746348 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f171809b3b0 0x7f171809e4e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f171806d8e0 tx=0x7f170801c5d0 comp rx=0 tx=0).stop 2024-04-26T04:18:52.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.926+0000 7f171ef5a640 1 -- 172.21.15.63:0/4095746348 shutdown_connections 2024-04-26T04:18:52.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.926+0000 7f171ef5a640 1 --2- 172.21.15.63:0/4095746348 >> v2:172.21.15.186:3300/0 conn(0x7f17180a0bb0 0x7f17180a0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:52.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.926+0000 7f171ef5a640 1 --2- 172.21.15.63:0/4095746348 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f171809b3b0 0x7f171809e4e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:52.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.926+0000 7f171ef5a640 1 -- 172.21.15.63:0/4095746348 >> 172.21.15.63:0/4095746348 conn(0x7f17180a1e70 msgr2=0x7f17180a2280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:52.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.926+0000 7f171ef5a640 1 -- 172.21.15.63:0/4095746348 shutdown_connections 2024-04-26T04:18:52.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.926+0000 7f171ef5a640 1 -- 172.21.15.63:0/4095746348 wait complete. 2024-04-26T04:18:52.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.926+0000 7f171ef5a640 10 monclient: build_initial_monmap 2024-04-26T04:18:52.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.926+0000 7f171ef5a640 10 monclient: monmap: 2024-04-26T04:18:52.938 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:52.938 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:18:52.938 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:18:52.938 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:18:52.939 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:52.939 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:52.939 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:18:52.939 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:18:52.939 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:18:52.939 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:52.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.926+0000 7f171ef5a640 1 Processor -- start 2024-04-26T04:18:52.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.926+0000 7f171ef5a640 1 -- start start 2024-04-26T04:18:52.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.926+0000 7f171ef5a640 10 monclient: init 2024-04-26T04:18:52.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f171ef5a640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:52.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f171ef5a640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T04:18:52.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f171ef5a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f171809b3b0 0x7f171813b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:52.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f171ef5a640 10 monclient(hunting): picked mon.noname-a con 0x7f171809b3b0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:18:52.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f171ef5a640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f17180a0bb0 0x7f17181d2e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:52.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f171ef5a640 10 monclient(hunting): picked mon.noname-c con 0x7f17180a0bb0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:52.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f171ef5a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17180a1500 0x7f171809a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:52.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f171ef5a640 10 monclient(hunting): picked mon.noname-b con 0x7f17180a1500 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:18:52.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f171ef5a640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:52.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f171ef5a640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f171813f490 con 0x7f171809b3b0 2024-04-26T04:18:52.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f171ef5a640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:52.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f171ef5a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f171809eae0 con 0x7f17180a1500 2024-04-26T04:18:52.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f171ef5a640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:52.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f171ef5a640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f171813f610 con 0x7f17180a0bb0 2024-04-26T04:18:52.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f171ef5a640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:52.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f171ef5a640 10 monclient(hunting): authenticate will time out at 964.640356s 2024-04-26T04:18:52.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f171ccf6640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f171809b3b0 0x7f171813b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:52.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f171ccf6640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f171809b3b0 0x7f171813b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:39352/0 (socket says 172.21.15.63:39352) 2024-04-26T04:18:52.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f171ccf6640 1 -- 172.21.15.63:0/3956538527 learned_addr learned my addr 172.21.15.63:0/3956538527 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:18:52.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f171ccf6640 10 monclient(hunting): get_auth_request con 0x7f171809b3b0 auth_method 0 2024-04-26T04:18:52.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f171ccf6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:52.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f171ccf6640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:52.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f171ccf6640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:52.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f1717fff640 1 --2- 172.21.15.63:0/3956538527 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f17180a0bb0 0x7f17181d2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:52.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f1717fff640 10 monclient(hunting): get_auth_request con 0x7f17180a0bb0 auth_method 0 2024-04-26T04:18:52.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f1717fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:52.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f1717fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:52.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f1717fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:52.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f171ccf6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:52.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f171ccf6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:52.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f171ccf6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:52.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f1717fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:52.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f1717fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:52.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f1717fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:52.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f171ccf6640 10 monclient(hunting): handle_auth_done global_id 4329 payload 824 2024-04-26T04:18:52.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f171ccf6640 1 -- 172.21.15.63:0/3956538527 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f17180a0bb0 msgr2=0x7f17181d2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:52.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f171ccf6640 1 --2- 172.21.15.63:0/3956538527 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f17180a0bb0 0x7f17181d2e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:52.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f171ccf6640 1 -- 172.21.15.63:0/3956538527 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17180a1500 msgr2=0x7f171809a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:52.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f171d4f7640 1 --2- 172.21.15.63:0/3956538527 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17180a1500 0x7f171809a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:52.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f171ccf6640 1 --2- 172.21.15.63:0/3956538527 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17180a1500 0x7f171809a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:52.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f171ccf6640 10 monclient: _finish_hunting 0 2024-04-26T04:18:52.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f171ccf6640 1 monclient: found mon.noname-a 2024-04-26T04:18:52.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f171ccf6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:52.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f171ccf6640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:18:52.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f171ccf6640 1 -- 172.21.15.63:0/3956538527 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17181d3d10 con 0x7f171809b3b0 2024-04-26T04:18:52.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f171d4f7640 10 monclient: get_auth_request con 0x7f17180a1500 auth_method 0 2024-04-26T04:18:52.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f1717fff640 1 --2- 172.21.15.63:0/3956538527 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f17180a0bb0 0x7f17181d2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T04:18:52.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f171d4f7640 1 --2- 172.21.15.63:0/3956538527 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17180a1500 0x7f171809a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T04:18:52.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f171ccf6640 1 -- 172.21.15.63:0/3956538527 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f171809b3b0 msgr2=0x7f171813b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:52.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f171ccf6640 1 --2- 172.21.15.63:0/3956538527 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f171809b3b0 0x7f171813b760 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f170801a200 tx=0x7f1708019370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:52.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f17157fa640 1 -- 172.21.15.63:0/3956538527 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17080c0070 con 0x7f171809b3b0 2024-04-26T04:18:52.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f17157fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:52.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f17157fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:18:52.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f17157fa640 10 monclient: dump: 2024-04-26T04:18:52.943 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:52.943 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:52.943 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:52.943 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:52.944 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:52.944 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:52.944 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:52.944 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:52.944 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:52.944 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:52.944 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f17157fa640 10 monclient: _finish_auth 0 2024-04-26T04:18:52.944 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f17157fa640 10 monclient: _check_auth_tickets 2024-04-26T04:18:52.944 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f17157fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:52.944 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f17157fa640 1 -- 172.21.15.63:0/3956538527 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17080b2870 con 0x7f171809b3b0 2024-04-26T04:18:52.944 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f171ef5a640 5 monclient: authenticate success, global_id 4329 2024-04-26T04:18:52.944 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f171ef5a640 10 monclient: _renew_subs 2024-04-26T04:18:52.944 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f171ef5a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:52.944 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f171ef5a640 1 -- 172.21.15.63:0/3956538527 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1718096ae0 con 0x7f171809b3b0 2024-04-26T04:18:52.944 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f17157fa640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:52.945 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f17157fa640 1 -- 172.21.15.63:0/3956538527 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17080ad040 con 0x7f171809b3b0 2024-04-26T04:18:52.945 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f17157fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:52.945 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f17157fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:18:52.945 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f17157fa640 10 monclient: dump: 2024-04-26T04:18:52.945 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:52.945 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:52.945 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:52.945 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:52.945 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:52.945 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:52.945 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:52.945 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:52.945 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:52.945 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:52.945 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f171ef5a640 10 monclient: _renew_subs 2024-04-26T04:18:52.945 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f171ef5a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:52.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f171ef5a640 1 -- 172.21.15.63:0/3956538527 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17181d33c0 con 0x7f171809b3b0 2024-04-26T04:18:52.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.930+0000 7f1716ffd640 4 set_mon_vals no callback set 2024-04-26T04:18:52.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.934+0000 7f171ef5a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:52.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.934+0000 7f171ef5a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:52.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.934+0000 7f171ef5a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:52.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.934+0000 7f171ef5a640 1 -- 172.21.15.63:0/3956538527 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1718062e10 con 0x7f171809b3b0 2024-04-26T04:18:52.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.934+0000 7f17157fa640 1 -- 172.21.15.63:0/3956538527 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f17080b0070 con 0x7f171809b3b0 2024-04-26T04:18:52.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.934+0000 7f17157fa640 1 --2- 172.21.15.63:0/3956538527 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f1704035f10 0x7f17040383d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:52.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.934+0000 7f17157fa640 1 -- 172.21.15.63:0/3956538527 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7f17080e3300 con 0x7f171809b3b0 2024-04-26T04:18:52.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.934+0000 7f1717fff640 1 --2- 172.21.15.63:0/3956538527 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f1704035f10 0x7f17040383d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:52.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.934+0000 7f1717fff640 10 monclient: get_auth_request con 0x7f1704035f10 auth_method 0 2024-04-26T04:18:52.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.934+0000 7f1717fff640 1 --2- 172.21.15.63:0/3956538527 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f1704035f10 0x7f17040383d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f171813be80 tx=0x7f170c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:52.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.938+0000 7f17157fa640 1 -- 172.21.15.63:0/3956538527 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f171809dde0 con 0x7f171809b3b0 2024-04-26T04:18:52.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.938+0000 7f17157fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:52.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:52.938+0000 7f17157fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:18:53.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.094+0000 7f171ef5a640 1 -- 172.21.15.63:0/3956538527 --> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f171813c300 con 0x7f1704035f10 2024-04-26T04:18:53.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.098+0000 7f17157fa640 1 -- 172.21.15.63:0/3956538527 <== mgr.4105 v2:172.21.15.186:6824/2583706639 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33770 (secure 0 0 0) 0x7f171813c300 con 0x7f1704035f10 2024-04-26T04:18:53.103 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-26T04:18:53.103 INFO:teuthology.orchestra.run.smithi063.stderr:dumped all 2024-04-26T04:18:53.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.098+0000 7f171ef5a640 1 -- 172.21.15.63:0/3956538527 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f1704035f10 msgr2=0x7f17040383d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:53.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.098+0000 7f171ef5a640 1 --2- 172.21.15.63:0/3956538527 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f1704035f10 0x7f17040383d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f171813be80 tx=0x7f170c008040 comp rx=0 tx=0).stop 2024-04-26T04:18:53.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.098+0000 7f171ef5a640 10 monclient: shutdown 2024-04-26T04:18:53.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.098+0000 7f171ef5a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:53.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.098+0000 7f171ef5a640 1 -- 172.21.15.63:0/3956538527 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f171809b3b0 msgr2=0x7f171813b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:53.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.098+0000 7f171ef5a640 1 --2- 172.21.15.63:0/3956538527 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f171809b3b0 0x7f171813b760 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f170801a200 tx=0x7f1708019370 comp rx=0 tx=0).stop 2024-04-26T04:18:53.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.098+0000 7f171ef5a640 1 -- 172.21.15.63:0/3956538527 shutdown_connections 2024-04-26T04:18:53.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.098+0000 7f171ef5a640 1 --2- 172.21.15.63:0/3956538527 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f1704035f10 0x7f17040383d0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:53.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.098+0000 7f171ef5a640 1 --2- 172.21.15.63:0/3956538527 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17180a1500 0x7f171809a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:53.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.098+0000 7f171ef5a640 1 --2- 172.21.15.63:0/3956538527 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f17180a0bb0 0x7f17181d2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:53.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.098+0000 7f171ef5a640 1 --2- 172.21.15.63:0/3956538527 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f171809b3b0 0x7f171813b760 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:53.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.098+0000 7f171ef5a640 1 -- 172.21.15.63:0/3956538527 >> 172.21.15.63:0/3956538527 conn(0x7f17180a1e70 msgr2=0x7f1718138540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:53.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.098+0000 7f171ef5a640 1 -- 172.21.15.63:0/3956538527 shutdown_connections 2024-04-26T04:18:53.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.098+0000 7f171ef5a640 1 -- 172.21.15.63:0/3956538527 wait complete. 2024-04-26T04:18:53.119 INFO:teuthology.orchestra.run.smithi063.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2024-04-26T04:18:51.447541+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":164460,"kb_used_data":1880,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566066580,"statfs":{"total":579820584960,"available":579652177920,"internally_reserved":0,"allocated":1925120,"data_stored":1313426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9567,"internal_metadata":165927585},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000724"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-26T04:18:47.497690+0000","last_change":"2024-04-26T04:18:47.131857+0000","last_active":"2024-04-26T04:18:47.497690+0000","last_peered":"2024-04-26T04:18:47.497690+0000","last_clean":"2024-04-26T04:18:47.497690+0000","last_became_active":"2024-04-26T04:18:47.131175+0000","last_became_peered":"2024-04-26T04:18:47.131175+0000","last_unstale":"2024-04-26T04:18:47.497690+0000","last_undegraded":"2024-04-26T04:18:47.497690+0000","last_fullsized":"2024-04-26T04:18:47.497690+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_clean_scrub_stamp":"2024-04-26T04:18:45.482773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:35:03.860574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-26T04:18:47.493565+0000","last_change":"2024-04-26T04:18:46.499700+0000","last_active":"2024-04-26T04:18:47.493565+0000","last_peered":"2024-04-26T04:18:47.493565+0000","last_clean":"2024-04-26T04:18:47.493565+0000","last_became_active":"2024-04-26T04:18:46.498983+0000","last_became_peered":"2024-04-26T04:18:46.498983+0000","last_unstale":"2024-04-26T04:18:47.493565+0000","last_undegraded":"2024-04-26T04:18:47.493565+0000","last_fullsized":"2024-04-26T04:18:47.493565+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_clean_scrub_stamp":"2024-04-26T04:18:45.482773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:51:56.740670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-26T04:18:47.497623+0000","last_change":"2024-04-26T04:18:46.507904+0000","last_active":"2024-04-26T04:18:47.497623+0000","last_peered":"2024-04-26T04:18:47.497623+0000","last_clean":"2024-04-26T04:18:47.497623+0000","last_became_active":"2024-04-26T04:18:46.507335+0000","last_became_peered":"2024-04-26T04:18:46.507335+0000","last_unstale":"2024-04-26T04:18:47.497623+0000","last_undegraded":"2024-04-26T04:18:47.497623+0000","last_fullsized":"2024-04-26T04:18:47.497623+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_clean_scrub_stamp":"2024-04-26T04:18:45.482773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:52:50.029369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-26T04:18:47.493398+0000","last_change":"2024-04-26T04:18:46.506682+0000","last_active":"2024-04-26T04:18:47.493398+0000","last_peered":"2024-04-26T04:18:47.493398+0000","last_clean":"2024-04-26T04:18:47.493398+0000","last_became_active":"2024-04-26T04:18:46.506209+0000","last_became_peered":"2024-04-26T04:18:46.506209+0000","last_unstale":"2024-04-26T04:18:47.493398+0000","last_undegraded":"2024-04-26T04:18:47.493398+0000","last_fullsized":"2024-04-26T04:18:47.493398+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_clean_scrub_stamp":"2024-04-26T04:18:45.482773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:20:00.475307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-26T04:18:47.836289+0000","last_change":"2024-04-26T04:18:46.504685+0000","last_active":"2024-04-26T04:18:47.836289+0000","last_peered":"2024-04-26T04:18:47.836289+0000","last_clean":"2024-04-26T04:18:47.836289+0000","last_became_active":"2024-04-26T04:18:46.504292+0000","last_became_peered":"2024-04-26T04:18:46.504292+0000","last_unstale":"2024-04-26T04:18:47.836289+0000","last_undegraded":"2024-04-26T04:18:47.836289+0000","last_fullsized":"2024-04-26T04:18:47.836289+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_clean_scrub_stamp":"2024-04-26T04:18:45.482773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:42:29.110482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-26T04:18:47.832700+0000","last_change":"2024-04-26T04:18:46.504010+0000","last_active":"2024-04-26T04:18:47.832700+0000","last_peered":"2024-04-26T04:18:47.832700+0000","last_clean":"2024-04-26T04:18:47.832700+0000","last_became_active":"2024-04-26T04:18:46.503343+0000","last_became_peered":"2024-04-26T04:18:46.503343+0000","last_unstale":"2024-04-26T04:18:47.832700+0000","last_undegraded":"2024-04-26T04:18:47.832700+0000","last_fullsized":"2024-04-26T04:18:47.832700+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_clean_scrub_stamp":"2024-04-26T04:18:45.482773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:19:38.324198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-26T04:18:47.497466+0000","last_change":"2024-04-26T04:18:46.503233+0000","last_active":"2024-04-26T04:18:47.497466+0000","last_peered":"2024-04-26T04:18:47.497466+0000","last_clean":"2024-04-26T04:18:47.497466+0000","last_became_active":"2024-04-26T04:18:46.502682+0000","last_became_peered":"2024-04-26T04:18:46.502682+0000","last_unstale":"2024-04-26T04:18:47.497466+0000","last_undegraded":"2024-04-26T04:18:47.497466+0000","last_fullsized":"2024-04-26T04:18:47.497466+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_clean_scrub_stamp":"2024-04-26T04:18:45.482773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:26:15.760818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-26T04:18:47.836255+0000","last_change":"2024-04-26T04:18:46.504844+0000","last_active":"2024-04-26T04:18:47.836255+0000","last_peered":"2024-04-26T04:18:47.836255+0000","last_clean":"2024-04-26T04:18:47.836255+0000","last_became_active":"2024-04-26T04:18:46.504332+0000","last_became_peered":"2024-04-26T04:18:46.504332+0000","last_unstale":"2024-04-26T04:18:47.836255+0000","last_undegraded":"2024-04-26T04:18:47.836255+0000","last_fullsized":"2024-04-26T04:18:47.836255+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_clean_scrub_stamp":"2024-04-26T04:18:45.482773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:13:26.538440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":164,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-04-26T04:18:47.497482+0000","last_change":"2024-04-26T04:18:45.505818+0000","last_active":"2024-04-26T04:18:47.497482+0000","last_peered":"2024-04-26T04:18:47.497482+0000","last_clean":"2024-04-26T04:18:47.497482+0000","last_became_active":"2024-04-26T04:18:45.504887+0000","last_became_peered":"2024-04-26T04:18:45.504887+0000","last_unstale":"2024-04-26T04:18:47.497482+0000","last_undegraded":"2024-04-26T04:18:47.497482+0000","last_fullsized":"2024-04-26T04:18:47.497482+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:44.470061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:44.470061+0000","last_clean_scrub_stamp":"2024-04-26T04:18:44.470061+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:42:56.629937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":12,"seq":51539607558,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27680,"kb_used_data":660,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344160,"statfs":{"total":96636764160,"available":96608419840,"internally_reserved":0,"allocated":675840,"data_stored":571523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":12,"seq":51539607557,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":143360,"data_stored":42595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607557,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":143360,"data_stored":42595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":143360,"data_stored":42595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640260,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":143360,"data_stored":42595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28128,"kb_used_data":660,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343712,"statfs":{"total":96636764160,"available":96607961088,"internally_reserved":0,"allocated":675840,"data_stored":571523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T04:18:53.120 INFO:tasks.mon_thrash.ceph_manager:clean! 2024-04-26T04:18:53.120 DEBUG:tasks.mon_thrash:Yielding 2024-04-26T04:18:53.121 INFO:teuthology.run_tasks:Running task exec... 2024-04-26T04:18:53.130 INFO:teuthology.task.exec:Executing custom commands... 2024-04-26T04:18:53.131 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi186.front.sepia.ceph.com 2024-04-26T04:18:53.131 DEBUG:teuthology.orchestra.run.smithi186:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-26T04:18:53.131 INFO:tasks.mon_thrash.mon_thrasher:start thrashing 2024-04-26T04:18:53.131 INFO:tasks.mon_thrash.mon_thrasher:seed: 1714105133, revive delay: 90.0, thrash delay: 1.0 thrash many: True, maintain quorum: True store thrash: False, probability: 50 freeze mon: prob 10.0 duration 15.0 2024-04-26T04:18:53.131 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-26T04:18:53.131 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T04:18:53.250 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.246+0000 7f347b0e8640 10 monclient: get_monmap_and_config 2024-04-26T04:18:53.250 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.246+0000 7f347b0e8640 10 monclient: build_initial_monmap 2024-04-26T04:18:53.250 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.246+0000 7f347b0e8640 10 monclient: monmap: 2024-04-26T04:18:53.250 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:53.250 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:53.250 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:53.250075+0000 2024-04-26T04:18:53.250 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:53.250075+0000 2024-04-26T04:18:53.250 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:53.250 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:53.251 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:18:53.251 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:18:53.251 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:18:53.251 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:18:53.251 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:18:53.251 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:53.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.246+0000 7f347b0e8640 1 Processor -- start 2024-04-26T04:18:53.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.246+0000 7f347b0e8640 1 -- start start 2024-04-26T04:18:53.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.246+0000 7f347b0e8640 10 monclient: init 2024-04-26T04:18:53.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.246+0000 7f347b0e8640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:53.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.246+0000 7f347b0e8640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-04-26T04:18:53.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.246+0000 7f347b0e8640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f3474072490 0x7f3474072870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:53.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.246+0000 7f347b0e8640 10 monclient(hunting): picked mon.noname-c con 0x7f3474072490 addr v2:172.21.15.186:3300/0 2024-04-26T04:18:53.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.246+0000 7f347b0e8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3474133be0 addr v1:172.21.15.63:6789/0 2024-04-26T04:18:53.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.246+0000 7f347b0e8640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f34740a1bb0 0x7f347413efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:53.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.246+0000 7f347b0e8640 10 monclient(hunting): picked mon.noname-a con 0x7f34740a1bb0 addr v2:172.21.15.63:3300/0 2024-04-26T04:18:53.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.246+0000 7f347b0e8640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f347406e750 con 0x7f3474133be0 2024-04-26T04:18:53.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.246+0000 7f347b0e8640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:53.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.246+0000 7f347b0e8640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7f347409f4c0 con 0x7f34740a1bb0 2024-04-26T04:18:53.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.246+0000 7f347b0e8640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:53.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.246+0000 7f347b0e8640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f3474072db0 con 0x7f3474072490 2024-04-26T04:18:53.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.246+0000 7f347b0e8640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:53.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.246+0000 7f347b0e8640 10 monclient(hunting): authenticate will time out at 964.957893s 2024-04-26T04:18:53.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.246+0000 7f3479685640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f34740a1bb0 0x7f347413efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:53.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.246+0000 7f3479685640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f34740a1bb0 0x7f347413efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:39368/0 (socket says 172.21.15.63:39368) 2024-04-26T04:18:53.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.246+0000 7f3479685640 1 -- 172.21.15.63:0/1595660430 learned_addr learned my addr 172.21.15.63:0/1595660430 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:18:53.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.246+0000 7f3479685640 10 monclient(hunting): get_auth_request con 0x7f34740a1bb0 auth_method 0 2024-04-26T04:18:53.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.246+0000 7f3479685640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:53.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.246+0000 7f3479685640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:53.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.246+0000 7f3479685640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:53.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.246+0000 7f3473fff640 1 --1- 172.21.15.63:0/1595660430 >> v1:172.21.15.63:6789/0 conn(0x7f3474133be0 0x7f34740a14a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:32820/0 (socket says 172.21.15.63:32820) 2024-04-26T04:18:53.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.246+0000 7f3479685640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:53.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.246+0000 7f3479685640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:53.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.246+0000 7f3479685640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:53.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.246+0000 7f3478e84640 1 --2- 172.21.15.63:0/1595660430 >> v2:172.21.15.186:3300/0 conn(0x7f3474072490 0x7f3474072870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:53.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.246+0000 7f3478e84640 10 monclient(hunting): get_auth_request con 0x7f3474072490 auth_method 0 2024-04-26T04:18:53.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.246+0000 7f3478e84640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:53.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.246+0000 7f3478e84640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:53.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.246+0000 7f3478e84640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:53.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.246+0000 7f3479685640 10 monclient(hunting): handle_auth_done global_id 4341 payload 274 2024-04-26T04:18:53.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.246+0000 7f3479685640 1 -- 172.21.15.63:0/1595660430 >> v2:172.21.15.186:3300/0 conn(0x7f3474072490 msgr2=0x7f3474072870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:53.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.246+0000 7f3479685640 1 --2- 172.21.15.63:0/1595660430 >> v2:172.21.15.186:3300/0 conn(0x7f3474072490 0x7f3474072870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:53.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.246+0000 7f3479685640 1 -- 172.21.15.63:0/1595660430 >> v1:172.21.15.63:6789/0 conn(0x7f3474133be0 legacy=0x7f34740a14a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:53.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.246+0000 7f3479685640 10 monclient: _finish_hunting 0 2024-04-26T04:18:53.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.246+0000 7f3479685640 1 monclient: found mon.noname-a 2024-04-26T04:18:53.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.246+0000 7f3479685640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:53.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.246+0000 7f3479685640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:18:53.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.246+0000 7f3479685640 1 -- 172.21.15.63:0/1595660430 --> v2:172.21.15.63:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f347409f930 con 0x7f34740a1bb0 2024-04-26T04:18:53.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.250+0000 7f3479685640 1 -- 172.21.15.63:0/1595660430 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f34740a1bb0 msgr2=0x7f347413efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:53.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.250+0000 7f3479685640 1 --2- 172.21.15.63:0/1595660430 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f34740a1bb0 0x7f347413efe0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f347409ef70 tx=0x7f346c01c6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30ce8f1f4ef332f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:53.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.250+0000 7f34737fe640 1 -- 172.21.15.63:0/1595660430 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f346c0a55b0 con 0x7f34740a1bb0 2024-04-26T04:18:53.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.250+0000 7f34737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:53.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.250+0000 7f34737fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:18:53.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.250+0000 7f34737fe640 10 monclient: dump: 2024-04-26T04:18:53.260 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:53.261 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:53.261 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:53.261 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:53.261 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:53.261 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:53.261 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:53.261 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:53.261 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:53.261 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:53.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.250+0000 7f34737fe640 10 monclient: _finish_auth 0 2024-04-26T04:18:53.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.250+0000 7f34737fe640 10 monclient: _check_auth_tickets 2024-04-26T04:18:53.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.250+0000 7f34737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:53.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.250+0000 7f347b0e8640 5 monclient: authenticate success, global_id 4341 2024-04-26T04:18:53.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.250+0000 7f34737fe640 1 -- 172.21.15.63:0/1595660430 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f346c0a5750 con 0x7f34740a1bb0 2024-04-26T04:18:53.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.250+0000 7f34737fe640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:53.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.250+0000 7f347b0e8640 10 monclient: get_monmap_and_config success 2024-04-26T04:18:53.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.250+0000 7f34737fe640 1 -- 172.21.15.63:0/1595660430 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f346c0a5b20 con 0x7f34740a1bb0 2024-04-26T04:18:53.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.250+0000 7f34737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:53.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.250+0000 7f347b0e8640 4 set_mon_vals no callback set 2024-04-26T04:18:53.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.250+0000 7f34737fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:18:53.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.250+0000 7f34737fe640 10 monclient: dump: 2024-04-26T04:18:53.262 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:53.262 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:53.262 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:53.262 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:53.262 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:53.262 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:53.262 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:53.262 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:53.263 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:53.263 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:53.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.250+0000 7f347b0e8640 10 monclient: shutdown 2024-04-26T04:18:53.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.250+0000 7f347b0e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:53.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.250+0000 7f347b0e8640 1 -- 172.21.15.63:0/1595660430 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f34740a1bb0 msgr2=0x7f347413efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:53.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.250+0000 7f347b0e8640 1 --2- 172.21.15.63:0/1595660430 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f34740a1bb0 0x7f347413efe0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f347409ef70 tx=0x7f346c01c6f0 comp rx=0 tx=0).stop 2024-04-26T04:18:53.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.250+0000 7f347b0e8640 1 -- 172.21.15.63:0/1595660430 shutdown_connections 2024-04-26T04:18:53.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.250+0000 7f347b0e8640 1 --2- 172.21.15.63:0/1595660430 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f34740a1bb0 0x7f347413efe0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:53.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.250+0000 7f347b0e8640 1 --2- 172.21.15.63:0/1595660430 >> v2:172.21.15.186:3300/0 conn(0x7f3474072490 0x7f3474072870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:53.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.250+0000 7f347b0e8640 1 -- 172.21.15.63:0/1595660430 >> 172.21.15.63:0/1595660430 conn(0x7f34740a6170 msgr2=0x7f34740a2590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:53.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.250+0000 7f347b0e8640 1 -- 172.21.15.63:0/1595660430 shutdown_connections 2024-04-26T04:18:53.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.250+0000 7f347b0e8640 1 -- 172.21.15.63:0/1595660430 wait complete. 2024-04-26T04:18:53.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.254+0000 7f347b0e8640 10 monclient: build_initial_monmap 2024-04-26T04:18:53.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.254+0000 7f347b0e8640 10 monclient: monmap: 2024-04-26T04:18:53.264 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:53.264 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:18:53.264 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:18:53.264 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:18:53.264 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:53.264 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:53.264 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:18:53.264 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:18:53.264 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:18:53.264 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:53.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.254+0000 7f347b0e8640 1 Processor -- start 2024-04-26T04:18:53.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.254+0000 7f347b0e8640 1 -- start start 2024-04-26T04:18:53.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.254+0000 7f347b0e8640 10 monclient: init 2024-04-26T04:18:53.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.254+0000 7f347b0e8640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:53.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.254+0000 7f347b0e8640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T04:18:53.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.254+0000 7f347b0e8640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3474072490 0x7f3474130600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:53.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.254+0000 7f347b0e8640 10 monclient(hunting): picked mon.noname-b con 0x7f3474072490 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:18:53.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.254+0000 7f347b0e8640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f34740a1bb0 0x7f3474130b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:53.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.254+0000 7f347b0e8640 10 monclient(hunting): picked mon.noname-c con 0x7f34740a1bb0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:53.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.254+0000 7f347b0e8640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3474133be0 0x7f34741cede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:53.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.254+0000 7f347b0e8640 10 monclient(hunting): picked mon.noname-a con 0x7f3474133be0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:18:53.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.254+0000 7f347b0e8640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:53.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.254+0000 7f347b0e8640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f347406c1c0 con 0x7f3474133be0 2024-04-26T04:18:53.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.254+0000 7f347b0e8640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:53.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.254+0000 7f347b0e8640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f347406c040 con 0x7f3474072490 2024-04-26T04:18:53.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.254+0000 7f347b0e8640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:53.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.254+0000 7f347b0e8640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f3474141d40 con 0x7f34740a1bb0 2024-04-26T04:18:53.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.254+0000 7f347b0e8640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:53.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.254+0000 7f347b0e8640 10 monclient(hunting): authenticate will time out at 964.965093s 2024-04-26T04:18:53.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.254+0000 7f3479685640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3474133be0 0x7f34741cede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:53.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.254+0000 7f3473fff640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f34740a1bb0 0x7f3474130b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:53.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.254+0000 7f3479685640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3474133be0 0x7f34741cede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:39374/0 (socket says 172.21.15.63:39374) 2024-04-26T04:18:53.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.254+0000 7f3479685640 1 -- 172.21.15.63:0/2222168856 learned_addr learned my addr 172.21.15.63:0/2222168856 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:18:53.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.254+0000 7f3479685640 10 monclient(hunting): get_auth_request con 0x7f3474133be0 auth_method 0 2024-04-26T04:18:53.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.254+0000 7f3473fff640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f34740a1bb0 0x7f3474130b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:36974/0 (socket says 172.21.15.63:36974) 2024-04-26T04:18:53.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.254+0000 7f3479685640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:53.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.254+0000 7f3479685640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:53.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.254+0000 7f3479685640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:53.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.254+0000 7f3473fff640 10 monclient(hunting): get_auth_request con 0x7f34740a1bb0 auth_method 0 2024-04-26T04:18:53.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.254+0000 7f3473fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:53.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.254+0000 7f3473fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:53.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.254+0000 7f3473fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:53.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.254+0000 7f3479685640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:53.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.254+0000 7f3479685640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:53.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.254+0000 7f3479685640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:53.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.254+0000 7f3473fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:53.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.254+0000 7f3473fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:53.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.254+0000 7f3473fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:53.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.254+0000 7f3478e84640 1 --2- 172.21.15.63:0/2222168856 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3474072490 0x7f3474130600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:53.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.254+0000 7f3479685640 10 monclient(hunting): handle_auth_done global_id 4344 payload 824 2024-04-26T04:18:53.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.254+0000 7f3479685640 1 -- 172.21.15.63:0/2222168856 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f34740a1bb0 msgr2=0x7f3474130b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:53.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.254+0000 7f3479685640 1 --2- 172.21.15.63:0/2222168856 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f34740a1bb0 0x7f3474130b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:53.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.254+0000 7f3479685640 1 -- 172.21.15.63:0/2222168856 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3474072490 msgr2=0x7f3474130600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:53.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.254+0000 7f3479685640 1 --2- 172.21.15.63:0/2222168856 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3474072490 0x7f3474130600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:53.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.254+0000 7f3479685640 10 monclient: _finish_hunting 0 2024-04-26T04:18:53.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.254+0000 7f3479685640 1 monclient: found mon.noname-a 2024-04-26T04:18:53.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.254+0000 7f3479685640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:53.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.254+0000 7f3479685640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:18:53.268 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.254+0000 7f3479685640 1 -- 172.21.15.63:0/2222168856 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f347413b4e0 con 0x7f3474133be0 2024-04-26T04:18:53.268 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.254+0000 7f3478e84640 10 monclient: get_auth_request con 0x7f3474072490 auth_method 0 2024-04-26T04:18:53.268 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.254+0000 7f3478e84640 1 --2- 172.21.15.63:0/2222168856 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3474072490 0x7f3474130600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T04:18:53.270 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.254+0000 7f3479685640 1 -- 172.21.15.63:0/2222168856 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3474133be0 msgr2=0x7f34741cede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:53.270 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.254+0000 7f3479685640 1 --2- 172.21.15.63:0/2222168856 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3474133be0 0x7f34741cede0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f346c01a320 tx=0x7f346c019490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:53.270 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.258+0000 7f34717fa640 1 -- 172.21.15.63:0/2222168856 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f346c0bc020 con 0x7f3474133be0 2024-04-26T04:18:53.270 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.258+0000 7f34717fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:53.270 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.258+0000 7f34717fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:18:53.270 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.258+0000 7f34717fa640 10 monclient: dump: 2024-04-26T04:18:53.270 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:53.270 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:53.270 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:53.270 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:53.270 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:53.270 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:53.270 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:53.271 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:53.271 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:53.271 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:53.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.258+0000 7f34717fa640 10 monclient: _finish_auth 0 2024-04-26T04:18:53.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.258+0000 7f34717fa640 10 monclient: _check_auth_tickets 2024-04-26T04:18:53.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.258+0000 7f34717fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:53.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.258+0000 7f347b0e8640 5 monclient: authenticate success, global_id 4344 2024-04-26T04:18:53.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.258+0000 7f347b0e8640 10 monclient: _renew_subs 2024-04-26T04:18:53.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.258+0000 7f347b0e8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:53.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.258+0000 7f347b0e8640 1 -- 172.21.15.63:0/2222168856 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f347413f210 con 0x7f3474133be0 2024-04-26T04:18:53.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.258+0000 7f34717fa640 1 -- 172.21.15.63:0/2222168856 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f346c0b8880 con 0x7f3474133be0 2024-04-26T04:18:53.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.258+0000 7f34717fa640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:53.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.258+0000 7f34717fa640 1 -- 172.21.15.63:0/2222168856 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f346c0a55b0 con 0x7f3474133be0 2024-04-26T04:18:53.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.258+0000 7f34717fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:53.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.258+0000 7f34717fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:18:53.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.258+0000 7f34717fa640 10 monclient: dump: 2024-04-26T04:18:53.272 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:53.272 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:53.272 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:53.272 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:53.272 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:53.272 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:53.272 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:53.272 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:53.273 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:53.273 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:53.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.258+0000 7f347b0e8640 10 monclient: _renew_subs 2024-04-26T04:18:53.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.258+0000 7f347b0e8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:53.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.258+0000 7f347b0e8640 1 -- 172.21.15.63:0/2222168856 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3474131080 con 0x7f3474133be0 2024-04-26T04:18:53.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.258+0000 7f3472ffd640 4 set_mon_vals no callback set 2024-04-26T04:18:53.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.258+0000 7f347b0e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:53.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.258+0000 7f347b0e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:53.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.258+0000 7f347b0e8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:53.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.258+0000 7f34717fa640 1 -- 172.21.15.63:0/2222168856 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f346c0bf020 con 0x7f3474133be0 2024-04-26T04:18:53.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.258+0000 7f347b0e8640 1 -- 172.21.15.63:0/2222168856 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3474062e10 con 0x7f3474133be0 2024-04-26T04:18:53.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.258+0000 7f34717fa640 1 --2- 172.21.15.63:0/2222168856 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f3460035ec0 0x7f3460038380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:53.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.258+0000 7f34717fa640 1 -- 172.21.15.63:0/2222168856 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 4346+0+0 (secure 0 0 0) 0x7f346c0e3b40 con 0x7f3474133be0 2024-04-26T04:18:53.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.258+0000 7f3478e84640 1 --2- 172.21.15.63:0/2222168856 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f3460035ec0 0x7f3460038380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:53.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.258+0000 7f3478e84640 10 monclient: get_auth_request con 0x7f3460035ec0 auth_method 0 2024-04-26T04:18:53.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.262+0000 7f3478e84640 1 --2- 172.21.15.63:0/2222168856 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f3460035ec0 0x7f3460038380 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f34680097a0 tx=0x7f3468008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:53.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.262+0000 7f34717fa640 1 -- 172.21.15.63:0/2222168856 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f346c0c6030 con 0x7f3474133be0 2024-04-26T04:18:53.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.262+0000 7f34717fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:53.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.262+0000 7f34717fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:18:53.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.474+0000 7f347b0e8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T04:18:53.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.474+0000 7f347b0e8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T04:18:53.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.474+0000 7f347b0e8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:53.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.474+0000 7f347b0e8640 1 -- 172.21.15.63:0/2222168856 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f347413e770 con 0x7f3474133be0 2024-04-26T04:18:53.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.474+0000 7f34717fa640 1 -- 172.21.15.63:0/2222168856 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f347406e750 con 0x7f3474133be0 2024-04-26T04:18:53.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.474+0000 7f34717fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T04:18:53.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.474+0000 7f34717fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:18:53.481 INFO:teuthology.orchestra.run.smithi063.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":19,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cd1c6a0c-4e4b-4014-89c0-2e535479ef39","modified":"2024-04-26T04:18:03.702339Z","created":"2024-04-26T04:18:03.702339Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3300","nonce":0},{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:3300","nonce":0},{"type":"v1","addr":"172.21.15.186:6789","nonce":0}]},"addr":"172.21.15.186:6789/0","public_addr":"172.21.15.186:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3301","nonce":0},{"type":"v1","addr":"172.21.15.63:6790","nonce":0}]},"addr":"172.21.15.63:6790/0","public_addr":"172.21.15.63:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T04:18:53.483 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.478+0000 7f347b0e8640 1 -- 172.21.15.63:0/2222168856 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f3460035ec0 msgr2=0x7f3460038380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:53.483 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.478+0000 7f347b0e8640 1 --2- 172.21.15.63:0/2222168856 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f3460035ec0 0x7f3460038380 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f34680097a0 tx=0x7f3468008040 comp rx=0 tx=0).stop 2024-04-26T04:18:53.483 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.478+0000 7f347b0e8640 10 monclient: shutdown 2024-04-26T04:18:53.483 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.478+0000 7f347b0e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:53.483 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.478+0000 7f347b0e8640 1 -- 172.21.15.63:0/2222168856 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3474133be0 msgr2=0x7f34741cede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:53.483 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.478+0000 7f347b0e8640 1 --2- 172.21.15.63:0/2222168856 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3474133be0 0x7f34741cede0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f346c01a320 tx=0x7f346c019490 comp rx=0 tx=0).stop 2024-04-26T04:18:53.483 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.478+0000 7f347b0e8640 1 -- 172.21.15.63:0/2222168856 shutdown_connections 2024-04-26T04:18:53.483 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.478+0000 7f347b0e8640 1 --2- 172.21.15.63:0/2222168856 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f3460035ec0 0x7f3460038380 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:53.483 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.478+0000 7f347b0e8640 1 --2- 172.21.15.63:0/2222168856 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3474133be0 0x7f34741cede0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:53.483 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.478+0000 7f347b0e8640 1 --2- 172.21.15.63:0/2222168856 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f34740a1bb0 0x7f3474130b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:53.483 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.478+0000 7f347b0e8640 1 --2- 172.21.15.63:0/2222168856 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3474072490 0x7f3474130600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:53.483 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.478+0000 7f347b0e8640 1 -- 172.21.15.63:0/2222168856 >> 172.21.15.63:0/2222168856 conn(0x7f34740a6170 msgr2=0x7f3474131620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:53.483 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.478+0000 7f347b0e8640 1 -- 172.21.15.63:0/2222168856 shutdown_connections 2024-04-26T04:18:53.483 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.478+0000 7f347b0e8640 1 -- 172.21.15.63:0/2222168856 wait complete. 2024-04-26T04:18:53.498 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-26T04:18:53.498 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-04-26T04:18:53.498 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-26T04:18:53.498 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T04:18:53.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.618+0000 7f8d3eed5640 10 monclient: get_monmap_and_config 2024-04-26T04:18:53.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.618+0000 7f8d3eed5640 10 monclient: build_initial_monmap 2024-04-26T04:18:53.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.618+0000 7f8d3eed5640 10 monclient: monmap: 2024-04-26T04:18:53.628 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:53.628 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:53.629 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:53.624655+0000 2024-04-26T04:18:53.629 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:53.624655+0000 2024-04-26T04:18:53.629 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:53.629 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:53.629 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:18:53.629 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:18:53.629 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:18:53.629 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:18:53.629 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:18:53.629 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:53.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.618+0000 7f8d3eed5640 1 Processor -- start 2024-04-26T04:18:53.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.618+0000 7f8d3eed5640 1 -- start start 2024-04-26T04:18:53.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.618+0000 7f8d3eed5640 10 monclient: init 2024-04-26T04:18:53.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.622+0000 7f8d3eed5640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:53.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.622+0000 7f8d3eed5640 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-04-26T04:18:53.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.622+0000 7f8d3eed5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8d380a2cd0 addr v1:172.21.15.186:6789/0 2024-04-26T04:18:53.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.622+0000 7f8d3eed5640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f8d380a30b0 0x7f8d3813c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:53.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.622+0000 7f8d3eed5640 10 monclient(hunting): picked mon.noname-b con 0x7f8d380a30b0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:53.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.622+0000 7f8d3eed5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8d3813c540 addr v1:172.21.15.63:6789/0 2024-04-26T04:18:53.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.622+0000 7f8d3eed5640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d380a4ee0 con 0x7f8d3813c540 2024-04-26T04:18:53.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.622+0000 7f8d3eed5640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d380a0fb0 con 0x7f8d380a2cd0 2024-04-26T04:18:53.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.622+0000 7f8d3eed5640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:53.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.622+0000 7f8d3eed5640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f8d380a5380 con 0x7f8d380a30b0 2024-04-26T04:18:53.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.622+0000 7f8d3eed5640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:53.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.622+0000 7f8d3eed5640 10 monclient(hunting): authenticate will time out at 965.332787s 2024-04-26T04:18:53.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.622+0000 7f8d37fff640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f8d380a30b0 0x7f8d3813c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:53.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.622+0000 7f8d3d472640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f8d3813c540 0x7f8d3813e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:32824/0 (socket says 172.21.15.63:32824) 2024-04-26T04:18:53.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.622+0000 7f8d3d472640 1 -- 172.21.15.63:0/314698232 learned_addr learned my addr 172.21.15.63:0/314698232 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-04-26T04:18:53.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.622+0000 7f8d37fff640 10 monclient(hunting): get_auth_request con 0x7f8d380a30b0 auth_method 0 2024-04-26T04:18:53.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.622+0000 7f8d37fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:53.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.622+0000 7f8d37fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:53.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.622+0000 7f8d37fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:53.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.622+0000 7f8d37fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:53.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.622+0000 7f8d37fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:53.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.622+0000 7f8d37fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:53.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.622+0000 7f8d37fff640 10 monclient(hunting): handle_auth_done global_id 4322 payload 274 2024-04-26T04:18:53.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.622+0000 7f8d37fff640 1 -- 172.21.15.63:0/314698232 >> v1:172.21.15.186:6789/0 conn(0x7f8d380a2cd0 legacy=0x7f8d380a47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:53.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.622+0000 7f8d37fff640 1 -- 172.21.15.63:0/314698232 >> v1:172.21.15.63:6789/0 conn(0x7f8d3813c540 legacy=0x7f8d3813e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:53.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.622+0000 7f8d37fff640 10 monclient: _finish_hunting 0 2024-04-26T04:18:53.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.622+0000 7f8d37fff640 1 monclient: found mon.noname-b 2024-04-26T04:18:53.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.622+0000 7f8d37fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:53.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.622+0000 7f8d37fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.63:3301/0 2024-04-26T04:18:53.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.622+0000 7f8d37fff640 1 -- 172.21.15.63:0/314698232 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d380a1290 con 0x7f8d380a30b0 2024-04-26T04:18:53.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.626+0000 7f8d37fff640 1 -- 172.21.15.63:0/314698232 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f8d380a30b0 msgr2=0x7f8d3813c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:53.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.626+0000 7f8d37fff640 1 --2- 172.21.15.63:0/314698232 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f8d380a30b0 0x7f8d3813c000 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c0097f0 tx=0x7f8d2c01c3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4726479527b3c30 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:53.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.626+0000 7f8d377fe640 1 -- 172.21.15.63:0/314698232 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d2c0b0070 con 0x7f8d380a30b0 2024-04-26T04:18:53.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.626+0000 7f8d377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:53.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.626+0000 7f8d377fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T04:18:53.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.626+0000 7f8d377fe640 10 monclient: dump: 2024-04-26T04:18:53.636 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:53.636 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:53.636 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:53.636 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:53.636 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:53.636 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:53.636 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:53.636 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:53.636 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:53.636 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:53.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.626+0000 7f8d377fe640 10 monclient: _finish_auth 0 2024-04-26T04:18:53.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.626+0000 7f8d377fe640 10 monclient: _check_auth_tickets 2024-04-26T04:18:53.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.626+0000 7f8d377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:53.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.626+0000 7f8d377fe640 1 -- 172.21.15.63:0/314698232 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d2c0ab600 con 0x7f8d380a30b0 2024-04-26T04:18:53.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.626+0000 7f8d377fe640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:53.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.626+0000 7f8d377fe640 1 -- 172.21.15.63:0/314698232 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d2c0ab9d0 con 0x7f8d380a30b0 2024-04-26T04:18:53.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.626+0000 7f8d377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:53.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.626+0000 7f8d377fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:18:53.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.626+0000 7f8d377fe640 10 monclient: dump: 2024-04-26T04:18:53.637 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:53.637 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:53.637 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:53.637 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:53.637 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:53.637 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:53.637 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:53.637 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:53.638 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:53.638 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:53.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.626+0000 7f8d3eed5640 5 monclient: authenticate success, global_id 4322 2024-04-26T04:18:53.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.626+0000 7f8d3eed5640 10 monclient: get_monmap_and_config success 2024-04-26T04:18:53.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.626+0000 7f8d3eed5640 4 set_mon_vals no callback set 2024-04-26T04:18:53.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.626+0000 7f8d3eed5640 10 monclient: shutdown 2024-04-26T04:18:53.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.626+0000 7f8d3eed5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:53.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.626+0000 7f8d3eed5640 1 -- 172.21.15.63:0/314698232 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f8d380a30b0 msgr2=0x7f8d3813c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:53.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.626+0000 7f8d3eed5640 1 --2- 172.21.15.63:0/314698232 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f8d380a30b0 0x7f8d3813c000 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c0097f0 tx=0x7f8d2c01c3b0 comp rx=0 tx=0).stop 2024-04-26T04:18:53.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.626+0000 7f8d3eed5640 1 -- 172.21.15.63:0/314698232 shutdown_connections 2024-04-26T04:18:53.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.626+0000 7f8d3eed5640 1 --2- 172.21.15.63:0/314698232 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f8d380a30b0 0x7f8d3813c000 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:53.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.626+0000 7f8d3eed5640 1 -- 172.21.15.63:0/314698232 >> 172.21.15.63:0/314698232 conn(0x7f8d380a5f40 msgr2=0x7f8d380a6320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:53.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.626+0000 7f8d3eed5640 1 -- 172.21.15.63:0/314698232 shutdown_connections 2024-04-26T04:18:53.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.626+0000 7f8d3eed5640 1 -- 172.21.15.63:0/314698232 wait complete. 2024-04-26T04:18:53.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.626+0000 7f8d3eed5640 10 monclient: build_initial_monmap 2024-04-26T04:18:53.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.626+0000 7f8d3eed5640 10 monclient: monmap: 2024-04-26T04:18:53.639 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:53.639 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:18:53.639 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:18:53.639 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:18:53.639 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:53.639 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:53.639 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:18:53.639 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:18:53.639 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:18:53.639 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:53.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.626+0000 7f8d3eed5640 1 Processor -- start 2024-04-26T04:18:53.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.626+0000 7f8d3eed5640 1 -- start start 2024-04-26T04:18:53.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.626+0000 7f8d3eed5640 10 monclient: init 2024-04-26T04:18:53.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.626+0000 7f8d3eed5640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:53.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.626+0000 7f8d3eed5640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T04:18:53.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.626+0000 7f8d3eed5640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d380a2cd0 0x7f8d381cf590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:53.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.626+0000 7f8d3eed5640 10 monclient(hunting): picked mon.noname-a con 0x7f8d380a2cd0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:18:53.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.626+0000 7f8d3eed5640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d380a30b0 0x7f8d3806ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:53.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.626+0000 7f8d3eed5640 10 monclient(hunting): picked mon.noname-b con 0x7f8d380a30b0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:18:53.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.626+0000 7f8d3eed5640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f8d3813c540 0x7f8d3806f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:53.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.626+0000 7f8d3eed5640 10 monclient(hunting): picked mon.noname-c con 0x7f8d3813c540 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:53.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.626+0000 7f8d3eed5640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:53.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.626+0000 7f8d3eed5640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f8d38142c00 con 0x7f8d380a2cd0 2024-04-26T04:18:53.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.626+0000 7f8d3eed5640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:53.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.626+0000 7f8d3eed5640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8d38142a80 con 0x7f8d380a30b0 2024-04-26T04:18:53.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.626+0000 7f8d3eed5640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:53.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.626+0000 7f8d3eed5640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f8d38142d80 con 0x7f8d3813c540 2024-04-26T04:18:53.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.626+0000 7f8d3eed5640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:53.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.626+0000 7f8d3eed5640 10 monclient(hunting): authenticate will time out at 965.339893s 2024-04-26T04:18:53.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.626+0000 7f8d3cc71640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d380a2cd0 0x7f8d381cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:53.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.626+0000 7f8d3cc71640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d380a2cd0 0x7f8d381cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:39378/0 (socket says 172.21.15.63:39378) 2024-04-26T04:18:53.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.630+0000 7f8d3cc71640 1 -- 172.21.15.63:0/347244099 learned_addr learned my addr 172.21.15.63:0/347244099 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:18:53.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.630+0000 7f8d3cc71640 10 monclient(hunting): get_auth_request con 0x7f8d380a2cd0 auth_method 0 2024-04-26T04:18:53.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.630+0000 7f8d3cc71640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:53.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.630+0000 7f8d3cc71640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:53.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.630+0000 7f8d3cc71640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:53.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.630+0000 7f8d3d472640 1 --2- 172.21.15.63:0/347244099 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f8d3813c540 0x7f8d3806f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:53.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.630+0000 7f8d3d472640 10 monclient(hunting): get_auth_request con 0x7f8d3813c540 auth_method 0 2024-04-26T04:18:53.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.630+0000 7f8d3d472640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:53.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.630+0000 7f8d3d472640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:53.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.630+0000 7f8d3d472640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:53.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.630+0000 7f8d3cc71640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:53.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.630+0000 7f8d3cc71640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:53.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.630+0000 7f8d3cc71640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:53.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.630+0000 7f8d3d472640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:53.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.630+0000 7f8d3d472640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:53.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.630+0000 7f8d3d472640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:53.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.630+0000 7f8d3cc71640 10 monclient(hunting): handle_auth_done global_id 4347 payload 824 2024-04-26T04:18:53.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.630+0000 7f8d3cc71640 1 -- 172.21.15.63:0/347244099 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f8d3813c540 msgr2=0x7f8d3806f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:53.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.630+0000 7f8d3cc71640 1 --2- 172.21.15.63:0/347244099 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f8d3813c540 0x7f8d3806f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:53.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.630+0000 7f8d3cc71640 1 -- 172.21.15.63:0/347244099 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d380a30b0 msgr2=0x7f8d3806ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:53.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.630+0000 7f8d37fff640 1 --2- 172.21.15.63:0/347244099 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d380a30b0 0x7f8d3806ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:53.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.630+0000 7f8d3cc71640 1 --2- 172.21.15.63:0/347244099 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d380a30b0 0x7f8d3806ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:53.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.630+0000 7f8d3cc71640 10 monclient: _finish_hunting 0 2024-04-26T04:18:53.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.630+0000 7f8d3cc71640 1 monclient: found mon.noname-a 2024-04-26T04:18:53.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.630+0000 7f8d3cc71640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:53.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.630+0000 7f8d3cc71640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:18:53.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.630+0000 7f8d3cc71640 1 -- 172.21.15.63:0/347244099 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d380a37d0 con 0x7f8d380a2cd0 2024-04-26T04:18:53.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.630+0000 7f8d3cc71640 1 -- 172.21.15.63:0/347244099 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d380a2cd0 msgr2=0x7f8d381cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:53.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.630+0000 7f8d3cc71640 1 --2- 172.21.15.63:0/347244099 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d380a2cd0 0x7f8d381cf590 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8d2800c7f0 tx=0x7f8d2800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:53.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.630+0000 7f8d357fa640 1 -- 172.21.15.63:0/347244099 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d28016020 con 0x7f8d380a2cd0 2024-04-26T04:18:53.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.630+0000 7f8d357fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:53.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.630+0000 7f8d357fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:18:53.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.630+0000 7f8d357fa640 10 monclient: dump: 2024-04-26T04:18:53.646 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:53.646 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:53.646 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:53.646 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:53.647 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:53.647 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:53.647 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:53.647 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:53.647 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:53.647 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:53.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.630+0000 7f8d357fa640 10 monclient: _finish_auth 0 2024-04-26T04:18:53.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.630+0000 7f8d357fa640 10 monclient: _check_auth_tickets 2024-04-26T04:18:53.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.630+0000 7f8d357fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:53.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.630+0000 7f8d357fa640 1 -- 172.21.15.63:0/347244099 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d2800ac60 con 0x7f8d380a2cd0 2024-04-26T04:18:53.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.630+0000 7f8d357fa640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:53.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.630+0000 7f8d357fa640 1 -- 172.21.15.63:0/347244099 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d28012660 con 0x7f8d380a2cd0 2024-04-26T04:18:53.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.630+0000 7f8d3eed5640 5 monclient: authenticate success, global_id 4347 2024-04-26T04:18:53.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.630+0000 7f8d357fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:53.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.630+0000 7f8d357fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:18:53.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.630+0000 7f8d357fa640 10 monclient: dump: 2024-04-26T04:18:53.648 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:53.648 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:53.648 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:53.648 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:53.648 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:53.648 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:53.648 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:53.648 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:53.648 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:53.648 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:53.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.630+0000 7f8d36ffd640 4 set_mon_vals no callback set 2024-04-26T04:18:53.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.630+0000 7f8d3eed5640 10 monclient: _renew_subs 2024-04-26T04:18:53.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.630+0000 7f8d3eed5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:53.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.630+0000 7f8d3eed5640 1 -- 172.21.15.63:0/347244099 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d38072b40 con 0x7f8d380a2cd0 2024-04-26T04:18:53.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.630+0000 7f8d3eed5640 10 monclient: _renew_subs 2024-04-26T04:18:53.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.630+0000 7f8d3eed5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:53.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.630+0000 7f8d3eed5640 1 -- 172.21.15.63:0/347244099 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d381c7e20 con 0x7f8d380a2cd0 2024-04-26T04:18:53.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.630+0000 7f8d3eed5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:53.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.630+0000 7f8d3eed5640 1 --2- 172.21.15.63:0/347244099 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d3802bfc0 0x7f8d3802c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:53.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.630+0000 7f8d3eed5640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:53.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.630+0000 7f8d3eed5640 1 -- 172.21.15.63:0/347244099 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f8d3813c220 con 0x7f8d3802bfc0 2024-04-26T04:18:53.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.630+0000 7f8d37fff640 1 --2- 172.21.15.63:0/347244099 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d3802bfc0 0x7f8d3802c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:53.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.630+0000 7f8d37fff640 10 monclient: get_auth_request con 0x7f8d3802bfc0 auth_method 0 2024-04-26T04:18:53.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.630+0000 7f8d37fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:53.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.630+0000 7f8d37fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:53.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.630+0000 7f8d37fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:53.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.630+0000 7f8d357fa640 1 -- 172.21.15.63:0/347244099 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f8d28019020 con 0x7f8d380a2cd0 2024-04-26T04:18:53.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.630+0000 7f8d37fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:53.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.630+0000 7f8d37fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:53.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.630+0000 7f8d37fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:53.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.630+0000 7f8d357fa640 1 --2- 172.21.15.63:0/347244099 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f8d24035ef0 0x7f8d240383b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:53.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.634+0000 7f8d37fff640 10 monclient(hunting): handle_auth_done global_id 4350 payload 824 2024-04-26T04:18:53.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.634+0000 7f8d357fa640 1 -- 172.21.15.63:0/347244099 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4738+0+0 (secure 0 0 0) 0x7f8d28048e80 con 0x7f8d380a2cd0 2024-04-26T04:18:53.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.634+0000 7f8d37fff640 1 -- 172.21.15.63:0/347244099 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8d38062e10 con 0x7f8d3802bfc0 2024-04-26T04:18:53.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.634+0000 7f8d3d472640 1 --2- 172.21.15.63:0/347244099 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f8d24035ef0 0x7f8d240383b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:53.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.634+0000 7f8d3d472640 10 monclient: get_auth_request con 0x7f8d24035ef0 auth_method 0 2024-04-26T04:18:53.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.634+0000 7f8d37fff640 1 -- 172.21.15.63:0/347244099 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d3802bfc0 msgr2=0x7f8d3802c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:53.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.634+0000 7f8d37fff640 1 --2- 172.21.15.63:0/347244099 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d3802bfc0 0x7f8d3802c3a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c0ad000 tx=0x7f8d2c01abf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:53.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.634+0000 7f8d357fa640 1 -- 172.21.15.63:0/347244099 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d2c0bb020 con 0x7f8d3802bfc0 2024-04-26T04:18:53.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.634+0000 7f8d357fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:53.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.634+0000 7f8d357fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:18:53.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.634+0000 7f8d357fa640 10 monclient: dump: 2024-04-26T04:18:53.651 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:53.651 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:53.651 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:53.651 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:53.651 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:53.651 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:53.651 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:53.651 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:53.652 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:53.652 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:53.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.634+0000 7f8d357fa640 1 -- 172.21.15.63:0/347244099 <== mon.0 v2:172.21.15.63:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8d2c0b6470 con 0x7f8d3802bfc0 2024-04-26T04:18:53.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.634+0000 7f8d357fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:53.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.634+0000 7f8d357fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:18:53.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.634+0000 7f8d357fa640 1 -- 172.21.15.63:0/347244099 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d3802bfc0 msgr2=0x7f8d3802c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:53.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.634+0000 7f8d357fa640 1 --2- 172.21.15.63:0/347244099 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d3802bfc0 0x7f8d3802c3a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c0ad000 tx=0x7f8d2c01abf0 comp rx=0 tx=0).stop 2024-04-26T04:18:53.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.634+0000 7f8d357fa640 1 -- 172.21.15.63:0/347244099 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d3802bfc0 msgr2=0x7f8d3802c3a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:18:53.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.634+0000 7f8d357fa640 1 --2- 172.21.15.63:0/347244099 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d3802bfc0 0x7f8d3802c3a0 secure :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c0ad000 tx=0x7f8d2c01abf0 comp rx=0 tx=0).stop 2024-04-26T04:18:53.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.634+0000 7f8d3d472640 1 --2- 172.21.15.63:0/347244099 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f8d24035ef0 0x7f8d240383b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8d30007ca0 tx=0x7f8d30007bf0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:53.654 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.642+0000 7f8d3eed5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T04:18:53.654 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.642+0000 7f8d3eed5640 1 --2- 172.21.15.63:0/347244099 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d3802c8e0 0x7f8d3802ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:53.654 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.642+0000 7f8d3eed5640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:53.654 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.642+0000 7f8d3eed5640 1 -- 172.21.15.63:0/347244099 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f8d38071410 con 0x7f8d3802c8e0 2024-04-26T04:18:53.654 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.642+0000 7f8d37fff640 1 --2- 172.21.15.63:0/347244099 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d3802c8e0 0x7f8d3802ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:53.654 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.642+0000 7f8d37fff640 10 monclient: get_auth_request con 0x7f8d3802c8e0 auth_method 0 2024-04-26T04:18:53.654 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.642+0000 7f8d37fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:53.654 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.642+0000 7f8d37fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:53.654 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.642+0000 7f8d37fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:53.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.642+0000 7f8d37fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:53.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.642+0000 7f8d37fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:53.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.642+0000 7f8d37fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:53.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.642+0000 7f8d37fff640 10 monclient(hunting): handle_auth_done global_id 4353 payload 824 2024-04-26T04:18:53.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.642+0000 7f8d37fff640 1 -- 172.21.15.63:0/347244099 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8d38139490 con 0x7f8d3802c8e0 2024-04-26T04:18:53.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.642+0000 7f8d37fff640 1 -- 172.21.15.63:0/347244099 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d3802c8e0 msgr2=0x7f8d3802ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:53.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.642+0000 7f8d37fff640 1 --2- 172.21.15.63:0/347244099 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d3802c8e0 0x7f8d3802ccf0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c009740 tx=0x7f8d2c019150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:53.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.642+0000 7f8d357fa640 1 -- 172.21.15.63:0/347244099 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d2c0bb070 con 0x7f8d3802c8e0 2024-04-26T04:18:53.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.642+0000 7f8d357fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:53.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.642+0000 7f8d357fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:18:53.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.642+0000 7f8d357fa640 10 monclient: dump: 2024-04-26T04:18:53.655 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:53.655 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:53.655 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:53.656 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:53.656 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:53.656 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:53.656 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:53.656 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:53.656 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:53.656 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:53.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.642+0000 7f8d357fa640 1 -- 172.21.15.63:0/347244099 <== mon.0 v2:172.21.15.63:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4722 (secure 0 0 0) 0x7f8d2c0b6620 con 0x7f8d3802c8e0 2024-04-26T04:18:53.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.642+0000 7f8d357fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T04:18:53.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.642+0000 7f8d357fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:18:53.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.642+0000 7f8d357fa640 1 -- 172.21.15.63:0/347244099 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d3802c8e0 msgr2=0x7f8d3802ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:53.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.642+0000 7f8d357fa640 1 --2- 172.21.15.63:0/347244099 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d3802c8e0 0x7f8d3802ccf0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c009740 tx=0x7f8d2c019150 comp rx=0 tx=0).stop 2024-04-26T04:18:53.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.642+0000 7f8d357fa640 1 -- 172.21.15.63:0/347244099 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d3802c8e0 msgr2=0x7f8d3802ccf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:18:53.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.642+0000 7f8d357fa640 1 --2- 172.21.15.63:0/347244099 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d3802c8e0 0x7f8d3802ccf0 secure :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c009740 tx=0x7f8d2c019150 comp rx=0 tx=0).stop 2024-04-26T04:18:53.657 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-26T04:18:53.657 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a", 2024-04-26T04:18:53.657 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 0, 2024-04-26T04:18:53.657 INFO:teuthology.orchestra.run.smithi063.stdout: "state": "leader", 2024-04-26T04:18:53.657 INFO:teuthology.orchestra.run.smithi063.stdout: "election_epoch": 8, 2024-04-26T04:18:53.657 INFO:teuthology.orchestra.run.smithi063.stdout: "uptime": 24795, 2024-04-26T04:18:53.657 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum": [ 2024-04-26T04:18:53.657 INFO:teuthology.orchestra.run.smithi063.stdout: 0, 2024-04-26T04:18:53.657 INFO:teuthology.orchestra.run.smithi063.stdout: 1, 2024-04-26T04:18:53.657 INFO:teuthology.orchestra.run.smithi063.stdout: 2 2024-04-26T04:18:53.657 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:18:53.658 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_age": 19, 2024-04-26T04:18:53.658 INFO:teuthology.orchestra.run.smithi063.stdout: "features": { 2024-04-26T04:18:53.658 INFO:teuthology.orchestra.run.smithi063.stdout: "required_con": "2450521708007866372", 2024-04-26T04:18:53.658 INFO:teuthology.orchestra.run.smithi063.stdout: "required_mon": [ 2024-04-26T04:18:53.658 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:18:53.658 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:18:53.658 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:18:53.658 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:18:53.658 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:18:53.658 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:18:53.658 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:18:53.658 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:18:53.658 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:18:53.659 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:18:53.659 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:18:53.659 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:18:53.659 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_con": "4540701547738038271", 2024-04-26T04:18:53.659 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_mon": [ 2024-04-26T04:18:53.659 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:18:53.659 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:18:53.659 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:18:53.659 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:18:53.659 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:18:53.659 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:18:53.659 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:18:53.659 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:18:53.659 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:18:53.660 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:18:53.660 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:18:53.660 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:18:53.660 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:18:53.660 INFO:teuthology.orchestra.run.smithi063.stdout: "outside_quorum": [], 2024-04-26T04:18:53.660 INFO:teuthology.orchestra.run.smithi063.stdout: "extra_probe_peers": [], 2024-04-26T04:18:53.660 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_provider": [], 2024-04-26T04:18:53.660 INFO:teuthology.orchestra.run.smithi063.stdout: "monmap": { 2024-04-26T04:18:53.660 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 1, 2024-04-26T04:18:53.660 INFO:teuthology.orchestra.run.smithi063.stdout: "fsid": "cd1c6a0c-4e4b-4014-89c0-2e535479ef39", 2024-04-26T04:18:53.660 INFO:teuthology.orchestra.run.smithi063.stdout: "modified": "2024-04-26T04:18:03.702339Z", 2024-04-26T04:18:53.660 INFO:teuthology.orchestra.run.smithi063.stdout: "created": "2024-04-26T04:18:03.702339Z", 2024-04-26T04:18:53.660 INFO:teuthology.orchestra.run.smithi063.stdout: "min_mon_release": 19, 2024-04-26T04:18:53.661 INFO:teuthology.orchestra.run.smithi063.stdout: "min_mon_release_name": "squid", 2024-04-26T04:18:53.661 INFO:teuthology.orchestra.run.smithi063.stdout: "election_strategy": 1, 2024-04-26T04:18:53.661 INFO:teuthology.orchestra.run.smithi063.stdout: "disallowed_leaders: ": "", 2024-04-26T04:18:53.661 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": false, 2024-04-26T04:18:53.661 INFO:teuthology.orchestra.run.smithi063.stdout: "tiebreaker_mon": "", 2024-04-26T04:18:53.661 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_ranks: ": "", 2024-04-26T04:18:53.661 INFO:teuthology.orchestra.run.smithi063.stdout: "features": { 2024-04-26T04:18:53.661 INFO:teuthology.orchestra.run.smithi063.stdout: "persistent": [ 2024-04-26T04:18:53.661 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:18:53.661 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:18:53.661 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:18:53.661 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:18:53.661 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:18:53.661 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:18:53.662 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:18:53.662 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:18:53.662 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:18:53.662 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:18:53.662 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:18:53.662 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:18:53.662 INFO:teuthology.orchestra.run.smithi063.stdout: "optional": [] 2024-04-26T04:18:53.662 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:18:53.662 INFO:teuthology.orchestra.run.smithi063.stdout: "mons": [ 2024-04-26T04:18:53.662 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:18:53.662 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 0, 2024-04-26T04:18:53.662 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a", 2024-04-26T04:18:53.662 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:18:53.663 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:18:53.663 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:18:53.663 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:18:53.663 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:3300", 2024-04-26T04:18:53.663 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:18:53.663 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:18:53.663 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:18:53.663 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:18:53.663 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6789", 2024-04-26T04:18:53.663 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:18:53.663 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:18:53.663 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:18:53.663 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:18:53.663 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6789/0", 2024-04-26T04:18:53.663 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6789/0", 2024-04-26T04:18:53.664 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:18:53.664 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:18:53.664 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:18:53.664 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:18:53.664 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:18:53.664 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 1, 2024-04-26T04:18:53.664 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b", 2024-04-26T04:18:53.664 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:18:53.664 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:18:53.664 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:18:53.664 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:18:53.664 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:3300", 2024-04-26T04:18:53.664 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:18:53.664 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:18:53.664 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:18:53.665 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:18:53.665 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:6789", 2024-04-26T04:18:53.665 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:18:53.665 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:18:53.665 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:18:53.665 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:18:53.665 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:6789/0", 2024-04-26T04:18:53.665 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.186:6789/0", 2024-04-26T04:18:53.665 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:18:53.665 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:18:53.665 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:18:53.665 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:18:53.665 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:18:53.665 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 2, 2024-04-26T04:18:53.666 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "c", 2024-04-26T04:18:53.666 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:18:53.666 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:18:53.666 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:18:53.666 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:18:53.666 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:3301", 2024-04-26T04:18:53.666 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:18:53.666 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:18:53.666 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:18:53.666 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:18:53.666 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6790", 2024-04-26T04:18:53.666 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:18:53.666 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:18:53.666 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:18:53.667 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:18:53.667 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6790/0", 2024-04-26T04:18:53.667 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6790/0", 2024-04-26T04:18:53.667 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:18:53.667 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:18:53.667 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:18:53.667 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:18:53.667 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:18:53.667 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:18:53.667 INFO:teuthology.orchestra.run.smithi063.stdout: "feature_map": { 2024-04-26T04:18:53.667 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": [ 2024-04-26T04:18:53.667 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:18:53.667 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:18:53.667 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:18:53.668 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 1 2024-04-26T04:18:53.668 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:18:53.668 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:18:53.668 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": [ 2024-04-26T04:18:53.668 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:18:53.668 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:18:53.668 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:18:53.668 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 2 2024-04-26T04:18:53.668 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:18:53.668 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:18:53.668 INFO:teuthology.orchestra.run.smithi063.stdout: "client": [ 2024-04-26T04:18:53.668 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:18:53.668 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:18:53.668 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:18:53.669 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 2 2024-04-26T04:18:53.669 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:18:53.669 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:18:53.669 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:18:53.669 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": false 2024-04-26T04:18:53.669 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-26T04:18:53.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.646+0000 7f8d3eed5640 1 -- 172.21.15.63:0/347244099 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f8d24035ef0 msgr2=0x7f8d240383b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:53.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.646+0000 7f8d3eed5640 1 --2- 172.21.15.63:0/347244099 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f8d24035ef0 0x7f8d240383b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8d30007ca0 tx=0x7f8d30007bf0 comp rx=0 tx=0).stop 2024-04-26T04:18:53.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.646+0000 7f8d3eed5640 10 monclient: shutdown 2024-04-26T04:18:53.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.646+0000 7f8d3eed5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:53.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.646+0000 7f8d3eed5640 1 -- 172.21.15.63:0/347244099 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d380a2cd0 msgr2=0x7f8d381cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:53.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.646+0000 7f8d3eed5640 1 --2- 172.21.15.63:0/347244099 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d380a2cd0 0x7f8d381cf590 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8d2800c7f0 tx=0x7f8d2800ccb0 comp rx=0 tx=0).stop 2024-04-26T04:18:53.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.646+0000 7f8d3d472640 1 -- 172.21.15.63:0/347244099 reap_dead start 2024-04-26T04:18:53.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.646+0000 7f8d3d472640 1 -- 172.21.15.63:0/347244099 reap_dead start 2024-04-26T04:18:53.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.646+0000 7f8d3eed5640 1 -- 172.21.15.63:0/347244099 shutdown_connections 2024-04-26T04:18:53.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.646+0000 7f8d3eed5640 1 -- 172.21.15.63:0/347244099 >> 172.21.15.63:0/347244099 conn(0x7f8d380a5f40 msgr2=0x7f8d380a7340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:53.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.646+0000 7f8d3eed5640 1 -- 172.21.15.63:0/347244099 shutdown_connections 2024-04-26T04:18:53.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.646+0000 7f8d3eed5640 1 -- 172.21.15.63:0/347244099 wait complete. 2024-04-26T04:18:53.671 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T04:18:53.671 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-26T04:18:53.672 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T04:18:53.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.778+0000 7f48f16d2640 10 monclient: get_monmap_and_config 2024-04-26T04:18:53.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.778+0000 7f48f16d2640 10 monclient: build_initial_monmap 2024-04-26T04:18:53.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.778+0000 7f48f16d2640 10 monclient: monmap: 2024-04-26T04:18:53.782 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:53.782 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:53.782 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:53.781912+0000 2024-04-26T04:18:53.782 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:53.781912+0000 2024-04-26T04:18:53.782 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:53.782 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:53.782 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:18:53.782 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:18:53.782 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:18:53.783 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:18:53.783 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:18:53.783 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:53.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.778+0000 7f48f16d2640 1 Processor -- start 2024-04-26T04:18:53.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.778+0000 7f48f16d2640 1 -- start start 2024-04-26T04:18:53.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.778+0000 7f48f16d2640 10 monclient: init 2024-04-26T04:18:53.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.778+0000 7f48f16d2640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:53.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.778+0000 7f48f16d2640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-04-26T04:18:53.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.778+0000 7f48f16d2640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f48ec0a2cd0 0x7f48ec0a30b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:53.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.778+0000 7f48f16d2640 10 monclient(hunting): picked mon.noname-b con 0x7f48ec0a2cd0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:53.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.778+0000 7f48f16d2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f48ec0a47d0 addr v1:172.21.15.63:6789/0 2024-04-26T04:18:53.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.778+0000 7f48f16d2640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f48ec0a4bd0 0x7f48ec1373f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:53.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.778+0000 7f48f16d2640 10 monclient(hunting): picked mon.noname-a con 0x7f48ec0a4bd0 addr v2:172.21.15.63:3300/0 2024-04-26T04:18:53.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.778+0000 7f48f16d2640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48ec0a0fb0 con 0x7f48ec0a47d0 2024-04-26T04:18:53.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.778+0000 7f48f16d2640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:53.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.778+0000 7f48f16d2640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7f48ec0a35f0 con 0x7f48ec0a4bd0 2024-04-26T04:18:53.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.778+0000 7f48f16d2640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:53.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.778+0000 7f48f16d2640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f48ec0a5380 con 0x7f48ec0a2cd0 2024-04-26T04:18:53.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.778+0000 7f48f16d2640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:53.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.778+0000 7f48f16d2640 10 monclient(hunting): authenticate will time out at 965.489543s 2024-04-26T04:18:53.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.778+0000 7f48eaffd640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f48ec0a2cd0 0x7f48ec0a30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:53.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.778+0000 7f48eb7fe640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f48ec0a4bd0 0x7f48ec1373f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:53.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.778+0000 7f48eaffd640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f48ec0a2cd0 0x7f48ec0a30b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:37012/0 (socket says 172.21.15.63:37012) 2024-04-26T04:18:53.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.778+0000 7f48eb7fe640 10 monclient(hunting): get_auth_request con 0x7f48ec0a4bd0 auth_method 0 2024-04-26T04:18:53.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.778+0000 7f48ea7fc640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f48ec0a47d0 0x7f48ec136ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:32826/0 (socket says 172.21.15.63:32826) 2024-04-26T04:18:53.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.778+0000 7f48eb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:53.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.778+0000 7f48eb7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:53.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.778+0000 7f48eaffd640 1 -- 172.21.15.63:0/4086858622 learned_addr learned my addr 172.21.15.63:0/4086858622 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:18:53.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.778+0000 7f48eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:53.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.778+0000 7f48eaffd640 10 monclient(hunting): get_auth_request con 0x7f48ec0a2cd0 auth_method 0 2024-04-26T04:18:53.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.778+0000 7f48eaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:53.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.778+0000 7f48eaffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:53.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.778+0000 7f48eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:53.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.778+0000 7f48eb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:53.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.778+0000 7f48eb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:53.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.778+0000 7f48eb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:53.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.778+0000 7f48eaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:53.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.778+0000 7f48eaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:53.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.778+0000 7f48eaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:53.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.778+0000 7f48eb7fe640 10 monclient(hunting): handle_auth_done global_id 4356 payload 274 2024-04-26T04:18:53.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.778+0000 7f48eb7fe640 1 -- 172.21.15.63:0/4086858622 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f48ec0a2cd0 msgr2=0x7f48ec0a30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:53.788 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.778+0000 7f48eb7fe640 1 --2- 172.21.15.63:0/4086858622 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f48ec0a2cd0 0x7f48ec0a30b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:53.788 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.778+0000 7f48eb7fe640 1 -- 172.21.15.63:0/4086858622 >> v1:172.21.15.63:6789/0 conn(0x7f48ec0a47d0 legacy=0x7f48ec136ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:53.788 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.778+0000 7f48eb7fe640 10 monclient: _finish_hunting 0 2024-04-26T04:18:53.788 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.778+0000 7f48eb7fe640 1 monclient: found mon.noname-a 2024-04-26T04:18:53.788 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.778+0000 7f48eb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:53.788 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.778+0000 7f48eb7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:18:53.788 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.778+0000 7f48eb7fe640 1 -- 172.21.15.63:0/4086858622 --> v2:172.21.15.63:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48ec0a1290 con 0x7f48ec0a4bd0 2024-04-26T04:18:53.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.782+0000 7f48eb7fe640 1 -- 172.21.15.63:0/4086858622 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f48ec0a4bd0 msgr2=0x7f48ec1373f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:53.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.782+0000 7f48eb7fe640 1 --2- 172.21.15.63:0/4086858622 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f48ec0a4bd0 0x7f48ec1373f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f48e40097c0 tx=0x7f48e401c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc074d47d6aba0ff server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:53.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.782+0000 7f48e9ffb640 1 -- 172.21.15.63:0/4086858622 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48e40b0070 con 0x7f48ec0a4bd0 2024-04-26T04:18:53.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.782+0000 7f48e9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:53.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.782+0000 7f48e9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:18:53.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.782+0000 7f48e9ffb640 10 monclient: dump: 2024-04-26T04:18:53.792 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:53.792 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:53.792 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:53.792 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:53.792 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:53.792 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:53.792 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:53.793 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:53.793 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:53.793 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:53.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.782+0000 7f48e9ffb640 10 monclient: _finish_auth 0 2024-04-26T04:18:53.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.782+0000 7f48e9ffb640 10 monclient: _check_auth_tickets 2024-04-26T04:18:53.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.782+0000 7f48e9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:53.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.782+0000 7f48e9ffb640 1 -- 172.21.15.63:0/4086858622 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48e40ab400 con 0x7f48ec0a4bd0 2024-04-26T04:18:53.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.782+0000 7f48f16d2640 5 monclient: authenticate success, global_id 4356 2024-04-26T04:18:53.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.782+0000 7f48f16d2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T04:18:53.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.782+0000 7f48e9ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:53.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.782+0000 7f48f16d2640 10 monclient: get_monmap_and_config success 2024-04-26T04:18:53.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.782+0000 7f48e9ffb640 1 -- 172.21.15.63:0/4086858622 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48e40ab7d0 con 0x7f48ec0a4bd0 2024-04-26T04:18:53.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.782+0000 7f48e9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:53.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.782+0000 7f48f16d2640 4 set_mon_vals no callback set 2024-04-26T04:18:53.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.782+0000 7f48e9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:18:53.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.782+0000 7f48e9ffb640 10 monclient: dump: 2024-04-26T04:18:53.794 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:53.794 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:53.794 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:53.794 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:53.794 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:53.794 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:53.794 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:53.794 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:53.794 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:53.794 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:53.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.782+0000 7f48f16d2640 10 monclient: shutdown 2024-04-26T04:18:53.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.782+0000 7f48f16d2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:53.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.782+0000 7f48f16d2640 1 -- 172.21.15.63:0/4086858622 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f48ec0a4bd0 msgr2=0x7f48ec1373f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:53.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.782+0000 7f48f16d2640 1 --2- 172.21.15.63:0/4086858622 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f48ec0a4bd0 0x7f48ec1373f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f48e40097c0 tx=0x7f48e401c3b0 comp rx=0 tx=0).stop 2024-04-26T04:18:53.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.782+0000 7f48f16d2640 1 -- 172.21.15.63:0/4086858622 shutdown_connections 2024-04-26T04:18:53.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.782+0000 7f48f16d2640 1 --2- 172.21.15.63:0/4086858622 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f48ec0a4bd0 0x7f48ec1373f0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:53.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.782+0000 7f48f16d2640 1 --2- 172.21.15.63:0/4086858622 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f48ec0a2cd0 0x7f48ec0a30b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:53.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.782+0000 7f48f16d2640 1 -- 172.21.15.63:0/4086858622 >> 172.21.15.63:0/4086858622 conn(0x7f48ec0a5f40 msgr2=0x7f48ec0a6320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:53.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.782+0000 7f48f16d2640 1 -- 172.21.15.63:0/4086858622 shutdown_connections 2024-04-26T04:18:53.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.782+0000 7f48f16d2640 1 -- 172.21.15.63:0/4086858622 wait complete. 2024-04-26T04:18:53.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.782+0000 7f48f16d2640 10 monclient: build_initial_monmap 2024-04-26T04:18:53.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.782+0000 7f48f16d2640 10 monclient: monmap: 2024-04-26T04:18:53.795 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:53.795 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:18:53.795 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:18:53.795 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:18:53.795 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:53.795 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:53.796 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:18:53.796 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:18:53.796 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:18:53.796 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:53.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.786+0000 7f48f16d2640 1 Processor -- start 2024-04-26T04:18:53.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.786+0000 7f48f16d2640 1 -- start start 2024-04-26T04:18:53.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.786+0000 7f48f16d2640 10 monclient: init 2024-04-26T04:18:53.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.786+0000 7f48f16d2640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:53.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.786+0000 7f48f16d2640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T04:18:53.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.786+0000 7f48f16d2640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f48ec0a2cd0 0x7f48ec1cf020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:53.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.786+0000 7f48f16d2640 10 monclient(hunting): picked mon.noname-c con 0x7f48ec0a2cd0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:53.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.786+0000 7f48f16d2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48ec0a47d0 0x7f48ec135bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:53.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.786+0000 7f48f16d2640 10 monclient(hunting): picked mon.noname-b con 0x7f48ec0a47d0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:18:53.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.786+0000 7f48f16d2640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f48ec0a4bd0 0x7f48ec13bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:53.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.786+0000 7f48f16d2640 10 monclient(hunting): picked mon.noname-a con 0x7f48ec0a4bd0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:18:53.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.786+0000 7f48f16d2640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:53.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.786+0000 7f48f16d2640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f48ec141bd0 con 0x7f48ec0a4bd0 2024-04-26T04:18:53.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.786+0000 7f48f16d2640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:53.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.786+0000 7f48f16d2640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f48ec141a50 con 0x7f48ec0a47d0 2024-04-26T04:18:53.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.786+0000 7f48f16d2640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:53.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.786+0000 7f48f16d2640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f48ec141d50 con 0x7f48ec0a2cd0 2024-04-26T04:18:53.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.786+0000 7f48f16d2640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:53.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.786+0000 7f48f16d2640 10 monclient(hunting): authenticate will time out at 965.496649s 2024-04-26T04:18:53.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.786+0000 7f48eaffd640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f48ec0a2cd0 0x7f48ec1cf020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:53.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.786+0000 7f48eaffd640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f48ec0a2cd0 0x7f48ec1cf020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:37016/0 (socket says 172.21.15.63:37016) 2024-04-26T04:18:53.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.786+0000 7f48eaffd640 1 -- 172.21.15.63:0/1234896608 learned_addr learned my addr 172.21.15.63:0/1234896608 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:18:53.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.786+0000 7f48eaffd640 10 monclient(hunting): get_auth_request con 0x7f48ec0a2cd0 auth_method 0 2024-04-26T04:18:53.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.786+0000 7f48eaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:53.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.786+0000 7f48eaffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:53.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.786+0000 7f48eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:53.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.786+0000 7f48eb7fe640 1 --2- 172.21.15.63:0/1234896608 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f48ec0a4bd0 0x7f48ec13bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:53.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.786+0000 7f48eb7fe640 10 monclient(hunting): get_auth_request con 0x7f48ec0a4bd0 auth_method 0 2024-04-26T04:18:53.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.786+0000 7f48eb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:53.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.786+0000 7f48eb7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:53.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.786+0000 7f48eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:53.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.786+0000 7f48eaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:53.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.786+0000 7f48eaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:53.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.786+0000 7f48eaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:53.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.786+0000 7f48eb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:53.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.786+0000 7f48eb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:53.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.786+0000 7f48eb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:53.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.786+0000 7f48eaffd640 10 monclient(hunting): handle_auth_done global_id 4331 payload 824 2024-04-26T04:18:53.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.786+0000 7f48eaffd640 1 -- 172.21.15.63:0/1234896608 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48ec0a47d0 msgr2=0x7f48ec135bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:53.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.786+0000 7f48eaffd640 1 --2- 172.21.15.63:0/1234896608 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48ec0a47d0 0x7f48ec135bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:53.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.786+0000 7f48eaffd640 1 -- 172.21.15.63:0/1234896608 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f48ec0a4bd0 msgr2=0x7f48ec13bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:53.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.786+0000 7f48eaffd640 1 --2- 172.21.15.63:0/1234896608 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f48ec0a4bd0 0x7f48ec13bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:53.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.786+0000 7f48eaffd640 10 monclient: _finish_hunting 0 2024-04-26T04:18:53.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.786+0000 7f48eaffd640 1 monclient: found mon.noname-c 2024-04-26T04:18:53.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.786+0000 7f48eaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:53.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.786+0000 7f48eaffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.63:3301/0 2024-04-26T04:18:53.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.786+0000 7f48eaffd640 1 -- 172.21.15.63:0/1234896608 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48ec0a37d0 con 0x7f48ec0a2cd0 2024-04-26T04:18:53.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.786+0000 7f48eaffd640 1 -- 172.21.15.63:0/1234896608 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f48ec0a2cd0 msgr2=0x7f48ec1cf020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:53.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.786+0000 7f48eaffd640 1 --2- 172.21.15.63:0/1234896608 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f48ec0a2cd0 0x7f48ec1cf020 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f48dc00efc0 tx=0x7f48dc00c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:53.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.786+0000 7f48e37fe640 1 -- 172.21.15.63:0/1234896608 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48dc014480 con 0x7f48ec0a2cd0 2024-04-26T04:18:53.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.786+0000 7f48e37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:53.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.786+0000 7f48e37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:18:53.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.786+0000 7f48e37fe640 10 monclient: dump: 2024-04-26T04:18:53.804 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:53.804 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:53.804 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:53.804 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:53.804 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:53.804 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:53.804 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:53.804 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:53.804 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:53.804 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:53.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.786+0000 7f48e37fe640 10 monclient: _finish_auth 0 2024-04-26T04:18:53.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.786+0000 7f48e37fe640 10 monclient: _check_auth_tickets 2024-04-26T04:18:53.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.786+0000 7f48e37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:53.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.786+0000 7f48f16d2640 5 monclient: authenticate success, global_id 4331 2024-04-26T04:18:53.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.786+0000 7f48e37fe640 1 -- 172.21.15.63:0/1234896608 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48dc019070 con 0x7f48ec0a2cd0 2024-04-26T04:18:53.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.786+0000 7f48e37fe640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:53.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.786+0000 7f48e37fe640 1 -- 172.21.15.63:0/1234896608 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48dc014840 con 0x7f48ec0a2cd0 2024-04-26T04:18:53.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.786+0000 7f48f16d2640 10 monclient: _renew_subs 2024-04-26T04:18:53.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.786+0000 7f48f16d2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:53.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.786+0000 7f48f16d2640 1 -- 172.21.15.63:0/1234896608 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48ec072960 con 0x7f48ec0a2cd0 2024-04-26T04:18:53.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.786+0000 7f48e37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:53.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.786+0000 7f48e8ff9640 4 set_mon_vals no callback set 2024-04-26T04:18:53.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.786+0000 7f48e37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:18:53.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.786+0000 7f48e37fe640 10 monclient: dump: 2024-04-26T04:18:53.805 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:53.805 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:53.805 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:53.805 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:53.806 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:53.806 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:53.806 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:53.806 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:53.806 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:53.806 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:53.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.786+0000 7f48f16d2640 10 monclient: _renew_subs 2024-04-26T04:18:53.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.786+0000 7f48f16d2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:18:53.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.786+0000 7f48f16d2640 1 -- 172.21.15.63:0/1234896608 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f48ec1202a0 con 0x7f48ec0a2cd0 2024-04-26T04:18:53.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.790+0000 7f48f16d2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:53.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.790+0000 7f48f16d2640 1 --2- 172.21.15.63:0/1234896608 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f48ec13c060 0x7f48ec13c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:53.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.790+0000 7f48f16d2640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:53.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.790+0000 7f48f16d2640 1 -- 172.21.15.63:0/1234896608 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f48ec13c980 con 0x7f48ec13c060 2024-04-26T04:18:53.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.790+0000 7f48ea7fc640 1 --2- 172.21.15.63:0/1234896608 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f48ec13c060 0x7f48ec13c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:53.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.790+0000 7f48ea7fc640 10 monclient: get_auth_request con 0x7f48ec13c060 auth_method 0 2024-04-26T04:18:53.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.790+0000 7f48ea7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:53.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.790+0000 7f48ea7fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:53.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.790+0000 7f48ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:53.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.790+0000 7f48ea7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:53.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.790+0000 7f48ea7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:53.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.790+0000 7f48ea7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:53.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.790+0000 7f48e37fe640 1 -- 172.21.15.63:0/1234896608 <== mon.2 v2:172.21.15.63:3301/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f48dc012050 con 0x7f48ec0a2cd0 2024-04-26T04:18:53.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.790+0000 7f48ea7fc640 10 monclient(hunting): handle_auth_done global_id 4334 payload 824 2024-04-26T04:18:53.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.790+0000 7f48ea7fc640 1 -- 172.21.15.63:0/1234896608 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f48ec1cff00 con 0x7f48ec13c060 2024-04-26T04:18:53.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.790+0000 7f48e37fe640 1 --2- 172.21.15.63:0/1234896608 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f48cc035f10 0x7f48cc0383d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:53.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.790+0000 7f48e37fe640 1 -- 172.21.15.63:0/1234896608 <== mon.2 v2:172.21.15.63:3301/0 5 ==== osd_map(16..16 src has 1..16) ==== 4738+0+0 (secure 0 0 0) 0x7f48dc0492e0 con 0x7f48ec0a2cd0 2024-04-26T04:18:53.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.790+0000 7f48eb7fe640 1 --2- 172.21.15.63:0/1234896608 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f48cc035f10 0x7f48cc0383d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:53.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.790+0000 7f48eb7fe640 10 monclient: get_auth_request con 0x7f48cc035f10 auth_method 0 2024-04-26T04:18:53.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.790+0000 7f48ea7fc640 1 -- 172.21.15.63:0/1234896608 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f48ec13c060 msgr2=0x7f48ec13c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:53.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.790+0000 7f48ea7fc640 1 --2- 172.21.15.63:0/1234896608 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f48ec13c060 0x7f48ec13c440 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f48ec13d3c0 tx=0x7f48d400c7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:53.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.790+0000 7f48e37fe640 1 -- 172.21.15.63:0/1234896608 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48d4017070 con 0x7f48ec13c060 2024-04-26T04:18:53.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.790+0000 7f48e37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:53.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.790+0000 7f48e37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:18:53.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.790+0000 7f48e37fe640 10 monclient: dump: 2024-04-26T04:18:53.808 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:53.808 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:53.808 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:53.808 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:53.808 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:53.808 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:53.808 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:53.808 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:53.808 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:53.809 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:53.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.790+0000 7f48e37fe640 1 -- 172.21.15.63:0/1234896608 <== mon.2 v2:172.21.15.63:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f48d4005800 con 0x7f48ec13c060 2024-04-26T04:18:53.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.790+0000 7f48e37fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:53.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.790+0000 7f48e37fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:18:53.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.790+0000 7f48e37fe640 1 -- 172.21.15.63:0/1234896608 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f48ec13c060 msgr2=0x7f48ec13c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:53.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.790+0000 7f48e37fe640 1 --2- 172.21.15.63:0/1234896608 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f48ec13c060 0x7f48ec13c440 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f48ec13d3c0 tx=0x7f48d400c7a0 comp rx=0 tx=0).stop 2024-04-26T04:18:53.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.790+0000 7f48e37fe640 1 -- 172.21.15.63:0/1234896608 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f48ec13c060 msgr2=0x7f48ec13c440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:18:53.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.790+0000 7f48e37fe640 1 --2- 172.21.15.63:0/1234896608 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f48ec13c060 0x7f48ec13c440 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:53.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.794+0000 7f48eb7fe640 1 --2- 172.21.15.63:0/1234896608 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f48cc035f10 0x7f48cc0383d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f48ec06d7c0 tx=0x7f48e4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:53.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.798+0000 7f48f16d2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T04:18:53.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.798+0000 7f48f16d2640 1 --2- 172.21.15.63:0/1234896608 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f48ec137cf0 0x7f48ec138100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:53.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.798+0000 7f48f16d2640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:53.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.798+0000 7f48f16d2640 1 -- 172.21.15.63:0/1234896608 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f48ec062e10 con 0x7f48ec137cf0 2024-04-26T04:18:53.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.798+0000 7f48ea7fc640 1 --2- 172.21.15.63:0/1234896608 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f48ec137cf0 0x7f48ec138100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:53.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.798+0000 7f48ea7fc640 10 monclient: get_auth_request con 0x7f48ec137cf0 auth_method 0 2024-04-26T04:18:53.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.798+0000 7f48ea7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:53.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.798+0000 7f48ea7fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:53.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.798+0000 7f48ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:53.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.798+0000 7f48ea7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:53.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.798+0000 7f48ea7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:53.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.798+0000 7f48ea7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:53.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.798+0000 7f48ea7fc640 10 monclient(hunting): handle_auth_done global_id 4337 payload 824 2024-04-26T04:18:53.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.798+0000 7f48ea7fc640 1 -- 172.21.15.63:0/1234896608 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f48ec0a7e60 con 0x7f48ec137cf0 2024-04-26T04:18:53.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.798+0000 7f48ea7fc640 1 -- 172.21.15.63:0/1234896608 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f48ec137cf0 msgr2=0x7f48ec138100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:53.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.798+0000 7f48ea7fc640 1 --2- 172.21.15.63:0/1234896608 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f48ec137cf0 0x7f48ec138100 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f48d4014000 tx=0x7f48d4004940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:53.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.802+0000 7f48e37fe640 1 -- 172.21.15.63:0/1234896608 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48d4017070 con 0x7f48ec137cf0 2024-04-26T04:18:53.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.802+0000 7f48e37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:53.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.802+0000 7f48e37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:18:53.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.802+0000 7f48e37fe640 10 monclient: dump: 2024-04-26T04:18:53.812 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:53.812 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:53.812 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:53.812 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:53.812 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:53.812 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:53.812 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:53.812 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:53.812 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:53.812 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:53.812 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-26T04:18:53.813 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "c", 2024-04-26T04:18:53.813 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 2, 2024-04-26T04:18:53.813 INFO:teuthology.orchestra.run.smithi063.stdout: "state": "peon", 2024-04-26T04:18:53.813 INFO:teuthology.orchestra.run.smithi063.stdout: "election_epoch": 8, 2024-04-26T04:18:53.813 INFO:teuthology.orchestra.run.smithi063.stdout: "uptime": 24955, 2024-04-26T04:18:53.813 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum": [ 2024-04-26T04:18:53.813 INFO:teuthology.orchestra.run.smithi063.stdout: 0, 2024-04-26T04:18:53.813 INFO:teuthology.orchestra.run.smithi063.stdout: 1, 2024-04-26T04:18:53.814 INFO:teuthology.orchestra.run.smithi063.stdout: 2 2024-04-26T04:18:53.814 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:18:53.814 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_age": 19, 2024-04-26T04:18:53.814 INFO:teuthology.orchestra.run.smithi063.stdout: "features": { 2024-04-26T04:18:53.814 INFO:teuthology.orchestra.run.smithi063.stdout: "required_con": "2450521708007866372", 2024-04-26T04:18:53.814 INFO:teuthology.orchestra.run.smithi063.stdout: "required_mon": [ 2024-04-26T04:18:53.814 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:18:53.814 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:18:53.814 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:18:53.814 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:18:53.814 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:18:53.815 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:18:53.815 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:18:53.815 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:18:53.815 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:18:53.815 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:18:53.815 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:18:53.815 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:18:53.815 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_con": "4540701547738038271", 2024-04-26T04:18:53.815 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_mon": [ 2024-04-26T04:18:53.815 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:18:53.815 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:18:53.815 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:18:53.815 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:18:53.816 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:18:53.816 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:18:53.816 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:18:53.816 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:18:53.816 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:18:53.816 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:18:53.816 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:18:53.816 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:18:53.816 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:18:53.816 INFO:teuthology.orchestra.run.smithi063.stdout: "outside_quorum": [], 2024-04-26T04:18:53.816 INFO:teuthology.orchestra.run.smithi063.stdout: "extra_probe_peers": [], 2024-04-26T04:18:53.816 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_provider": [], 2024-04-26T04:18:53.816 INFO:teuthology.orchestra.run.smithi063.stdout: "monmap": { 2024-04-26T04:18:53.816 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 1, 2024-04-26T04:18:53.817 INFO:teuthology.orchestra.run.smithi063.stdout: "fsid": "cd1c6a0c-4e4b-4014-89c0-2e535479ef39", 2024-04-26T04:18:53.817 INFO:teuthology.orchestra.run.smithi063.stdout: "modified": "2024-04-26T04:18:03.702339Z", 2024-04-26T04:18:53.817 INFO:teuthology.orchestra.run.smithi063.stdout: "created": "2024-04-26T04:18:03.702339Z", 2024-04-26T04:18:53.817 INFO:teuthology.orchestra.run.smithi063.stdout: "min_mon_release": 19, 2024-04-26T04:18:53.817 INFO:teuthology.orchestra.run.smithi063.stdout: "min_mon_release_name": "squid", 2024-04-26T04:18:53.817 INFO:teuthology.orchestra.run.smithi063.stdout: "election_strategy": 1, 2024-04-26T04:18:53.817 INFO:teuthology.orchestra.run.smithi063.stdout: "disallowed_leaders: ": "", 2024-04-26T04:18:53.817 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": false, 2024-04-26T04:18:53.817 INFO:teuthology.orchestra.run.smithi063.stdout: "tiebreaker_mon": "", 2024-04-26T04:18:53.817 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_ranks: ": "", 2024-04-26T04:18:53.817 INFO:teuthology.orchestra.run.smithi063.stdout: "features": { 2024-04-26T04:18:53.817 INFO:teuthology.orchestra.run.smithi063.stdout: "persistent": [ 2024-04-26T04:18:53.817 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:18:53.817 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:18:53.817 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:18:53.818 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:18:53.818 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:18:53.818 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:18:53.818 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:18:53.818 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:18:53.818 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:18:53.818 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:18:53.818 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:18:53.818 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:18:53.818 INFO:teuthology.orchestra.run.smithi063.stdout: "optional": [] 2024-04-26T04:18:53.818 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:18:53.818 INFO:teuthology.orchestra.run.smithi063.stdout: "mons": [ 2024-04-26T04:18:53.819 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:18:53.819 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 0, 2024-04-26T04:18:53.819 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a", 2024-04-26T04:18:53.819 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:18:53.819 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:18:53.819 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:18:53.819 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:18:53.819 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:3300", 2024-04-26T04:18:53.819 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:18:53.819 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:18:53.819 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:18:53.819 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:18:53.819 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6789", 2024-04-26T04:18:53.819 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:18:53.819 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:18:53.820 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:18:53.820 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:18:53.820 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6789/0", 2024-04-26T04:18:53.820 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6789/0", 2024-04-26T04:18:53.820 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:18:53.820 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:18:53.820 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:18:53.820 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:18:53.820 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:18:53.820 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 1, 2024-04-26T04:18:53.820 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b", 2024-04-26T04:18:53.820 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:18:53.820 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:18:53.820 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:18:53.821 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:18:53.821 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:3300", 2024-04-26T04:18:53.821 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:18:53.821 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:18:53.821 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:18:53.821 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:18:53.821 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:6789", 2024-04-26T04:18:53.821 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:18:53.821 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:18:53.821 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:18:53.821 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:18:53.821 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:6789/0", 2024-04-26T04:18:53.821 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.186:6789/0", 2024-04-26T04:18:53.821 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:18:53.821 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:18:53.822 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:18:53.822 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:18:53.822 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:18:53.822 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 2, 2024-04-26T04:18:53.822 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "c", 2024-04-26T04:18:53.822 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:18:53.822 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:18:53.822 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:18:53.822 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:18:53.822 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:3301", 2024-04-26T04:18:53.822 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:18:53.822 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:18:53.822 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:18:53.822 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:18:53.823 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6790", 2024-04-26T04:18:53.823 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:18:53.823 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:18:53.823 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:18:53.823 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:18:53.823 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6790/0", 2024-04-26T04:18:53.823 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6790/0", 2024-04-26T04:18:53.823 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:18:53.823 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:18:53.823 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:18:53.823 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:18:53.823 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:18:53.823 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:18:53.823 INFO:teuthology.orchestra.run.smithi063.stdout: "feature_map": { 2024-04-26T04:18:53.824 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": [ 2024-04-26T04:18:53.824 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:18:53.824 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:18:53.824 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:18:53.824 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 1 2024-04-26T04:18:53.824 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:18:53.824 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:18:53.824 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": [ 2024-04-26T04:18:53.824 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:18:53.824 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:18:53.824 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:18:53.824 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 1 2024-04-26T04:18:53.824 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:18:53.825 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:18:53.825 INFO:teuthology.orchestra.run.smithi063.stdout: "client": [ 2024-04-26T04:18:53.825 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:18:53.825 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:18:53.825 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:18:53.825 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 3 2024-04-26T04:18:53.825 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:18:53.825 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:18:53.825 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:18:53.825 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": false 2024-04-26T04:18:53.825 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-26T04:18:53.830 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.802+0000 7f48e37fe640 1 -- 172.21.15.63:0/1234896608 <== mon.2 v2:172.21.15.63:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4720 (secure 0 0 0) 0x7f48d4022410 con 0x7f48ec137cf0 2024-04-26T04:18:53.830 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.802+0000 7f48e37fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T04:18:53.830 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.802+0000 7f48e37fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:18:53.830 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.802+0000 7f48e37fe640 1 -- 172.21.15.63:0/1234896608 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f48ec137cf0 msgr2=0x7f48ec138100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:53.831 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.802+0000 7f48e37fe640 1 --2- 172.21.15.63:0/1234896608 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f48ec137cf0 0x7f48ec138100 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f48d4014000 tx=0x7f48d4004940 comp rx=0 tx=0).stop 2024-04-26T04:18:53.831 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.802+0000 7f48e37fe640 1 -- 172.21.15.63:0/1234896608 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f48ec137cf0 msgr2=0x7f48ec138100 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:18:53.831 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.802+0000 7f48e37fe640 1 --2- 172.21.15.63:0/1234896608 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f48ec137cf0 0x7f48ec138100 secure :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f48d4014000 tx=0x7f48d4004940 comp rx=0 tx=0).stop 2024-04-26T04:18:53.831 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.802+0000 7f48f16d2640 1 -- 172.21.15.63:0/1234896608 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f48cc035f10 msgr2=0x7f48cc0383d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:53.831 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.802+0000 7f48f16d2640 1 --2- 172.21.15.63:0/1234896608 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f48cc035f10 0x7f48cc0383d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f48ec06d7c0 tx=0x7f48e4008040 comp rx=0 tx=0).stop 2024-04-26T04:18:53.831 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.802+0000 7f48f16d2640 10 monclient: shutdown 2024-04-26T04:18:53.831 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.802+0000 7f48f16d2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:53.831 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.802+0000 7f48f16d2640 1 -- 172.21.15.63:0/1234896608 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f48ec0a2cd0 msgr2=0x7f48ec1cf020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:53.831 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.802+0000 7f48f16d2640 1 --2- 172.21.15.63:0/1234896608 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f48ec0a2cd0 0x7f48ec1cf020 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f48dc00efc0 tx=0x7f48dc00c490 comp rx=0 tx=0).stop 2024-04-26T04:18:53.831 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.802+0000 7f48eb7fe640 1 -- 172.21.15.63:0/1234896608 reap_dead start 2024-04-26T04:18:53.831 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.802+0000 7f48eb7fe640 1 -- 172.21.15.63:0/1234896608 reap_dead start 2024-04-26T04:18:53.831 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.802+0000 7f48f16d2640 1 -- 172.21.15.63:0/1234896608 shutdown_connections 2024-04-26T04:18:53.831 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.802+0000 7f48f16d2640 1 -- 172.21.15.63:0/1234896608 >> 172.21.15.63:0/1234896608 conn(0x7f48ec0a5f40 msgr2=0x7f48ec0a1a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:53.831 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.802+0000 7f48f16d2640 1 -- 172.21.15.63:0/1234896608 shutdown_connections 2024-04-26T04:18:53.831 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.802+0000 7f48f16d2640 1 -- 172.21.15.63:0/1234896608 wait complete. 2024-04-26T04:18:53.832 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T04:18:53.832 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-26T04:18:53.832 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T04:18:53.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.930+0000 7fdd34dcb640 10 monclient: get_monmap_and_config 2024-04-26T04:18:53.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.930+0000 7fdd34dcb640 10 monclient: build_initial_monmap 2024-04-26T04:18:53.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.930+0000 7fdd34dcb640 10 monclient: monmap: 2024-04-26T04:18:53.937 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:53.937 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:53.937 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:53.936948+0000 2024-04-26T04:18:53.937 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:53.936948+0000 2024-04-26T04:18:53.937 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:53.937 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:53.937 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:18:53.937 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:18:53.937 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:18:53.937 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:18:53.938 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:18:53.938 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:53.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.930+0000 7fdd34dcb640 1 Processor -- start 2024-04-26T04:18:53.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.934+0000 7fdd34dcb640 1 -- start start 2024-04-26T04:18:53.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.934+0000 7fdd34dcb640 10 monclient: init 2024-04-26T04:18:53.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.934+0000 7fdd34dcb640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:53.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.934+0000 7fdd34dcb640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-04-26T04:18:53.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.934+0000 7fdd34dcb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdd3009b3b0 addr v1:172.21.15.63:6789/0 2024-04-26T04:18:53.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.934+0000 7fdd34dcb640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fdd30137320 0x7fdd30139710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:53.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.934+0000 7fdd34dcb640 10 monclient(hunting): picked mon.noname-c con 0x7fdd30137320 addr v2:172.21.15.186:3300/0 2024-04-26T04:18:53.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.934+0000 7fdd34dcb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdd30139c50 addr v1:172.21.15.186:6789/0 2024-04-26T04:18:53.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.934+0000 7fdd34dcb640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd3009dde0 con 0x7fdd3009b3b0 2024-04-26T04:18:53.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.934+0000 7fdd34dcb640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd3009c9b0 con 0x7fdd30139c50 2024-04-26T04:18:53.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.934+0000 7fdd34dcb640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:53.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.934+0000 7fdd34dcb640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fdd3009c100 con 0x7fdd30137320 2024-04-26T04:18:53.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.934+0000 7fdd34dcb640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:53.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.934+0000 7fdd34dcb640 10 monclient(hunting): authenticate will time out at 965.644738s 2024-04-26T04:18:53.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.934+0000 7fdd2e59c640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7fdd3009b3b0 0x7fdd3009e4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:32840/0 (socket says 172.21.15.63:32840) 2024-04-26T04:18:53.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.934+0000 7fdd2e59c640 1 -- 172.21.15.63:0/3035079922 learned_addr learned my addr 172.21.15.63:0/3035079922 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-04-26T04:18:53.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.934+0000 7fdd2dd9b640 1 --2- 172.21.15.63:0/3035079922 >> v2:172.21.15.186:3300/0 conn(0x7fdd30137320 0x7fdd30139710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:53.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.934+0000 7fdd2dd9b640 10 monclient(hunting): get_auth_request con 0x7fdd30137320 auth_method 0 2024-04-26T04:18:53.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.934+0000 7fdd2dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:53.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.934+0000 7fdd2dd9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:53.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.934+0000 7fdd2dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:53.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.934+0000 7fdd2dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:53.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.934+0000 7fdd2dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:53.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.934+0000 7fdd2dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:53.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.934+0000 7fdd2dd9b640 10 monclient(hunting): handle_auth_done global_id 4312 payload 274 2024-04-26T04:18:53.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.934+0000 7fdd2dd9b640 1 -- 172.21.15.63:0/3035079922 >> v1:172.21.15.186:6789/0 conn(0x7fdd30139c50 legacy=0x7fdd3013c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:53.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.934+0000 7fdd2dd9b640 1 -- 172.21.15.63:0/3035079922 >> v1:172.21.15.63:6789/0 conn(0x7fdd3009b3b0 legacy=0x7fdd3009e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:53.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.934+0000 7fdd2dd9b640 10 monclient: _finish_hunting 0 2024-04-26T04:18:53.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.934+0000 7fdd2dd9b640 1 monclient: found mon.noname-c 2024-04-26T04:18:53.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.934+0000 7fdd2dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:53.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.934+0000 7fdd2dd9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.186:3300/0 2024-04-26T04:18:53.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.934+0000 7fdd2dd9b640 1 -- 172.21.15.63:0/3035079922 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd3009db00 con 0x7fdd30137320 2024-04-26T04:18:53.944 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.938+0000 7fdd2dd9b640 1 -- 172.21.15.63:0/3035079922 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd30137320 msgr2=0x7fdd30139710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:53.944 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.938+0000 7fdd2dd9b640 1 --2- 172.21.15.63:0/3035079922 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd30137320 0x7fdd30139710 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fdd180097f0 tx=0x7fdd1801c100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b47174f2b7a6ed86 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:53.945 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.938+0000 7fdd2d59a640 1 -- 172.21.15.63:0/3035079922 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd180b1070 con 0x7fdd30137320 2024-04-26T04:18:53.945 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.938+0000 7fdd2d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:53.945 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.938+0000 7fdd2d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:18:53.945 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.938+0000 7fdd2d59a640 10 monclient: dump: 2024-04-26T04:18:53.945 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:53.945 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:53.946 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:53.946 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:53.946 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:53.946 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:53.946 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:53.946 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:53.946 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:53.946 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:53.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.938+0000 7fdd2d59a640 10 monclient: _finish_auth 0 2024-04-26T04:18:53.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.938+0000 7fdd2d59a640 10 monclient: _check_auth_tickets 2024-04-26T04:18:53.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.938+0000 7fdd2d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:53.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.938+0000 7fdd2d59a640 1 -- 172.21.15.63:0/3035079922 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd180ac640 con 0x7fdd30137320 2024-04-26T04:18:53.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.938+0000 7fdd2d59a640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:53.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.938+0000 7fdd2d59a640 1 -- 172.21.15.63:0/3035079922 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd180aca10 con 0x7fdd30137320 2024-04-26T04:18:53.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.938+0000 7fdd2d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:53.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.938+0000 7fdd2d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T04:18:53.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.938+0000 7fdd2d59a640 10 monclient: dump: 2024-04-26T04:18:53.947 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:53.947 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:53.947 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:53.947 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:53.947 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:53.947 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:53.947 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:53.947 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:53.947 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:53.947 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:53.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.938+0000 7fdd34dcb640 5 monclient: authenticate success, global_id 4312 2024-04-26T04:18:53.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.938+0000 7fdd34dcb640 10 monclient: get_monmap_and_config success 2024-04-26T04:18:53.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.938+0000 7fdd34dcb640 4 set_mon_vals no callback set 2024-04-26T04:18:53.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.938+0000 7fdd34dcb640 10 monclient: shutdown 2024-04-26T04:18:53.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.938+0000 7fdd34dcb640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:53.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.938+0000 7fdd34dcb640 1 -- 172.21.15.63:0/3035079922 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd30137320 msgr2=0x7fdd30139710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:53.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.938+0000 7fdd34dcb640 1 --2- 172.21.15.63:0/3035079922 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd30137320 0x7fdd30139710 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fdd180097f0 tx=0x7fdd1801c100 comp rx=0 tx=0).stop 2024-04-26T04:18:53.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.938+0000 7fdd34dcb640 1 -- 172.21.15.63:0/3035079922 shutdown_connections 2024-04-26T04:18:53.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.938+0000 7fdd34dcb640 1 --2- 172.21.15.63:0/3035079922 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd30137320 0x7fdd30139710 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:53.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.938+0000 7fdd34dcb640 1 -- 172.21.15.63:0/3035079922 >> 172.21.15.63:0/3035079922 conn(0x7fdd300a1e70 msgr2=0x7fdd300a2280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:53.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.938+0000 7fdd34dcb640 1 -- 172.21.15.63:0/3035079922 shutdown_connections 2024-04-26T04:18:53.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.938+0000 7fdd34dcb640 1 -- 172.21.15.63:0/3035079922 wait complete. 2024-04-26T04:18:53.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.938+0000 7fdd34dcb640 10 monclient: build_initial_monmap 2024-04-26T04:18:53.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.938+0000 7fdd34dcb640 10 monclient: monmap: 2024-04-26T04:18:53.948 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:53.949 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:18:53.949 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:18:53.949 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:18:53.949 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:53.949 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:53.949 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:18:53.949 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:18:53.949 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:18:53.949 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:53.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.942+0000 7fdd34dcb640 1 Processor -- start 2024-04-26T04:18:53.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.942+0000 7fdd34dcb640 1 -- start start 2024-04-26T04:18:53.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.942+0000 7fdd34dcb640 10 monclient: init 2024-04-26T04:18:53.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.942+0000 7fdd34dcb640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:53.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.942+0000 7fdd34dcb640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T04:18:53.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.942+0000 7fdd34dcb640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdd3009b3b0 0x7fdd30136310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:53.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.942+0000 7fdd34dcb640 10 monclient(hunting): picked mon.noname-a con 0x7fdd3009b3b0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:18:53.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.942+0000 7fdd34dcb640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd30137320 0x7fdd30136850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:53.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.942+0000 7fdd34dcb640 10 monclient(hunting): picked mon.noname-b con 0x7fdd30137320 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:18:53.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.942+0000 7fdd34dcb640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fdd30139c50 0x7fdd3012a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:53.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.942+0000 7fdd34dcb640 10 monclient(hunting): picked mon.noname-c con 0x7fdd30139c50 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:53.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.942+0000 7fdd34dcb640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:53.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.942+0000 7fdd34dcb640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fdd30140340 con 0x7fdd3009b3b0 2024-04-26T04:18:53.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.942+0000 7fdd34dcb640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:53.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.942+0000 7fdd34dcb640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fdd301401c0 con 0x7fdd30137320 2024-04-26T04:18:53.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.942+0000 7fdd34dcb640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:53.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.942+0000 7fdd34dcb640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7fdd301404c0 con 0x7fdd30139c50 2024-04-26T04:18:53.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.942+0000 7fdd34dcb640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:53.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.942+0000 7fdd34dcb640 10 monclient(hunting): authenticate will time out at 965.652839s 2024-04-26T04:18:53.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.942+0000 7fdd2e59c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdd3009b3b0 0x7fdd30136310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:53.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.942+0000 7fdd2e59c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdd3009b3b0 0x7fdd30136310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:39424/0 (socket says 172.21.15.63:39424) 2024-04-26T04:18:53.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.942+0000 7fdd2e59c640 1 -- 172.21.15.63:0/869974079 learned_addr learned my addr 172.21.15.63:0/869974079 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:18:53.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.942+0000 7fdd2e59c640 10 monclient(hunting): get_auth_request con 0x7fdd3009b3b0 auth_method 0 2024-04-26T04:18:53.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.942+0000 7fdd2e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:53.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.942+0000 7fdd2e59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:53.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.942+0000 7fdd2e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:53.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.942+0000 7fdd2ed9d640 1 --2- 172.21.15.63:0/869974079 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fdd30139c50 0x7fdd3012a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:53.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.942+0000 7fdd2ed9d640 10 monclient(hunting): get_auth_request con 0x7fdd30139c50 auth_method 0 2024-04-26T04:18:53.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.942+0000 7fdd2ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:53.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.942+0000 7fdd2ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:53.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.942+0000 7fdd2ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:53.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.942+0000 7fdd2e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:53.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.942+0000 7fdd2e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:53.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.942+0000 7fdd2e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:53.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.942+0000 7fdd2ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:53.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.942+0000 7fdd2ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:53.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.942+0000 7fdd2ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:53.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.942+0000 7fdd2e59c640 10 monclient(hunting): handle_auth_done global_id 4362 payload 824 2024-04-26T04:18:53.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.942+0000 7fdd2e59c640 1 -- 172.21.15.63:0/869974079 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fdd30139c50 msgr2=0x7fdd3012a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:53.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.942+0000 7fdd2dd9b640 1 --2- 172.21.15.63:0/869974079 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd30137320 0x7fdd30136850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:53.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.942+0000 7fdd2e59c640 1 --2- 172.21.15.63:0/869974079 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fdd30139c50 0x7fdd3012a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:53.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.942+0000 7fdd2e59c640 1 -- 172.21.15.63:0/869974079 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd30137320 msgr2=0x7fdd30136850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:53.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.942+0000 7fdd2e59c640 1 --2- 172.21.15.63:0/869974079 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd30137320 0x7fdd30136850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:53.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.942+0000 7fdd2e59c640 10 monclient: _finish_hunting 0 2024-04-26T04:18:53.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.942+0000 7fdd2e59c640 1 monclient: found mon.noname-a 2024-04-26T04:18:53.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.942+0000 7fdd2e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:53.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.942+0000 7fdd2e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:18:53.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.942+0000 7fdd2e59c640 1 -- 172.21.15.63:0/869974079 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd300723d0 con 0x7fdd3009b3b0 2024-04-26T04:18:53.958 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.942+0000 7fdd2ed9d640 1 --2- 172.21.15.63:0/869974079 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fdd30139c50 0x7fdd3012a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T04:18:53.958 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.942+0000 7fdd2e59c640 1 -- 172.21.15.63:0/869974079 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdd3009b3b0 msgr2=0x7fdd30136310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:53.958 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.942+0000 7fdd2e59c640 1 --2- 172.21.15.63:0/869974079 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdd3009b3b0 0x7fdd30136310 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fdd2800cc10 tx=0x7fdd2800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:53.958 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.942+0000 7fdd26ffd640 1 -- 172.21.15.63:0/869974079 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd28012400 con 0x7fdd3009b3b0 2024-04-26T04:18:53.958 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.942+0000 7fdd26ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:53.958 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.942+0000 7fdd26ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:18:53.958 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.942+0000 7fdd26ffd640 10 monclient: dump: 2024-04-26T04:18:53.958 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:53.958 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:53.958 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:53.958 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:53.958 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:53.958 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:53.958 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:53.959 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:53.959 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:53.959 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:53.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.942+0000 7fdd26ffd640 10 monclient: _finish_auth 0 2024-04-26T04:18:53.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.942+0000 7fdd26ffd640 10 monclient: _check_auth_tickets 2024-04-26T04:18:53.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.942+0000 7fdd26ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:53.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.942+0000 7fdd26ffd640 1 -- 172.21.15.63:0/869974079 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd280125e0 con 0x7fdd3009b3b0 2024-04-26T04:18:53.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.942+0000 7fdd26ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:53.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.942+0000 7fdd26ffd640 1 -- 172.21.15.63:0/869974079 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd280129d0 con 0x7fdd3009b3b0 2024-04-26T04:18:53.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.942+0000 7fdd26ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:53.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.942+0000 7fdd26ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:18:53.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.942+0000 7fdd26ffd640 10 monclient: dump: 2024-04-26T04:18:53.959 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:53.959 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:53.960 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:53.960 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:53.960 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:53.960 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:53.960 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:53.960 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:53.960 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:53.960 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:53.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.942+0000 7fdd2cd99640 4 set_mon_vals no callback set 2024-04-26T04:18:53.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.946+0000 7fdd34dcb640 5 monclient: authenticate success, global_id 4362 2024-04-26T04:18:53.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.946+0000 7fdd34dcb640 10 monclient: _renew_subs 2024-04-26T04:18:53.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.946+0000 7fdd34dcb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:53.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.946+0000 7fdd34dcb640 1 -- 172.21.15.63:0/869974079 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd3012cd60 con 0x7fdd3009b3b0 2024-04-26T04:18:53.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.946+0000 7fdd34dcb640 10 monclient: _renew_subs 2024-04-26T04:18:53.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.946+0000 7fdd34dcb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:53.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.946+0000 7fdd34dcb640 1 -- 172.21.15.63:0/869974079 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd3012b560 con 0x7fdd3009b3b0 2024-04-26T04:18:53.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.946+0000 7fdd34dcb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:53.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.946+0000 7fdd34dcb640 1 --2- 172.21.15.63:0/869974079 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd301ccb20 0x7fdd3012a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:53.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.946+0000 7fdd34dcb640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:53.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.946+0000 7fdd34dcb640 1 -- 172.21.15.63:0/869974079 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fdd300504d0 con 0x7fdd301ccb20 2024-04-26T04:18:53.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.946+0000 7fdd26ffd640 1 -- 172.21.15.63:0/869974079 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fdd28012b70 con 0x7fdd3009b3b0 2024-04-26T04:18:53.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.946+0000 7fdd26ffd640 1 --2- 172.21.15.63:0/869974079 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fdd10035ec0 0x7fdd10038380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:53.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.946+0000 7fdd2dd9b640 1 --2- 172.21.15.63:0/869974079 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd301ccb20 0x7fdd3012a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:53.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.946+0000 7fdd2dd9b640 10 monclient: get_auth_request con 0x7fdd301ccb20 auth_method 0 2024-04-26T04:18:53.961 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.946+0000 7fdd2dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:53.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.946+0000 7fdd2dd9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:53.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.946+0000 7fdd2dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:53.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.946+0000 7fdd26ffd640 1 -- 172.21.15.63:0/869974079 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4738+0+0 (secure 0 0 0) 0x7fdd280495b0 con 0x7fdd3009b3b0 2024-04-26T04:18:53.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.946+0000 7fdd2dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:53.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.946+0000 7fdd2dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:53.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.946+0000 7fdd2dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:53.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.946+0000 7fdd2ed9d640 1 --2- 172.21.15.63:0/869974079 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fdd10035ec0 0x7fdd10038380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:53.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.946+0000 7fdd2ed9d640 10 monclient: get_auth_request con 0x7fdd10035ec0 auth_method 0 2024-04-26T04:18:53.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.946+0000 7fdd2dd9b640 10 monclient(hunting): handle_auth_done global_id 4315 payload 824 2024-04-26T04:18:53.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.946+0000 7fdd2dd9b640 1 -- 172.21.15.63:0/869974079 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdd30062e10 con 0x7fdd301ccb20 2024-04-26T04:18:53.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.950+0000 7fdd2dd9b640 1 -- 172.21.15.63:0/869974079 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd301ccb20 msgr2=0x7fdd3012a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:53.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.950+0000 7fdd2dd9b640 1 --2- 172.21.15.63:0/869974079 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd301ccb20 0x7fdd3012a7a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fdd180ae000 tx=0x7fdd1801a940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:53.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.950+0000 7fdd2ed9d640 1 --2- 172.21.15.63:0/869974079 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fdd10035ec0 0x7fdd10038380 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fdd3009ec60 tx=0x7fdd20007d10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:53.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.950+0000 7fdd26ffd640 1 -- 172.21.15.63:0/869974079 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd180bb020 con 0x7fdd301ccb20 2024-04-26T04:18:53.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.950+0000 7fdd26ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:53.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.950+0000 7fdd26ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T04:18:53.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.950+0000 7fdd26ffd640 10 monclient: dump: 2024-04-26T04:18:53.963 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:53.963 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:53.963 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:53.963 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:53.963 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:53.963 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:53.963 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:53.963 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:53.963 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:53.963 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:53.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.950+0000 7fdd26ffd640 1 -- 172.21.15.63:0/869974079 <== mon.1 v2:172.21.15.186:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fdd180b5600 con 0x7fdd301ccb20 2024-04-26T04:18:53.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.950+0000 7fdd26ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:53.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.950+0000 7fdd26ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:18:53.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.950+0000 7fdd26ffd640 1 -- 172.21.15.63:0/869974079 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd301ccb20 msgr2=0x7fdd3012a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:53.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.950+0000 7fdd26ffd640 1 --2- 172.21.15.63:0/869974079 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd301ccb20 0x7fdd3012a7a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fdd180ae000 tx=0x7fdd1801a940 comp rx=0 tx=0).stop 2024-04-26T04:18:53.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.950+0000 7fdd26ffd640 1 -- 172.21.15.63:0/869974079 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd301ccb20 msgr2=0x7fdd3012a7a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:18:53.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.950+0000 7fdd26ffd640 1 --2- 172.21.15.63:0/869974079 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd301ccb20 0x7fdd3012a7a0 secure :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fdd180ae000 tx=0x7fdd1801a940 comp rx=0 tx=0).stop 2024-04-26T04:18:53.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.954+0000 7fdd34dcb640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T04:18:53.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.954+0000 7fdd34dcb640 1 --2- 172.21.15.63:0/869974079 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd3012c700 0x7fdd3012ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:53.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.954+0000 7fdd34dcb640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:53.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.954+0000 7fdd34dcb640 1 -- 172.21.15.63:0/869974079 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fdd30068430 con 0x7fdd3012c700 2024-04-26T04:18:53.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.954+0000 7fdd2dd9b640 1 --2- 172.21.15.63:0/869974079 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd3012c700 0x7fdd3012ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:53.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.954+0000 7fdd2dd9b640 10 monclient: get_auth_request con 0x7fdd3012c700 auth_method 0 2024-04-26T04:18:53.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.954+0000 7fdd2dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:53.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.954+0000 7fdd2dd9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:53.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.954+0000 7fdd2dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:53.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.954+0000 7fdd2dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:53.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.954+0000 7fdd2dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:53.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.954+0000 7fdd2dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:53.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.958+0000 7fdd2dd9b640 10 monclient(hunting): handle_auth_done global_id 4318 payload 824 2024-04-26T04:18:53.967 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.958+0000 7fdd2dd9b640 1 -- 172.21.15.63:0/869974079 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fdd30136d90 con 0x7fdd3012c700 2024-04-26T04:18:53.967 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.958+0000 7fdd2dd9b640 1 -- 172.21.15.63:0/869974079 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd3012c700 msgr2=0x7fdd3012ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:53.967 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.958+0000 7fdd2dd9b640 1 --2- 172.21.15.63:0/869974079 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd3012c700 0x7fdd3012ace0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fdd18009740 tx=0x7fdd18018ea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:53.967 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.958+0000 7fdd26ffd640 1 -- 172.21.15.63:0/869974079 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd180bb030 con 0x7fdd3012c700 2024-04-26T04:18:53.967 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.958+0000 7fdd26ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:53.967 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.958+0000 7fdd26ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T04:18:53.967 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.958+0000 7fdd26ffd640 10 monclient: dump: 2024-04-26T04:18:53.967 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:53.967 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:53.967 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:53.967 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:53.967 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:53.967 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:53.967 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:53.968 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:53.968 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:53.968 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:53.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.958+0000 7fdd26ffd640 1 -- 172.21.15.63:0/869974079 <== mon.1 v2:172.21.15.186:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4890 (secure 0 0 0) 0x7fdd180c2220 con 0x7fdd3012c700 2024-04-26T04:18:53.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.958+0000 7fdd26ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T04:18:53.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.958+0000 7fdd26ffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:18:53.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.958+0000 7fdd26ffd640 1 -- 172.21.15.63:0/869974079 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd3012c700 msgr2=0x7fdd3012ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:53.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.958+0000 7fdd26ffd640 1 --2- 172.21.15.63:0/869974079 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd3012c700 0x7fdd3012ace0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fdd18009740 tx=0x7fdd18018ea0 comp rx=0 tx=0).stop 2024-04-26T04:18:53.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.958+0000 7fdd26ffd640 1 -- 172.21.15.63:0/869974079 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd3012c700 msgr2=0x7fdd3012ace0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:18:53.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.958+0000 7fdd26ffd640 1 --2- 172.21.15.63:0/869974079 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd3012c700 0x7fdd3012ace0 secure :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fdd18009740 tx=0x7fdd18018ea0 comp rx=0 tx=0).stop 2024-04-26T04:18:53.968 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-26T04:18:53.969 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b", 2024-04-26T04:18:53.969 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 1, 2024-04-26T04:18:53.969 INFO:teuthology.orchestra.run.smithi063.stdout: "state": "peon", 2024-04-26T04:18:53.969 INFO:teuthology.orchestra.run.smithi063.stdout: "election_epoch": 8, 2024-04-26T04:18:53.969 INFO:teuthology.orchestra.run.smithi063.stdout: "uptime": 25075, 2024-04-26T04:18:53.969 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum": [ 2024-04-26T04:18:53.969 INFO:teuthology.orchestra.run.smithi063.stdout: 0, 2024-04-26T04:18:53.969 INFO:teuthology.orchestra.run.smithi063.stdout: 1, 2024-04-26T04:18:53.969 INFO:teuthology.orchestra.run.smithi063.stdout: 2 2024-04-26T04:18:53.969 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:18:53.969 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_age": 19, 2024-04-26T04:18:53.969 INFO:teuthology.orchestra.run.smithi063.stdout: "features": { 2024-04-26T04:18:53.969 INFO:teuthology.orchestra.run.smithi063.stdout: "required_con": "2450521708007866372", 2024-04-26T04:18:53.969 INFO:teuthology.orchestra.run.smithi063.stdout: "required_mon": [ 2024-04-26T04:18:53.970 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:18:53.970 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:18:53.970 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:18:53.970 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:18:53.970 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:18:53.970 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:18:53.970 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:18:53.970 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:18:53.970 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:18:53.970 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:18:53.970 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:18:53.970 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:18:53.970 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_con": "4540701547738038271", 2024-04-26T04:18:53.970 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_mon": [ 2024-04-26T04:18:53.971 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:18:53.971 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:18:53.971 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:18:53.971 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:18:53.971 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:18:53.971 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:18:53.971 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:18:53.971 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:18:53.971 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:18:53.971 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:18:53.971 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:18:53.971 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:18:53.971 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:18:53.971 INFO:teuthology.orchestra.run.smithi063.stdout: "outside_quorum": [], 2024-04-26T04:18:53.972 INFO:teuthology.orchestra.run.smithi063.stdout: "extra_probe_peers": [], 2024-04-26T04:18:53.972 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_provider": [], 2024-04-26T04:18:53.972 INFO:teuthology.orchestra.run.smithi063.stdout: "monmap": { 2024-04-26T04:18:53.972 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 1, 2024-04-26T04:18:53.972 INFO:teuthology.orchestra.run.smithi063.stdout: "fsid": "cd1c6a0c-4e4b-4014-89c0-2e535479ef39", 2024-04-26T04:18:53.972 INFO:teuthology.orchestra.run.smithi063.stdout: "modified": "2024-04-26T04:18:03.702339Z", 2024-04-26T04:18:53.972 INFO:teuthology.orchestra.run.smithi063.stdout: "created": "2024-04-26T04:18:03.702339Z", 2024-04-26T04:18:53.972 INFO:teuthology.orchestra.run.smithi063.stdout: "min_mon_release": 19, 2024-04-26T04:18:53.972 INFO:teuthology.orchestra.run.smithi063.stdout: "min_mon_release_name": "squid", 2024-04-26T04:18:53.972 INFO:teuthology.orchestra.run.smithi063.stdout: "election_strategy": 1, 2024-04-26T04:18:53.972 INFO:teuthology.orchestra.run.smithi063.stdout: "disallowed_leaders: ": "", 2024-04-26T04:18:53.972 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": false, 2024-04-26T04:18:53.972 INFO:teuthology.orchestra.run.smithi063.stdout: "tiebreaker_mon": "", 2024-04-26T04:18:53.972 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_ranks: ": "", 2024-04-26T04:18:53.972 INFO:teuthology.orchestra.run.smithi063.stdout: "features": { 2024-04-26T04:18:53.973 INFO:teuthology.orchestra.run.smithi063.stdout: "persistent": [ 2024-04-26T04:18:53.973 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:18:53.973 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:18:53.973 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:18:53.973 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:18:53.973 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:18:53.973 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:18:53.973 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:18:53.973 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:18:53.973 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:18:53.973 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:18:53.973 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:18:53.973 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:18:53.973 INFO:teuthology.orchestra.run.smithi063.stdout: "optional": [] 2024-04-26T04:18:53.974 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:18:53.974 INFO:teuthology.orchestra.run.smithi063.stdout: "mons": [ 2024-04-26T04:18:53.974 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:18:53.974 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 0, 2024-04-26T04:18:53.974 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a", 2024-04-26T04:18:53.974 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:18:53.974 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:18:53.974 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:18:53.974 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:18:53.974 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:3300", 2024-04-26T04:18:53.974 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:18:53.974 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:18:53.974 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:18:53.974 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:18:53.975 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6789", 2024-04-26T04:18:53.975 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:18:53.975 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:18:53.975 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:18:53.975 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:18:53.975 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6789/0", 2024-04-26T04:18:53.975 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6789/0", 2024-04-26T04:18:53.975 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:18:53.975 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:18:53.975 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:18:53.975 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:18:53.975 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:18:53.975 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 1, 2024-04-26T04:18:53.975 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b", 2024-04-26T04:18:53.975 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:18:53.976 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:18:53.976 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:18:53.976 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:18:53.976 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:3300", 2024-04-26T04:18:53.976 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:18:53.976 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:18:53.976 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:18:53.976 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:18:53.976 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:6789", 2024-04-26T04:18:53.976 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:18:53.976 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:18:53.976 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:18:53.976 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:18:53.976 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:6789/0", 2024-04-26T04:18:53.977 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.186:6789/0", 2024-04-26T04:18:53.977 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:18:53.977 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:18:53.977 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:18:53.977 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:18:53.977 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:18:53.977 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 2, 2024-04-26T04:18:53.977 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "c", 2024-04-26T04:18:53.977 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:18:53.977 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:18:53.977 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:18:53.977 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:18:53.977 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:3301", 2024-04-26T04:18:53.977 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:18:53.978 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:18:53.978 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:18:53.978 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:18:53.978 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6790", 2024-04-26T04:18:53.978 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:18:53.978 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:18:53.978 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:18:53.978 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:18:53.978 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6790/0", 2024-04-26T04:18:53.978 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6790/0", 2024-04-26T04:18:53.978 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:18:53.978 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:18:53.978 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:18:53.978 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:18:53.979 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:18:53.979 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:18:53.979 INFO:teuthology.orchestra.run.smithi063.stdout: "feature_map": { 2024-04-26T04:18:53.979 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": [ 2024-04-26T04:18:53.979 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:18:53.979 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:18:53.979 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:18:53.979 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 1 2024-04-26T04:18:53.979 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:18:53.979 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:18:53.979 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": [ 2024-04-26T04:18:53.979 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:18:53.979 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:18:53.979 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:18:53.979 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 3 2024-04-26T04:18:53.980 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:18:53.980 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:18:53.980 INFO:teuthology.orchestra.run.smithi063.stdout: "client": [ 2024-04-26T04:18:53.980 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:18:53.980 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:18:53.980 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:18:53.980 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 6 2024-04-26T04:18:53.980 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:18:53.980 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:18:53.980 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": [ 2024-04-26T04:18:53.980 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:18:53.980 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:18:53.980 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:18:53.980 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 1 2024-04-26T04:18:53.981 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:18:53.981 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:18:53.981 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:18:53.981 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": false 2024-04-26T04:18:53.981 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-26T04:18:53.982 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.958+0000 7fdd34dcb640 1 -- 172.21.15.63:0/869974079 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fdd10035ec0 msgr2=0x7fdd10038380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:53.982 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.958+0000 7fdd34dcb640 1 --2- 172.21.15.63:0/869974079 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fdd10035ec0 0x7fdd10038380 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fdd3009ec60 tx=0x7fdd20007d10 comp rx=0 tx=0).stop 2024-04-26T04:18:53.982 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.958+0000 7fdd34dcb640 10 monclient: shutdown 2024-04-26T04:18:53.982 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.958+0000 7fdd34dcb640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:53.982 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.958+0000 7fdd34dcb640 1 -- 172.21.15.63:0/869974079 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdd3009b3b0 msgr2=0x7fdd30136310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:53.982 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.958+0000 7fdd34dcb640 1 --2- 172.21.15.63:0/869974079 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdd3009b3b0 0x7fdd30136310 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fdd2800cc10 tx=0x7fdd2800a430 comp rx=0 tx=0).stop 2024-04-26T04:18:53.982 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.958+0000 7fdd2ed9d640 1 -- 172.21.15.63:0/869974079 reap_dead start 2024-04-26T04:18:53.983 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.962+0000 7fdd2ed9d640 1 -- 172.21.15.63:0/869974079 reap_dead start 2024-04-26T04:18:53.983 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.962+0000 7fdd34dcb640 1 -- 172.21.15.63:0/869974079 shutdown_connections 2024-04-26T04:18:53.983 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.962+0000 7fdd34dcb640 1 -- 172.21.15.63:0/869974079 >> 172.21.15.63:0/869974079 conn(0x7fdd300a1e70 msgr2=0x7fdd3013c040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:53.983 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.962+0000 7fdd34dcb640 1 -- 172.21.15.63:0/869974079 shutdown_connections 2024-04-26T04:18:53.983 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:53.962+0000 7fdd34dcb640 1 -- 172.21.15.63:0/869974079 wait complete. 2024-04-26T04:18:53.983 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T04:18:53.983 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-04-26T04:18:53.983 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-04-26T04:18:53.984 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-04-26T04:18:53.984 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-04-26T04:18:53.984 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-04-26T04:18:53.984 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T04:18:53.994 INFO:tasks.ceph.mon.c:Stopped 2024-04-26T04:18:53.994 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-04-26T04:18:53.994 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T04:18:54.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.138+0000 7f753a9c4640 10 monclient: get_monmap_and_config 2024-04-26T04:18:54.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.138+0000 7f753a9c4640 10 monclient: build_initial_monmap 2024-04-26T04:18:54.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.138+0000 7f753a9c4640 10 monclient: monmap: 2024-04-26T04:18:54.145 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:54.145 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:54.145 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:54.144829+0000 2024-04-26T04:18:54.145 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:54.144829+0000 2024-04-26T04:18:54.145 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:54.145 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:54.145 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:18:54.145 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:18:54.145 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:18:54.145 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:18:54.145 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:18:54.145 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:54.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.142+0000 7f753a9c4640 1 Processor -- start 2024-04-26T04:18:54.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.142+0000 7f753a9c4640 1 -- start start 2024-04-26T04:18:54.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.142+0000 7f753a9c4640 10 monclient: init 2024-04-26T04:18:54.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.142+0000 7f753a9c4640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:54.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.142+0000 7f753a9c4640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-04-26T04:18:54.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.142+0000 7f753a9c4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f75340a2cd0 addr v1:172.21.15.63:6789/0 2024-04-26T04:18:54.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.142+0000 7f753a9c4640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f75340a30b0 0x7f753413c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:54.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.142+0000 7f753a9c4640 10 monclient(hunting): picked mon.noname-a con 0x7f75340a30b0 addr v2:172.21.15.63:3300/0 2024-04-26T04:18:54.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.142+0000 7f753a9c4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f753413c540 addr v1:172.21.15.186:6789/0 2024-04-26T04:18:54.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.142+0000 7f753a9c4640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75340a4ee0 con 0x7f75340a2cd0 2024-04-26T04:18:54.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.142+0000 7f753a9c4640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75340a0fb0 con 0x7f753413c540 2024-04-26T04:18:54.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.142+0000 7f753a9c4640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:54.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.142+0000 7f753a9c4640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7f75340a5380 con 0x7f75340a30b0 2024-04-26T04:18:54.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.142+0000 7f753a9c4640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:54.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.142+0000 7f753a9c4640 10 monclient(hunting): authenticate will time out at 965.852930s 2024-04-26T04:18:54.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.142+0000 7f75337fe640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f75340a30b0 0x7f753413c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:54.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.142+0000 7f7533fff640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f75340a2cd0 0x7f75340a47d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:32856/0 (socket says 172.21.15.63:32856) 2024-04-26T04:18:54.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.142+0000 7f7533fff640 1 -- 172.21.15.63:0/1324299993 learned_addr learned my addr 172.21.15.63:0/1324299993 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-04-26T04:18:54.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.142+0000 7f75337fe640 10 monclient(hunting): get_auth_request con 0x7f75340a30b0 auth_method 0 2024-04-26T04:18:54.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.142+0000 7f75337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:54.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.142+0000 7f75337fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:54.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.142+0000 7f75337fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:54.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.142+0000 7f75337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:54.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.142+0000 7f75337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:54.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.142+0000 7f75337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:54.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.142+0000 7f75337fe640 10 monclient(hunting): handle_auth_done global_id 4365 payload 274 2024-04-26T04:18:54.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.142+0000 7f75337fe640 1 -- 172.21.15.63:0/1324299993 >> v1:172.21.15.186:6789/0 conn(0x7f753413c540 legacy=0x7f753413e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:54.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.142+0000 7f75337fe640 1 -- 172.21.15.63:0/1324299993 >> v1:172.21.15.63:6789/0 conn(0x7f75340a2cd0 legacy=0x7f75340a47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:54.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.142+0000 7f75337fe640 10 monclient: _finish_hunting 0 2024-04-26T04:18:54.150 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.142+0000 7f75337fe640 1 monclient: found mon.noname-a 2024-04-26T04:18:54.150 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.142+0000 7f75337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:54.150 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.142+0000 7f75337fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:18:54.150 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.142+0000 7f75337fe640 1 -- 172.21.15.63:0/1324299993 --> v2:172.21.15.63:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75340a1290 con 0x7f75340a30b0 2024-04-26T04:18:54.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.146+0000 7f75337fe640 1 -- 172.21.15.63:0/1324299993 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f75340a30b0 msgr2=0x7f753413c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:54.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.146+0000 7f75337fe640 1 --2- 172.21.15.63:0/1324299993 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f75340a30b0 0x7f753413c000 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f75280097c0 tx=0x7f752801c370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6bdec2b62c4043f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:54.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.146+0000 7f7532ffd640 1 -- 172.21.15.63:0/1324299993 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75280b0070 con 0x7f75340a30b0 2024-04-26T04:18:54.153 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.146+0000 7f7532ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:54.153 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.146+0000 7f7532ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:18:54.153 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.146+0000 7f7532ffd640 10 monclient: dump: 2024-04-26T04:18:54.153 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:54.153 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:54.153 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:54.153 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:54.153 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:54.153 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:54.153 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:54.153 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:54.153 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:54.153 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:54.153 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.146+0000 7f7532ffd640 10 monclient: _finish_auth 0 2024-04-26T04:18:54.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.146+0000 7f7532ffd640 10 monclient: _check_auth_tickets 2024-04-26T04:18:54.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.146+0000 7f7532ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:54.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.146+0000 7f7532ffd640 1 -- 172.21.15.63:0/1324299993 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75280ab600 con 0x7f75340a30b0 2024-04-26T04:18:54.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.146+0000 7f7532ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:54.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.146+0000 7f7532ffd640 1 -- 172.21.15.63:0/1324299993 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75280ab9d0 con 0x7f75340a30b0 2024-04-26T04:18:54.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.146+0000 7f753a9c4640 5 monclient: authenticate success, global_id 4365 2024-04-26T04:18:54.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.146+0000 7f753a9c4640 10 monclient: get_monmap_and_config success 2024-04-26T04:18:54.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.146+0000 7f753a9c4640 4 set_mon_vals no callback set 2024-04-26T04:18:54.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.146+0000 7f7532ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:54.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.146+0000 7f7532ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:18:54.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.146+0000 7f7532ffd640 10 monclient: dump: 2024-04-26T04:18:54.154 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:54.154 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:54.154 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:54.155 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:54.155 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:54.155 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:54.155 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:54.155 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:54.155 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:54.155 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:54.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.146+0000 7f753a9c4640 10 monclient: shutdown 2024-04-26T04:18:54.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.146+0000 7f753a9c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:54.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.146+0000 7f753a9c4640 1 -- 172.21.15.63:0/1324299993 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f75340a30b0 msgr2=0x7f753413c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:54.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.146+0000 7f753a9c4640 1 --2- 172.21.15.63:0/1324299993 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f75340a30b0 0x7f753413c000 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f75280097c0 tx=0x7f752801c370 comp rx=0 tx=0).stop 2024-04-26T04:18:54.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.146+0000 7f753a9c4640 1 -- 172.21.15.63:0/1324299993 shutdown_connections 2024-04-26T04:18:54.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.146+0000 7f753a9c4640 1 --2- 172.21.15.63:0/1324299993 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f75340a30b0 0x7f753413c000 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:54.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.146+0000 7f753a9c4640 1 -- 172.21.15.63:0/1324299993 >> 172.21.15.63:0/1324299993 conn(0x7f75340a5f40 msgr2=0x7f75340a6320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:54.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.146+0000 7f753a9c4640 1 -- 172.21.15.63:0/1324299993 shutdown_connections 2024-04-26T04:18:54.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.146+0000 7f753a9c4640 1 -- 172.21.15.63:0/1324299993 wait complete. 2024-04-26T04:18:54.160 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.146+0000 7f753a9c4640 10 monclient: build_initial_monmap 2024-04-26T04:18:54.160 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.146+0000 7f753a9c4640 10 monclient: monmap: 2024-04-26T04:18:54.160 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:54.160 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:18:54.160 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:18:54.160 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:18:54.160 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:54.160 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:54.160 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:18:54.160 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:18:54.160 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:18:54.160 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:54.160 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.150+0000 7f753a9c4640 1 Processor -- start 2024-04-26T04:18:54.161 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.150+0000 7f753a9c4640 1 -- start start 2024-04-26T04:18:54.161 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.150+0000 7f753a9c4640 10 monclient: init 2024-04-26T04:18:54.161 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.150+0000 7f753a9c4640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:54.161 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.150+0000 7f753a9c4640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T04:18:54.161 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.150+0000 7f753a9c4640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f75340a2cd0 0x7f75341cf000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:54.161 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.150+0000 7f753a9c4640 10 monclient(hunting): picked mon.noname-a con 0x7f75340a2cd0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:18:54.161 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.150+0000 7f753a9c4640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f75340a30b0 0x7f75341cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:54.161 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.150+0000 7f753a9c4640 10 monclient(hunting): picked mon.noname-c con 0x7f75340a30b0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:54.161 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.150+0000 7f753a9c4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f753413c540 0x7f753406f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:54.161 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.150+0000 7f753a9c4640 10 monclient(hunting): picked mon.noname-b con 0x7f753413c540 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:18:54.161 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.150+0000 7f753a9c4640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:54.161 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.150+0000 7f753a9c4640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f7534142c00 con 0x7f75340a2cd0 2024-04-26T04:18:54.161 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.150+0000 7f753a9c4640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:54.161 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.150+0000 7f753a9c4640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7534142a80 con 0x7f753413c540 2024-04-26T04:18:54.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.150+0000 7f753a9c4640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:54.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.150+0000 7f753a9c4640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f7534142d80 con 0x7f75340a30b0 2024-04-26T04:18:54.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.150+0000 7f753a9c4640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:54.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.150+0000 7f753a9c4640 10 monclient(hunting): authenticate will time out at 965.860655s 2024-04-26T04:18:54.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.150+0000 7f75337fe640 1 -- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f75340a30b0 msgr2=0x7f75341cf7d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:3301/0 2024-04-26T04:18:54.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.150+0000 7f75337fe640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f75340a30b0 0x7f75341cf7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:18:54.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.150+0000 7f7533fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f75340a2cd0 0x7f75341cf000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:54.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.150+0000 7f7533fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f75340a2cd0 0x7f75341cf000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:39444/0 (socket says 172.21.15.63:39444) 2024-04-26T04:18:54.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.150+0000 7f7533fff640 1 -- 172.21.15.63:0/2443922135 learned_addr learned my addr 172.21.15.63:0/2443922135 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:18:54.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.150+0000 7f7533fff640 10 monclient(hunting): get_auth_request con 0x7f75340a2cd0 auth_method 0 2024-04-26T04:18:54.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.150+0000 7f7533fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:54.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.150+0000 7f7533fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:54.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.150+0000 7f7533fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:54.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.150+0000 7f7533fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:54.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.150+0000 7f7533fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:54.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.150+0000 7f7533fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:54.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.150+0000 7f7533fff640 10 monclient(hunting): handle_auth_done global_id 4368 payload 824 2024-04-26T04:18:54.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.150+0000 7f7533fff640 1 -- 172.21.15.63:0/2443922135 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f75340a30b0 msgr2=0x7f75341cf7d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T04:18:54.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.150+0000 7f7533fff640 1 --2- 172.21.15.63:0/2443922135 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f75340a30b0 0x7f75341cf7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:54.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.150+0000 7f7533fff640 1 -- 172.21.15.63:0/2443922135 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f753413c540 msgr2=0x7f753406f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:54.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.150+0000 7f7533fff640 1 --2- 172.21.15.63:0/2443922135 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f753413c540 0x7f753406f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:54.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.150+0000 7f7533fff640 10 monclient: _finish_hunting 0 2024-04-26T04:18:54.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.150+0000 7f7533fff640 1 monclient: found mon.noname-a 2024-04-26T04:18:54.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.150+0000 7f7533fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:54.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.150+0000 7f7533fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:18:54.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.150+0000 7f7533fff640 1 -- 172.21.15.63:0/2443922135 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f753413c130 con 0x7f75340a2cd0 2024-04-26T04:18:54.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.150+0000 7f7533fff640 1 -- 172.21.15.63:0/2443922135 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f75340a2cd0 msgr2=0x7f75341cf000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:54.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.150+0000 7f7533fff640 1 --2- 172.21.15.63:0/2443922135 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f75340a2cd0 0x7f75341cf000 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f752400c470 tx=0x7f752400c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:54.164 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.150+0000 7f7530ff9640 1 -- 172.21.15.63:0/2443922135 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7524007070 con 0x7f75340a2cd0 2024-04-26T04:18:54.164 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.150+0000 7f7530ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:54.164 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.150+0000 7f7530ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:18:54.164 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.150+0000 7f7530ff9640 10 monclient: dump: 2024-04-26T04:18:54.164 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:54.164 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:54.164 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:54.164 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:54.164 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:54.164 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:54.164 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:54.164 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:54.164 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:54.164 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:54.164 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.150+0000 7f7530ff9640 10 monclient: _finish_auth 0 2024-04-26T04:18:54.165 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.150+0000 7f7530ff9640 10 monclient: _check_auth_tickets 2024-04-26T04:18:54.165 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.150+0000 7f7530ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:54.165 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.150+0000 7f753a9c4640 5 monclient: authenticate success, global_id 4368 2024-04-26T04:18:54.165 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.150+0000 7f7530ff9640 1 -- 172.21.15.63:0/2443922135 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7524007210 con 0x7f75340a2cd0 2024-04-26T04:18:54.165 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.150+0000 7f7530ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:54.165 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.150+0000 7f7530ff9640 1 -- 172.21.15.63:0/2443922135 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7524007600 con 0x7f75340a2cd0 2024-04-26T04:18:54.165 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.150+0000 7f753a9c4640 10 monclient: _renew_subs 2024-04-26T04:18:54.165 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.150+0000 7f753a9c4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:54.165 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.150+0000 7f753a9c4640 1 -- 172.21.15.63:0/2443922135 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7534070cf0 con 0x7f75340a2cd0 2024-04-26T04:18:54.165 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.150+0000 7f7530ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:54.165 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.150+0000 7f7530ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:18:54.165 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.150+0000 7f7530ff9640 10 monclient: dump: 2024-04-26T04:18:54.165 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:54.165 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:54.165 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:54.166 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:54.166 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:54.166 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:54.166 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:54.166 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:54.166 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:54.166 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:54.166 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.150+0000 7f75327fc640 4 set_mon_vals no callback set 2024-04-26T04:18:54.166 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.150+0000 7f753a9c4640 10 monclient: _renew_subs 2024-04-26T04:18:54.166 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.150+0000 7f753a9c4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:54.166 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.150+0000 7f753a9c4640 1 -- 172.21.15.63:0/2443922135 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7534072740 con 0x7f75340a2cd0 2024-04-26T04:18:54.166 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.154+0000 7f753a9c4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:54.166 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.154+0000 7f753a9c4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:54.166 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.154+0000 7f753a9c4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:54.167 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.154+0000 7f753a9c4640 1 -- 172.21.15.63:0/2443922135 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7534062e10 con 0x7f75340a2cd0 2024-04-26T04:18:54.167 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.154+0000 7f7530ff9640 1 -- 172.21.15.63:0/2443922135 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f752401b400 con 0x7f75340a2cd0 2024-04-26T04:18:54.167 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.154+0000 7f7530ff9640 1 --2- 172.21.15.63:0/2443922135 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f7520035ec0 0x7f7520038380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:54.167 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.154+0000 7f7530ff9640 1 -- 172.21.15.63:0/2443922135 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4738+0+0 (secure 0 0 0) 0x7f7524048d50 con 0x7f75340a2cd0 2024-04-26T04:18:54.167 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.154+0000 7f75337fe640 1 --2- 172.21.15.63:0/2443922135 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f7520035ec0 0x7f7520038380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:54.167 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.154+0000 7f75337fe640 10 monclient: get_auth_request con 0x7f7520035ec0 auth_method 0 2024-04-26T04:18:54.167 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.154+0000 7f75337fe640 1 --2- 172.21.15.63:0/2443922135 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f7520035ec0 0x7f7520038380 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f75280ab000 tx=0x7f7528008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:54.167 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.158+0000 7f7530ff9640 1 -- 172.21.15.63:0/2443922135 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f7524047770 con 0x7f75340a2cd0 2024-04-26T04:18:54.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.158+0000 7f7530ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:54.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.158+0000 7f7530ff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:18:54.375 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.370+0000 7f753a9c4640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T04:18:54.375 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.370+0000 7f753a9c4640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T04:18:54.375 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.370+0000 7f753a9c4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:54.376 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.370+0000 7f753a9c4640 1 -- 172.21.15.63:0/2443922135 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f753413fe30 con 0x7f75340a2cd0 2024-04-26T04:18:54.376 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.370+0000 7f7530ff9640 1 -- 172.21.15.63:0/2443922135 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f752401a760 con 0x7f75340a2cd0 2024-04-26T04:18:54.376 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.370+0000 7f7530ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T04:18:54.376 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.370+0000 7f7530ff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:18:54.376 INFO:teuthology.orchestra.run.smithi063.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cd1c6a0c-4e4b-4014-89c0-2e535479ef39","modified":"2024-04-26T04:18:03.702339Z","created":"2024-04-26T04:18:03.702339Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3300","nonce":0},{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:3300","nonce":0},{"type":"v1","addr":"172.21.15.186:6789","nonce":0}]},"addr":"172.21.15.186:6789/0","public_addr":"172.21.15.186:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3301","nonce":0},{"type":"v1","addr":"172.21.15.63:6790","nonce":0}]},"addr":"172.21.15.63:6790/0","public_addr":"172.21.15.63:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T04:18:54.378 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.374+0000 7f753a9c4640 1 -- 172.21.15.63:0/2443922135 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f7520035ec0 msgr2=0x7f7520038380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:54.378 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.374+0000 7f753a9c4640 1 --2- 172.21.15.63:0/2443922135 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f7520035ec0 0x7f7520038380 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f75280ab000 tx=0x7f7528008040 comp rx=0 tx=0).stop 2024-04-26T04:18:54.378 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.374+0000 7f753a9c4640 10 monclient: shutdown 2024-04-26T04:18:54.379 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.374+0000 7f753a9c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:54.379 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.374+0000 7f753a9c4640 1 -- 172.21.15.63:0/2443922135 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f75340a2cd0 msgr2=0x7f75341cf000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:54.379 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.374+0000 7f753a9c4640 1 --2- 172.21.15.63:0/2443922135 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f75340a2cd0 0x7f75341cf000 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f752400c470 tx=0x7f752400c930 comp rx=0 tx=0).stop 2024-04-26T04:18:54.379 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.374+0000 7f753a9c4640 1 -- 172.21.15.63:0/2443922135 shutdown_connections 2024-04-26T04:18:54.379 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.374+0000 7f753a9c4640 1 --2- 172.21.15.63:0/2443922135 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f7520035ec0 0x7f7520038380 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:54.379 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.374+0000 7f753a9c4640 1 --2- 172.21.15.63:0/2443922135 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f753413c540 0x7f753406f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:54.379 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.374+0000 7f753a9c4640 1 --2- 172.21.15.63:0/2443922135 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f75340a30b0 0x7f75341cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:54.379 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.374+0000 7f753a9c4640 1 --2- 172.21.15.63:0/2443922135 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f75340a2cd0 0x7f75341cf000 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:54.379 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.374+0000 7f753a9c4640 1 -- 172.21.15.63:0/2443922135 >> 172.21.15.63:0/2443922135 conn(0x7f75340a5f40 msgr2=0x7f75340a1a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:54.379 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.374+0000 7f753a9c4640 1 -- 172.21.15.63:0/2443922135 shutdown_connections 2024-04-26T04:18:54.379 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:54.374+0000 7f753a9c4640 1 -- 172.21.15.63:0/2443922135 wait complete. 2024-04-26T04:18:57.395 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T04:18:57.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.522+0000 7f9b76005640 10 monclient: get_monmap_and_config 2024-04-26T04:18:57.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.522+0000 7f9b76005640 10 monclient: build_initial_monmap 2024-04-26T04:18:57.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.522+0000 7f9b76005640 10 monclient: monmap: 2024-04-26T04:18:57.527 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:57.527 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:57.527 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:57.527200+0000 2024-04-26T04:18:57.528 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:57.527200+0000 2024-04-26T04:18:57.528 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:57.528 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:57.528 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:18:57.528 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:18:57.528 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:18:57.528 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:18:57.528 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:18:57.528 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:57.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.522+0000 7f9b76005640 1 Processor -- start 2024-04-26T04:18:57.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.522+0000 7f9b76005640 1 -- start start 2024-04-26T04:18:57.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.522+0000 7f9b76005640 10 monclient: init 2024-04-26T04:18:57.531 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.522+0000 7f9b76005640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:57.531 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.522+0000 7f9b76005640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-04-26T04:18:57.531 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.522+0000 7f9b76005640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9b7009fe60 0x7f9b701372f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:57.531 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.522+0000 7f9b76005640 10 monclient(hunting): picked mon.noname-b con 0x7f9b7009fe60 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:57.531 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.522+0000 7f9b76005640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9b700a0240 addr v1:172.21.15.186:6789/0 2024-04-26T04:18:57.531 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.522+0000 7f9b76005640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f9b7013c100 0x7f9b7013e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:57.531 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.522+0000 7f9b76005640 10 monclient(hunting): picked mon.noname-a con 0x7f9b7013c100 addr v2:172.21.15.63:3300/0 2024-04-26T04:18:57.531 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.522+0000 7f9b76005640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b70137830 con 0x7f9b700a0240 2024-04-26T04:18:57.531 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.522+0000 7f9b76005640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:57.531 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.522+0000 7f9b76005640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7f9b700a3730 con 0x7f9b7013c100 2024-04-26T04:18:57.531 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.522+0000 7f9b76005640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:57.531 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.522+0000 7f9b76005640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f9b700a0c60 con 0x7f9b7009fe60 2024-04-26T04:18:57.531 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.522+0000 7f9b76005640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:57.531 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.522+0000 7f9b76005640 10 monclient(hunting): authenticate will time out at 969.235431s 2024-04-26T04:18:57.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.522+0000 7f9b6f7fe640 1 -- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9b7009fe60 msgr2=0x7f9b701372f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.63:3301/0 2024-04-26T04:18:57.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.522+0000 7f9b6f7fe640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9b7009fe60 0x7f9b701372f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:18:57.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.522+0000 7f9b6ffff640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f9b7013c100 0x7f9b7013e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:57.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.522+0000 7f9b6ffff640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f9b7013c100 0x7f9b7013e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:39452/0 (socket says 172.21.15.63:39452) 2024-04-26T04:18:57.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.522+0000 7f9b6ffff640 1 -- 172.21.15.63:0/2111479040 learned_addr learned my addr 172.21.15.63:0/2111479040 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:18:57.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.522+0000 7f9b6ffff640 10 monclient(hunting): get_auth_request con 0x7f9b7013c100 auth_method 0 2024-04-26T04:18:57.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.522+0000 7f9b6ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:57.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.522+0000 7f9b6ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:57.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.522+0000 7f9b6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:57.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.522+0000 7f9b6ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:57.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.522+0000 7f9b6ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:57.533 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.522+0000 7f9b6ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:57.533 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.526+0000 7f9b6ffff640 10 monclient(hunting): handle_auth_done global_id 4371 payload 274 2024-04-26T04:18:57.533 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.526+0000 7f9b6ffff640 1 -- 172.21.15.63:0/2111479040 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9b7009fe60 msgr2=0x7f9b701372f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T04:18:57.533 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.526+0000 7f9b6ffff640 1 --2- 172.21.15.63:0/2111479040 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9b7009fe60 0x7f9b701372f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:57.533 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.526+0000 7f9b6ffff640 1 -- 172.21.15.63:0/2111479040 >> v1:172.21.15.186:6789/0 conn(0x7f9b700a0240 legacy=0x7f9b700a2a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:18:57.533 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.526+0000 7f9b6ffff640 10 monclient: _finish_hunting 0 2024-04-26T04:18:57.533 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.526+0000 7f9b6ffff640 1 monclient: found mon.noname-a 2024-04-26T04:18:57.533 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.526+0000 7f9b6ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:57.533 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.526+0000 7f9b6ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:18:57.533 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.526+0000 7f9b6ffff640 1 -- 172.21.15.63:0/2111479040 --> v2:172.21.15.63:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b700a0f40 con 0x7f9b7013c100 2024-04-26T04:18:57.534 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.530+0000 7f9b6ffff640 1 -- 172.21.15.63:0/2111479040 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9b7013c100 msgr2=0x7f9b7013e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:57.534 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.530+0000 7f9b6ffff640 1 --2- 172.21.15.63:0/2111479040 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9b7013c100 0x7f9b7013e4f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9b68009e50 tx=0x7f9b6801c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e64950e215b9da6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:57.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.530+0000 7f9b6e7fc640 1 -- 172.21.15.63:0/2111479040 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b680a5740 con 0x7f9b7013c100 2024-04-26T04:18:57.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.530+0000 7f9b6e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:57.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.530+0000 7f9b6e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:18:57.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.530+0000 7f9b6e7fc640 10 monclient: dump: 2024-04-26T04:18:57.535 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:57.535 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:57.535 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:57.535 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:57.535 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:57.535 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:57.535 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:57.536 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:57.536 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:57.536 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:57.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.530+0000 7f9b6e7fc640 10 monclient: _finish_auth 0 2024-04-26T04:18:57.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.530+0000 7f9b6e7fc640 10 monclient: _check_auth_tickets 2024-04-26T04:18:57.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.530+0000 7f9b6e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:57.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.530+0000 7f9b6e7fc640 1 -- 172.21.15.63:0/2111479040 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b680a58e0 con 0x7f9b7013c100 2024-04-26T04:18:57.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.530+0000 7f9b6e7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:57.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.530+0000 7f9b6e7fc640 1 -- 172.21.15.63:0/2111479040 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b680a5cb0 con 0x7f9b7013c100 2024-04-26T04:18:57.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.530+0000 7f9b76005640 5 monclient: authenticate success, global_id 4371 2024-04-26T04:18:57.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.530+0000 7f9b76005640 10 monclient: get_monmap_and_config success 2024-04-26T04:18:57.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.530+0000 7f9b76005640 4 set_mon_vals no callback set 2024-04-26T04:18:57.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.530+0000 7f9b6e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:57.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.530+0000 7f9b6e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:18:57.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.530+0000 7f9b6e7fc640 10 monclient: dump: 2024-04-26T04:18:57.537 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:57.537 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:57.537 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:57.537 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:57.537 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:57.537 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:57.537 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:57.537 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:57.537 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:57.537 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:57.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.530+0000 7f9b76005640 10 monclient: shutdown 2024-04-26T04:18:57.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.530+0000 7f9b76005640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:57.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.530+0000 7f9b76005640 1 -- 172.21.15.63:0/2111479040 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9b7013c100 msgr2=0x7f9b7013e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:57.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.530+0000 7f9b76005640 1 --2- 172.21.15.63:0/2111479040 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9b7013c100 0x7f9b7013e4f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9b68009e50 tx=0x7f9b6801c8a0 comp rx=0 tx=0).stop 2024-04-26T04:18:57.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.530+0000 7f9b76005640 1 -- 172.21.15.63:0/2111479040 shutdown_connections 2024-04-26T04:18:57.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.530+0000 7f9b76005640 1 --2- 172.21.15.63:0/2111479040 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9b7013c100 0x7f9b7013e4f0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:57.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.530+0000 7f9b76005640 1 --2- 172.21.15.63:0/2111479040 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9b7009fe60 0x7f9b701372f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:57.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.530+0000 7f9b76005640 1 -- 172.21.15.63:0/2111479040 >> 172.21.15.63:0/2111479040 conn(0x7f9b700a5fb0 msgr2=0x7f9b700a63c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:57.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.530+0000 7f9b76005640 1 -- 172.21.15.63:0/2111479040 shutdown_connections 2024-04-26T04:18:57.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.530+0000 7f9b76005640 1 -- 172.21.15.63:0/2111479040 wait complete. 2024-04-26T04:18:57.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.530+0000 7f9b76005640 10 monclient: build_initial_monmap 2024-04-26T04:18:57.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.530+0000 7f9b76005640 10 monclient: monmap: 2024-04-26T04:18:57.543 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:18:57.543 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:18:57.543 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:18:57.544 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:18:57.544 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:18:57.544 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:57.544 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:18:57.544 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:18:57.544 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:18:57.544 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:57.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.530+0000 7f9b76005640 1 Processor -- start 2024-04-26T04:18:57.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.530+0000 7f9b76005640 1 -- start start 2024-04-26T04:18:57.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.530+0000 7f9b76005640 10 monclient: init 2024-04-26T04:18:57.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.530+0000 7f9b76005640 10 monclient: _reopen_session rank -1 2024-04-26T04:18:57.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.530+0000 7f9b76005640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T04:18:57.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.530+0000 7f9b76005640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9b7009fe60 0x7f9b700722e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:57.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.530+0000 7f9b76005640 10 monclient(hunting): picked mon.noname-b con 0x7f9b7009fe60 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:18:57.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.530+0000 7f9b76005640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9b700a0240 0x7f9b70072820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:57.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.530+0000 7f9b76005640 10 monclient(hunting): picked mon.noname-c con 0x7f9b700a0240 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:18:57.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.530+0000 7f9b76005640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9b7013c100 0x7f9b7006e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:57.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.530+0000 7f9b76005640 10 monclient(hunting): picked mon.noname-a con 0x7f9b7013c100 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:18:57.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.530+0000 7f9b76005640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:57.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.530+0000 7f9b76005640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f9b701416a0 con 0x7f9b7013c100 2024-04-26T04:18:57.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.530+0000 7f9b76005640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:57.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.530+0000 7f9b76005640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9b70141520 con 0x7f9b7009fe60 2024-04-26T04:18:57.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.530+0000 7f9b76005640 10 monclient(hunting): start opening mon connection 2024-04-26T04:18:57.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.530+0000 7f9b76005640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f9b70141820 con 0x7f9b700a0240 2024-04-26T04:18:57.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.530+0000 7f9b76005640 10 monclient(hunting): _renew_subs 2024-04-26T04:18:57.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.530+0000 7f9b76005640 10 monclient(hunting): authenticate will time out at 969.243926s 2024-04-26T04:18:57.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.530+0000 7f9b6effd640 1 -- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9b700a0240 msgr2=0x7f9b70072820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:3301/0 2024-04-26T04:18:57.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.530+0000 7f9b6effd640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9b700a0240 0x7f9b70072820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:18:57.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.534+0000 7f9b6ffff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9b7013c100 0x7f9b7006e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:57.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.534+0000 7f9b6ffff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9b7013c100 0x7f9b7006e320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:39464/0 (socket says 172.21.15.63:39464) 2024-04-26T04:18:57.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.534+0000 7f9b6ffff640 1 -- 172.21.15.63:0/3851785776 learned_addr learned my addr 172.21.15.63:0/3851785776 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:18:57.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.534+0000 7f9b6ffff640 10 monclient(hunting): get_auth_request con 0x7f9b7013c100 auth_method 0 2024-04-26T04:18:57.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.534+0000 7f9b6ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:57.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.534+0000 7f9b6ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:57.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.534+0000 7f9b6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:57.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.534+0000 7f9b6ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:18:57.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.534+0000 7f9b6ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:18:57.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.534+0000 7f9b6ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:18:57.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.534+0000 7f9b6f7fe640 1 --2- 172.21.15.63:0/3851785776 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9b7009fe60 0x7f9b700722e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:57.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.534+0000 7f9b6f7fe640 10 monclient(hunting): get_auth_request con 0x7f9b7009fe60 auth_method 0 2024-04-26T04:18:57.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.534+0000 7f9b6f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:18:57.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.534+0000 7f9b6f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:18:57.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.534+0000 7f9b6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:18:57.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.534+0000 7f9b6ffff640 10 monclient(hunting): handle_auth_done global_id 4374 payload 824 2024-04-26T04:18:57.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.534+0000 7f9b6ffff640 1 -- 172.21.15.63:0/3851785776 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9b700a0240 msgr2=0x7f9b70072820 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T04:18:57.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.534+0000 7f9b6ffff640 1 --2- 172.21.15.63:0/3851785776 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9b700a0240 0x7f9b70072820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:57.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.534+0000 7f9b6ffff640 1 -- 172.21.15.63:0/3851785776 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9b7009fe60 msgr2=0x7f9b700722e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:57.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.534+0000 7f9b6ffff640 1 --2- 172.21.15.63:0/3851785776 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9b7009fe60 0x7f9b700722e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:57.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.534+0000 7f9b6ffff640 10 monclient: _finish_hunting 0 2024-04-26T04:18:57.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.534+0000 7f9b6ffff640 1 monclient: found mon.noname-a 2024-04-26T04:18:57.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.534+0000 7f9b6ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:18:57.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.534+0000 7f9b6ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:18:57.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.534+0000 7f9b6ffff640 1 -- 172.21.15.63:0/3851785776 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b700a7820 con 0x7f9b7013c100 2024-04-26T04:18:57.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.534+0000 7f9b6ffff640 1 -- 172.21.15.63:0/3851785776 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9b7013c100 msgr2=0x7f9b7006e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:18:57.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.534+0000 7f9b6ffff640 1 --2- 172.21.15.63:0/3851785776 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9b7013c100 0x7f9b7006e320 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f9b680af040 tx=0x7f9b68019640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:57.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.534+0000 7f9b57fff640 1 -- 172.21.15.63:0/3851785776 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b680bc020 con 0x7f9b7013c100 2024-04-26T04:18:57.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.534+0000 7f9b57fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:57.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.534+0000 7f9b57fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:18:57.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.534+0000 7f9b57fff640 10 monclient: dump: 2024-04-26T04:18:57.548 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:57.548 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:57.548 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:57.548 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:57.548 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:57.549 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:57.549 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:57.549 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:57.549 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:57.549 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:57.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.534+0000 7f9b57fff640 10 monclient: _finish_auth 0 2024-04-26T04:18:57.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.534+0000 7f9b57fff640 10 monclient: _check_auth_tickets 2024-04-26T04:18:57.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.534+0000 7f9b57fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:18:57.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.534+0000 7f9b57fff640 1 -- 172.21.15.63:0/3851785776 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b680b8880 con 0x7f9b7013c100 2024-04-26T04:18:57.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.534+0000 7f9b76005640 5 monclient: authenticate success, global_id 4374 2024-04-26T04:18:57.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.534+0000 7f9b76005640 10 monclient: _renew_subs 2024-04-26T04:18:57.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.534+0000 7f9b76005640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:57.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.534+0000 7f9b76005640 1 -- 172.21.15.63:0/3851785776 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b7013b850 con 0x7f9b7013c100 2024-04-26T04:18:57.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.534+0000 7f9b57fff640 10 monclient: handle_config config(0 keys) 2024-04-26T04:18:57.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.534+0000 7f9b57fff640 1 -- 172.21.15.63:0/3851785776 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b680b8c30 con 0x7f9b7013c100 2024-04-26T04:18:57.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.534+0000 7f9b57fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:18:57.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.534+0000 7f9b57fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:18:57.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.534+0000 7f9b57fff640 10 monclient: dump: 2024-04-26T04:18:57.550 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:18:57.550 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:18:57.550 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:57.550 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:18:57.550 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:18:57.550 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:18:57.550 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:18:57.551 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:18:57.551 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:18:57.551 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:18:57.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.534+0000 7f9b76005640 10 monclient: _renew_subs 2024-04-26T04:18:57.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.534+0000 7f9b76005640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:57.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.534+0000 7f9b6d7fa640 4 set_mon_vals no callback set 2024-04-26T04:18:57.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.534+0000 7f9b76005640 1 -- 172.21.15.63:0/3851785776 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b7006f840 con 0x7f9b7013c100 2024-04-26T04:18:57.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.538+0000 7f9b76005640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:57.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.538+0000 7f9b76005640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:57.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.538+0000 7f9b76005640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:57.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.538+0000 7f9b76005640 1 -- 172.21.15.63:0/3851785776 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b70062e10 con 0x7f9b7013c100 2024-04-26T04:18:57.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.538+0000 7f9b57fff640 1 -- 172.21.15.63:0/3851785776 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f9b680bf020 con 0x7f9b7013c100 2024-04-26T04:18:57.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.538+0000 7f9b57fff640 1 --2- 172.21.15.63:0/3851785776 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f9b5c035ec0 0x7f9b5c038380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:18:57.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.538+0000 7f9b57fff640 1 -- 172.21.15.63:0/3851785776 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4738+0+0 (secure 0 0 0) 0x7f9b680e3fd0 con 0x7f9b7013c100 2024-04-26T04:18:57.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.538+0000 7f9b6f7fe640 1 --2- 172.21.15.63:0/3851785776 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f9b5c035ec0 0x7f9b5c038380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:18:57.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.538+0000 7f9b6f7fe640 10 monclient: get_auth_request con 0x7f9b5c035ec0 auth_method 0 2024-04-26T04:18:57.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.538+0000 7f9b6f7fe640 1 --2- 172.21.15.63:0/3851785776 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f9b5c035ec0 0x7f9b5c038380 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9b60006fd0 tx=0x7f9b60008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:18:57.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.542+0000 7f9b57fff640 1 -- 172.21.15.63:0/3851785776 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f9b680c6030 con 0x7f9b7013c100 2024-04-26T04:18:57.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.542+0000 7f9b57fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:18:57.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.542+0000 7f9b57fff640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:18:57.762 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.758+0000 7f9b76005640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T04:18:57.762 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.758+0000 7f9b76005640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T04:18:57.763 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.758+0000 7f9b76005640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:18:57.763 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.758+0000 7f9b76005640 1 -- 172.21.15.63:0/3851785776 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f9b70070880 con 0x7f9b7013c100 2024-04-26T04:18:57.763 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.758+0000 7f9b57fff640 1 -- 172.21.15.63:0/3851785776 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f9b680165e0 con 0x7f9b7013c100 2024-04-26T04:18:57.763 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.758+0000 7f9b57fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T04:18:57.763 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.758+0000 7f9b57fff640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:18:57.763 INFO:teuthology.orchestra.run.smithi063.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":23,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cd1c6a0c-4e4b-4014-89c0-2e535479ef39","modified":"2024-04-26T04:18:03.702339Z","created":"2024-04-26T04:18:03.702339Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3300","nonce":0},{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:3300","nonce":0},{"type":"v1","addr":"172.21.15.186:6789","nonce":0}]},"addr":"172.21.15.186:6789/0","public_addr":"172.21.15.186:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3301","nonce":0},{"type":"v1","addr":"172.21.15.63:6790","nonce":0}]},"addr":"172.21.15.63:6790/0","public_addr":"172.21.15.63:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T04:18:57.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.758+0000 7f9b76005640 1 -- 172.21.15.63:0/3851785776 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f9b5c035ec0 msgr2=0x7f9b5c038380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:57.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.758+0000 7f9b76005640 1 --2- 172.21.15.63:0/3851785776 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f9b5c035ec0 0x7f9b5c038380 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9b60006fd0 tx=0x7f9b60008040 comp rx=0 tx=0).stop 2024-04-26T04:18:57.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.758+0000 7f9b76005640 10 monclient: shutdown 2024-04-26T04:18:57.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.758+0000 7f9b76005640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:18:57.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.758+0000 7f9b76005640 1 -- 172.21.15.63:0/3851785776 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9b7013c100 msgr2=0x7f9b7006e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:18:57.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.758+0000 7f9b76005640 1 --2- 172.21.15.63:0/3851785776 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9b7013c100 0x7f9b7006e320 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f9b680af040 tx=0x7f9b68019640 comp rx=0 tx=0).stop 2024-04-26T04:18:57.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.758+0000 7f9b76005640 1 -- 172.21.15.63:0/3851785776 shutdown_connections 2024-04-26T04:18:57.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.758+0000 7f9b76005640 1 --2- 172.21.15.63:0/3851785776 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f9b5c035ec0 0x7f9b5c038380 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:57.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.758+0000 7f9b76005640 1 --2- 172.21.15.63:0/3851785776 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9b7013c100 0x7f9b7006e320 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:57.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.758+0000 7f9b76005640 1 --2- 172.21.15.63:0/3851785776 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9b700a0240 0x7f9b70072820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:57.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.758+0000 7f9b76005640 1 --2- 172.21.15.63:0/3851785776 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9b7009fe60 0x7f9b700722e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:18:57.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.758+0000 7f9b76005640 1 -- 172.21.15.63:0/3851785776 >> 172.21.15.63:0/3851785776 conn(0x7f9b700a5fb0 msgr2=0x7f9b700a7dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:18:57.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.758+0000 7f9b76005640 1 -- 172.21.15.63:0/3851785776 shutdown_connections 2024-04-26T04:18:57.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:18:57.758+0000 7f9b76005640 1 -- 172.21.15.63:0/3851785776 wait complete. 2024-04-26T04:19:00.781 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T04:19:00.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.902+0000 7f5dbf4a0640 10 monclient: get_monmap_and_config 2024-04-26T04:19:00.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.902+0000 7f5dbf4a0640 10 monclient: build_initial_monmap 2024-04-26T04:19:00.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.902+0000 7f5dbf4a0640 10 monclient: monmap: 2024-04-26T04:19:00.909 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:19:00.909 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:19:00.909 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:19:00.908457+0000 2024-04-26T04:19:00.909 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:19:00.908457+0000 2024-04-26T04:19:00.909 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:19:00.910 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:19:00.910 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:19:00.910 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:19:00.910 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:19:00.910 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:19:00.910 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:19:00.910 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:19:00.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.902+0000 7f5dbf4a0640 1 Processor -- start 2024-04-26T04:19:00.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.902+0000 7f5dbf4a0640 1 -- start start 2024-04-26T04:19:00.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.902+0000 7f5dbf4a0640 10 monclient: init 2024-04-26T04:19:00.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.906+0000 7f5dbf4a0640 10 monclient: _reopen_session rank -1 2024-04-26T04:19:00.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.906+0000 7f5dbf4a0640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-04-26T04:19:00.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.906+0000 7f5dbf4a0640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f5db809b3b0 0x7f5db809e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:19:00.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.906+0000 7f5dbf4a0640 10 monclient(hunting): picked mon.noname-c con 0x7f5db809b3b0 addr v2:172.21.15.186:3300/0 2024-04-26T04:19:00.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.906+0000 7f5dbf4a0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5db80a0bb0 addr v1:172.21.15.186:6789/0 2024-04-26T04:19:00.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.906+0000 7f5dbf4a0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5db80a16d0 addr v1:172.21.15.63:6789/0 2024-04-26T04:19:00.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.906+0000 7f5dbf4a0640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5db809dde0 con 0x7f5db80a16d0 2024-04-26T04:19:00.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.906+0000 7f5dbf4a0640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5db809c9b0 con 0x7f5db80a0bb0 2024-04-26T04:19:00.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.906+0000 7f5dbf4a0640 10 monclient(hunting): start opening mon connection 2024-04-26T04:19:00.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.906+0000 7f5dbf4a0640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f5db809c100 con 0x7f5db809b3b0 2024-04-26T04:19:00.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.906+0000 7f5dbf4a0640 10 monclient(hunting): _renew_subs 2024-04-26T04:19:00.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.906+0000 7f5dbf4a0640 10 monclient(hunting): authenticate will time out at 972.616140s 2024-04-26T04:19:00.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.906+0000 7f5dbda3d640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f5db80a16d0 0x7f5db813c350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:39026/0 (socket says 172.21.15.63:39026) 2024-04-26T04:19:00.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.906+0000 7f5dbda3d640 1 -- 172.21.15.63:0/454938924 learned_addr learned my addr 172.21.15.63:0/454938924 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-04-26T04:19:00.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.906+0000 7f5dbd23c640 1 --2- 172.21.15.63:0/454938924 >> v2:172.21.15.186:3300/0 conn(0x7f5db809b3b0 0x7f5db809e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:19:00.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.906+0000 7f5dbd23c640 10 monclient(hunting): get_auth_request con 0x7f5db809b3b0 auth_method 0 2024-04-26T04:19:00.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.906+0000 7f5dbd23c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:19:00.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.906+0000 7f5dbd23c640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:19:00.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.906+0000 7f5dbd23c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:19:00.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.906+0000 7f5dbd23c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:19:00.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.906+0000 7f5dbd23c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:19:00.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.906+0000 7f5dbd23c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:19:00.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.906+0000 7f5dbd23c640 10 monclient(hunting): handle_auth_done global_id 4324 payload 274 2024-04-26T04:19:00.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.906+0000 7f5dbd23c640 1 -- 172.21.15.63:0/454938924 >> v1:172.21.15.186:6789/0 conn(0x7f5db80a0bb0 legacy=0x7f5db80a0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:19:00.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.906+0000 7f5dbd23c640 1 -- 172.21.15.63:0/454938924 >> v1:172.21.15.63:6789/0 conn(0x7f5db80a16d0 legacy=0x7f5db813c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:19:00.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.906+0000 7f5dbd23c640 10 monclient: _finish_hunting 0 2024-04-26T04:19:00.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.906+0000 7f5dbd23c640 1 monclient: found mon.noname-c 2024-04-26T04:19:00.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.906+0000 7f5dbd23c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:19:00.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.906+0000 7f5dbd23c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.186:3300/0 2024-04-26T04:19:00.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.906+0000 7f5dbd23c640 1 -- 172.21.15.63:0/454938924 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5db809db00 con 0x7f5db809b3b0 2024-04-26T04:19:00.919 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.910+0000 7f5dbd23c640 1 -- 172.21.15.63:0/454938924 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5db809b3b0 msgr2=0x7f5db809e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:19:00.919 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.910+0000 7f5dbd23c640 1 --2- 172.21.15.63:0/454938924 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5db809b3b0 0x7f5db809e4e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5db00097c0 tx=0x7f5db001c100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd83b94ecfea6122 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:19:00.920 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.910+0000 7f5db7fff640 1 -- 172.21.15.63:0/454938924 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5db00b1070 con 0x7f5db809b3b0 2024-04-26T04:19:00.920 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.910+0000 7f5db7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:19:00.920 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.910+0000 7f5db7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:19:00.920 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.910+0000 7f5db7fff640 10 monclient: dump: 2024-04-26T04:19:00.920 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:19:00.920 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:19:00.920 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:19:00.920 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:19:00.920 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:19:00.920 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:19:00.921 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:19:00.921 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:19:00.921 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:19:00.921 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:19:00.921 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.910+0000 7f5db7fff640 10 monclient: _finish_auth 0 2024-04-26T04:19:00.921 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.910+0000 7f5db7fff640 10 monclient: _check_auth_tickets 2024-04-26T04:19:00.921 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.910+0000 7f5db7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:19:00.921 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.910+0000 7f5dbf4a0640 5 monclient: authenticate success, global_id 4324 2024-04-26T04:19:00.921 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.910+0000 7f5dbf4a0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T04:19:00.921 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.910+0000 7f5db7fff640 1 -- 172.21.15.63:0/454938924 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5db00ac440 con 0x7f5db809b3b0 2024-04-26T04:19:00.922 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.910+0000 7f5db7fff640 10 monclient: handle_config config(0 keys) 2024-04-26T04:19:00.922 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.910+0000 7f5dbf4a0640 10 monclient: get_monmap_and_config success 2024-04-26T04:19:00.922 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.910+0000 7f5dbf4a0640 4 set_mon_vals no callback set 2024-04-26T04:19:00.922 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.910+0000 7f5db7fff640 1 -- 172.21.15.63:0/454938924 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5db00ac810 con 0x7f5db809b3b0 2024-04-26T04:19:00.922 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.910+0000 7f5db7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:19:00.922 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.910+0000 7f5db7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T04:19:00.922 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.910+0000 7f5db7fff640 10 monclient: dump: 2024-04-26T04:19:00.922 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:19:00.922 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:19:00.923 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:19:00.923 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:19:00.923 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:19:00.923 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:19:00.923 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:19:00.923 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:19:00.923 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:19:00.923 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:19:00.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.910+0000 7f5dbf4a0640 10 monclient: shutdown 2024-04-26T04:19:00.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.910+0000 7f5dbf4a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:19:00.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.910+0000 7f5dbf4a0640 1 -- 172.21.15.63:0/454938924 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5db809b3b0 msgr2=0x7f5db809e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:19:00.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.910+0000 7f5dbf4a0640 1 --2- 172.21.15.63:0/454938924 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5db809b3b0 0x7f5db809e4e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5db00097c0 tx=0x7f5db001c100 comp rx=0 tx=0).stop 2024-04-26T04:19:00.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.910+0000 7f5dbf4a0640 1 -- 172.21.15.63:0/454938924 shutdown_connections 2024-04-26T04:19:00.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.910+0000 7f5dbf4a0640 1 --2- 172.21.15.63:0/454938924 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5db809b3b0 0x7f5db809e4e0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:19:00.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.910+0000 7f5dbf4a0640 1 -- 172.21.15.63:0/454938924 >> 172.21.15.63:0/454938924 conn(0x7f5db80a1e70 msgr2=0x7f5db80a2280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:19:00.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.910+0000 7f5dbf4a0640 1 -- 172.21.15.63:0/454938924 shutdown_connections 2024-04-26T04:19:00.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.910+0000 7f5dbf4a0640 1 -- 172.21.15.63:0/454938924 wait complete. 2024-04-26T04:19:00.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.910+0000 7f5dbf4a0640 10 monclient: build_initial_monmap 2024-04-26T04:19:00.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.910+0000 7f5dbf4a0640 10 monclient: monmap: 2024-04-26T04:19:00.924 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:19:00.924 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:19:00.924 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:19:00.924 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:19:00.924 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:19:00.924 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:19:00.925 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:19:00.925 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:19:00.925 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:19:00.925 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:19:00.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.910+0000 7f5dbf4a0640 1 Processor -- start 2024-04-26T04:19:00.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.910+0000 7f5dbf4a0640 1 -- start start 2024-04-26T04:19:00.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.910+0000 7f5dbf4a0640 10 monclient: init 2024-04-26T04:19:00.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.914+0000 7f5dbf4a0640 10 monclient: _reopen_session rank -1 2024-04-26T04:19:00.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.914+0000 7f5dbf4a0640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T04:19:00.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.914+0000 7f5dbf4a0640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5db809b3b0 0x7f5db81385a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:19:00.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.914+0000 7f5dbf4a0640 10 monclient(hunting): picked mon.noname-a con 0x7f5db809b3b0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:19:00.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.914+0000 7f5dbf4a0640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f5db80a0bb0 0x7f5db8138ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:19:00.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.914+0000 7f5dbf4a0640 10 monclient(hunting): picked mon.noname-c con 0x7f5db80a0bb0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:19:00.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.914+0000 7f5dbf4a0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5db80a16d0 0x7f5db808d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:19:00.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.914+0000 7f5dbf4a0640 10 monclient(hunting): picked mon.noname-b con 0x7f5db80a16d0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:19:00.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.914+0000 7f5dbf4a0640 10 monclient(hunting): start opening mon connection 2024-04-26T04:19:00.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.914+0000 7f5dbf4a0640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f5db8140a30 con 0x7f5db809b3b0 2024-04-26T04:19:00.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.914+0000 7f5dbf4a0640 10 monclient(hunting): start opening mon connection 2024-04-26T04:19:00.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.914+0000 7f5dbf4a0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5db81408b0 con 0x7f5db80a16d0 2024-04-26T04:19:00.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.914+0000 7f5dbf4a0640 10 monclient(hunting): start opening mon connection 2024-04-26T04:19:00.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.914+0000 7f5dbf4a0640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f5db8140bb0 con 0x7f5db80a0bb0 2024-04-26T04:19:00.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.914+0000 7f5dbf4a0640 10 monclient(hunting): _renew_subs 2024-04-26T04:19:00.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.914+0000 7f5dbf4a0640 10 monclient(hunting): authenticate will time out at 972.624502s 2024-04-26T04:19:00.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.914+0000 7f5dbca3b640 1 -- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f5db80a0bb0 msgr2=0x7f5db8138ae0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:3301/0 2024-04-26T04:19:00.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.914+0000 7f5dbca3b640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f5db80a0bb0 0x7f5db8138ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:19:00.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.914+0000 7f5dbd23c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5db809b3b0 0x7f5db81385a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:19:00.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.914+0000 7f5dbd23c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5db809b3b0 0x7f5db81385a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:39328/0 (socket says 172.21.15.63:39328) 2024-04-26T04:19:00.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.914+0000 7f5dbd23c640 1 -- 172.21.15.63:0/1486119925 learned_addr learned my addr 172.21.15.63:0/1486119925 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:19:00.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.914+0000 7f5dbd23c640 10 monclient(hunting): get_auth_request con 0x7f5db809b3b0 auth_method 0 2024-04-26T04:19:00.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.914+0000 7f5dbd23c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:19:00.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.914+0000 7f5dbd23c640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:19:00.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.914+0000 7f5dbd23c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:19:00.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.914+0000 7f5dbd23c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:19:00.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.914+0000 7f5dbd23c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:19:00.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.914+0000 7f5dbd23c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:19:00.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.914+0000 7f5dbd23c640 10 monclient(hunting): handle_auth_done global_id 4377 payload 824 2024-04-26T04:19:00.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.914+0000 7f5dbda3d640 1 --2- 172.21.15.63:0/1486119925 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5db80a16d0 0x7f5db808d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:19:00.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.914+0000 7f5dbd23c640 1 -- 172.21.15.63:0/1486119925 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f5db80a0bb0 msgr2=0x7f5db8138ae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T04:19:00.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.914+0000 7f5dbd23c640 1 --2- 172.21.15.63:0/1486119925 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f5db80a0bb0 0x7f5db8138ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:19:00.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.914+0000 7f5dbd23c640 1 -- 172.21.15.63:0/1486119925 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5db80a16d0 msgr2=0x7f5db808d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:19:00.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.914+0000 7f5dbd23c640 1 --2- 172.21.15.63:0/1486119925 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5db80a16d0 0x7f5db808d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:19:00.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.914+0000 7f5dbd23c640 10 monclient: _finish_hunting 0 2024-04-26T04:19:00.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.914+0000 7f5dbd23c640 1 monclient: found mon.noname-a 2024-04-26T04:19:00.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.914+0000 7f5dbd23c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:19:00.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.914+0000 7f5dbd23c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:19:00.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.914+0000 7f5dbd23c640 1 -- 172.21.15.63:0/1486119925 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5db80a4c50 con 0x7f5db809b3b0 2024-04-26T04:19:00.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.914+0000 7f5dbda3d640 10 monclient: get_auth_request con 0x7f5db80a16d0 auth_method 0 2024-04-26T04:19:00.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.914+0000 7f5dbda3d640 1 --2- 172.21.15.63:0/1486119925 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5db80a16d0 0x7f5db808d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T04:19:00.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.914+0000 7f5dbd23c640 1 -- 172.21.15.63:0/1486119925 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5db809b3b0 msgr2=0x7f5db81385a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:19:00.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.914+0000 7f5dbd23c640 1 --2- 172.21.15.63:0/1486119925 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5db809b3b0 0x7f5db81385a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f5db00097c0 tx=0x7f5db0018ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:19:00.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.914+0000 7f5db5ffb640 1 -- 172.21.15.63:0/1486119925 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5db00b9020 con 0x7f5db809b3b0 2024-04-26T04:19:00.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.914+0000 7f5db5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:19:00.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.914+0000 7f5db5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:19:00.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.914+0000 7f5db5ffb640 10 monclient: dump: 2024-04-26T04:19:00.933 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:19:00.933 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:19:00.933 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:19:00.934 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:19:00.934 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:19:00.934 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:19:00.934 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:19:00.934 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:19:00.934 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:19:00.934 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:19:00.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.914+0000 7f5db5ffb640 10 monclient: _finish_auth 0 2024-04-26T04:19:00.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.914+0000 7f5db5ffb640 10 monclient: _check_auth_tickets 2024-04-26T04:19:00.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.914+0000 7f5db5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:19:00.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.914+0000 7f5dbf4a0640 5 monclient: authenticate success, global_id 4377 2024-04-26T04:19:00.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.914+0000 7f5dbf4a0640 10 monclient: _renew_subs 2024-04-26T04:19:00.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.914+0000 7f5db5ffb640 1 -- 172.21.15.63:0/1486119925 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5db00b5900 con 0x7f5db809b3b0 2024-04-26T04:19:00.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.914+0000 7f5dbf4a0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:19:00.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.914+0000 7f5dbf4a0640 1 -- 172.21.15.63:0/1486119925 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5db8090750 con 0x7f5db809b3b0 2024-04-26T04:19:00.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.914+0000 7f5db5ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T04:19:00.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.914+0000 7f5db5ffb640 1 -- 172.21.15.63:0/1486119925 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5db00b1040 con 0x7f5db809b3b0 2024-04-26T04:19:00.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.914+0000 7f5db5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:19:00.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.914+0000 7f5db5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:19:00.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.914+0000 7f5db5ffb640 10 monclient: dump: 2024-04-26T04:19:00.935 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:19:00.935 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:19:00.935 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:19:00.936 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:19:00.936 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:19:00.936 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:19:00.936 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:19:00.936 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:19:00.936 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:19:00.936 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:19:00.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.914+0000 7f5dbf4a0640 10 monclient: _renew_subs 2024-04-26T04:19:00.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.914+0000 7f5dbf4a0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:19:00.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.914+0000 7f5dbf4a0640 1 -- 172.21.15.63:0/1486119925 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5db8090190 con 0x7f5db809b3b0 2024-04-26T04:19:00.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.914+0000 7f5db77fe640 4 set_mon_vals no callback set 2024-04-26T04:19:00.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.918+0000 7f5dbf4a0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:19:00.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.918+0000 7f5dbf4a0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:19:00.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.918+0000 7f5dbf4a0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:19:00.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.918+0000 7f5dbf4a0640 1 -- 172.21.15.63:0/1486119925 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5db8062e10 con 0x7f5db809b3b0 2024-04-26T04:19:00.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.918+0000 7f5db5ffb640 1 -- 172.21.15.63:0/1486119925 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f5db00bc020 con 0x7f5db809b3b0 2024-04-26T04:19:00.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.918+0000 7f5db5ffb640 1 --2- 172.21.15.63:0/1486119925 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f5da4035e70 0x7f5da4038330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:19:00.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.918+0000 7f5db5ffb640 1 -- 172.21.15.63:0/1486119925 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4738+0+0 (secure 0 0 0) 0x7f5db00e2940 con 0x7f5db809b3b0 2024-04-26T04:19:00.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.918+0000 7f5dbca3b640 1 --2- 172.21.15.63:0/1486119925 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f5da4035e70 0x7f5da4038330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:19:00.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.918+0000 7f5dbca3b640 10 monclient: get_auth_request con 0x7f5da4035e70 auth_method 0 2024-04-26T04:19:00.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.922+0000 7f5dbca3b640 1 --2- 172.21.15.63:0/1486119925 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f5da4035e70 0x7f5da4038330 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5dac007fd0 tx=0x7f5dac007da0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:19:00.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.922+0000 7f5db5ffb640 1 -- 172.21.15.63:0/1486119925 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f5db80a4c50 con 0x7f5db809b3b0 2024-04-26T04:19:00.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.922+0000 7f5db5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:19:00.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:00.922+0000 7f5db5ffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:19:01.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:01.138+0000 7f5dbf4a0640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T04:19:01.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:01.138+0000 7f5dbf4a0640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T04:19:01.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:01.138+0000 7f5dbf4a0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:19:01.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:01.138+0000 7f5dbf4a0640 1 -- 172.21.15.63:0/1486119925 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5db80903d0 con 0x7f5db809b3b0 2024-04-26T04:19:01.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:01.138+0000 7f5db5ffb640 1 -- 172.21.15.63:0/1486119925 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f5db809dde0 con 0x7f5db809b3b0 2024-04-26T04:19:01.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:01.138+0000 7f5db5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T04:19:01.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:01.138+0000 7f5db5ffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:19:01.146 INFO:teuthology.orchestra.run.smithi063.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":27,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cd1c6a0c-4e4b-4014-89c0-2e535479ef39","modified":"2024-04-26T04:18:03.702339Z","created":"2024-04-26T04:18:03.702339Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3300","nonce":0},{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:3300","nonce":0},{"type":"v1","addr":"172.21.15.186:6789","nonce":0}]},"addr":"172.21.15.186:6789/0","public_addr":"172.21.15.186:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3301","nonce":0},{"type":"v1","addr":"172.21.15.63:6790","nonce":0}]},"addr":"172.21.15.63:6790/0","public_addr":"172.21.15.63:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T04:19:01.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:01.142+0000 7f5dbf4a0640 1 -- 172.21.15.63:0/1486119925 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f5da4035e70 msgr2=0x7f5da4038330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:19:01.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:01.142+0000 7f5dbf4a0640 1 --2- 172.21.15.63:0/1486119925 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f5da4035e70 0x7f5da4038330 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5dac007fd0 tx=0x7f5dac007da0 comp rx=0 tx=0).stop 2024-04-26T04:19:01.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:01.142+0000 7f5dbf4a0640 10 monclient: shutdown 2024-04-26T04:19:01.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:01.142+0000 7f5dbf4a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:19:01.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:01.142+0000 7f5dbf4a0640 1 -- 172.21.15.63:0/1486119925 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5db809b3b0 msgr2=0x7f5db81385a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:19:01.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:01.142+0000 7f5dbf4a0640 1 --2- 172.21.15.63:0/1486119925 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5db809b3b0 0x7f5db81385a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f5db00097c0 tx=0x7f5db0018ea0 comp rx=0 tx=0).stop 2024-04-26T04:19:01.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:01.142+0000 7f5dbf4a0640 1 -- 172.21.15.63:0/1486119925 shutdown_connections 2024-04-26T04:19:01.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:01.142+0000 7f5dbf4a0640 1 --2- 172.21.15.63:0/1486119925 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f5da4035e70 0x7f5da4038330 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:19:01.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:01.142+0000 7f5dbf4a0640 1 --2- 172.21.15.63:0/1486119925 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5db80a16d0 0x7f5db808d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:19:01.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:01.142+0000 7f5dbf4a0640 1 --2- 172.21.15.63:0/1486119925 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f5db80a0bb0 0x7f5db8138ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:19:01.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:01.142+0000 7f5dbf4a0640 1 --2- 172.21.15.63:0/1486119925 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5db809b3b0 0x7f5db81385a0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:19:01.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:01.142+0000 7f5dbf4a0640 1 -- 172.21.15.63:0/1486119925 >> 172.21.15.63:0/1486119925 conn(0x7f5db80a1e70 msgr2=0x7f5db8070970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:19:01.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:01.142+0000 7f5dbf4a0640 1 -- 172.21.15.63:0/1486119925 shutdown_connections 2024-04-26T04:19:01.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:01.142+0000 7f5dbf4a0640 1 -- 172.21.15.63:0/1486119925 wait complete. 2024-04-26T04:19:04.164 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T04:19:04.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:04.290+0000 7fde13aa2640 10 monclient: get_monmap_and_config 2024-04-26T04:19:04.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:04.290+0000 7fde13aa2640 10 monclient: build_initial_monmap 2024-04-26T04:19:04.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:04.290+0000 7fde13aa2640 10 monclient: monmap: 2024-04-26T04:19:04.295 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:19:04.295 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:19:04.295 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:19:04.295025+0000 2024-04-26T04:19:04.295 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:19:04.295025+0000 2024-04-26T04:19:04.295 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:19:04.295 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:19:04.295 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:19:04.296 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:19:04.296 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:19:04.296 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:19:04.296 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:19:04.296 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:19:04.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:04.290+0000 7fde13aa2640 1 Processor -- start 2024-04-26T04:19:04.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:04.290+0000 7fde13aa2640 1 -- start start 2024-04-26T04:19:04.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:04.290+0000 7fde13aa2640 10 monclient: init 2024-04-26T04:19:04.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:04.290+0000 7fde13aa2640 10 monclient: _reopen_session rank -1 2024-04-26T04:19:04.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:04.290+0000 7fde13aa2640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-04-26T04:19:04.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:04.290+0000 7fde13aa2640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fde0c0a2cd0 0x7fde0c0a30b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:19:04.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:04.290+0000 7fde13aa2640 10 monclient(hunting): picked mon.noname-c con 0x7fde0c0a2cd0 addr v2:172.21.15.186:3300/0 2024-04-26T04:19:04.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:04.290+0000 7fde13aa2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fde0c0a47d0 addr v1:172.21.15.186:6789/0 2024-04-26T04:19:04.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:04.290+0000 7fde13aa2640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fde0c0a4c00 0x7fde0c1373f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:19:04.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:04.290+0000 7fde13aa2640 10 monclient(hunting): picked mon.noname-b con 0x7fde0c0a4c00 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:19:04.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:04.290+0000 7fde13aa2640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde0c0a0fb0 con 0x7fde0c0a47d0 2024-04-26T04:19:04.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:04.290+0000 7fde13aa2640 10 monclient(hunting): start opening mon connection 2024-04-26T04:19:04.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:04.290+0000 7fde13aa2640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fde0c0a35f0 con 0x7fde0c0a2cd0 2024-04-26T04:19:04.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:04.290+0000 7fde13aa2640 10 monclient(hunting): start opening mon connection 2024-04-26T04:19:04.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:04.290+0000 7fde13aa2640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7fde0c0a5380 con 0x7fde0c0a4c00 2024-04-26T04:19:04.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:04.290+0000 7fde13aa2640 10 monclient(hunting): _renew_subs 2024-04-26T04:19:04.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:04.290+0000 7fde13aa2640 10 monclient(hunting): authenticate will time out at 976.002763s 2024-04-26T04:19:04.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:04.290+0000 7fde1203f640 1 -- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fde0c0a4c00 msgr2=0x7fde0c1373f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.63:3301/0 2024-04-26T04:19:04.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:04.290+0000 7fde1203f640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fde0c0a4c00 0x7fde0c1373f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:19:04.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:04.290+0000 7fde1183e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fde0c0a2cd0 0x7fde0c0a30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:19:04.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:04.290+0000 7fde1183e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fde0c0a2cd0 0x7fde0c0a30b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.63:41538/0 (socket says 172.21.15.63:41538) 2024-04-26T04:19:04.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:04.290+0000 7fde1103d640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fde0c0a47d0 0x7fde0c136ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.63:49240/0 (socket says 172.21.15.63:49240) 2024-04-26T04:19:04.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:04.290+0000 7fde1183e640 1 -- 172.21.15.63:0/1754741848 learned_addr learned my addr 172.21.15.63:0/1754741848 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:19:04.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:04.290+0000 7fde1183e640 10 monclient(hunting): get_auth_request con 0x7fde0c0a2cd0 auth_method 0 2024-04-26T04:19:04.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:04.290+0000 7fde1183e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:19:04.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:04.290+0000 7fde1183e640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:19:04.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:04.290+0000 7fde1183e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:19:04.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:04.290+0000 7fde1183e640 1 -- 172.21.15.63:0/1754741848 >> v2:172.21.15.186:3300/0 conn(0x7fde0c0a2cd0 msgr2=0x7fde0c0a30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T04:19:04.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:04.290+0000 7fde1183e640 1 -- 172.21.15.63:0/1754741848 >> v2:172.21.15.186:3300/0 conn(0x7fde0c0a2cd0 msgr2=0x7fde0c0a30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:19:04.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:04.290+0000 7fde1183e640 1 --2- 172.21.15.63:0/1754741848 >> v2:172.21.15.186:3300/0 conn(0x7fde0c0a2cd0 0x7fde0c0a30b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:19:04.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:04.290+0000 7fde1183e640 1 --2- 172.21.15.63:0/1754741848 >> v2:172.21.15.186:3300/0 conn(0x7fde0c0a2cd0 0x7fde0c0a30b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:19:04.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:04.294+0000 7fde1103d640 1 -- 172.21.15.63:0/1754741848 >> v1:172.21.15.186:6789/0 conn(0x7fde0c0a47d0 legacy=0x7fde0c136ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:19:04.497 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:04.494+0000 7fde1203f640 1 -- 172.21.15.63:0/1754741848 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fde0c0a4c00 msgr2=0x7fde0c1373f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.63:3301/0 2024-04-26T04:19:04.498 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:04.494+0000 7fde1203f640 1 --2- 172.21.15.63:0/1754741848 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fde0c0a4c00 0x7fde0c1373f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T04:19:04.498 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:04.494+0000 7fde1183e640 1 --2- 172.21.15.63:0/1754741848 >> v2:172.21.15.186:3300/0 conn(0x7fde0c0a2cd0 0x7fde0c0a30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:19:04.499 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:04.494+0000 7fde1183e640 10 monclient(hunting): get_auth_request con 0x7fde0c0a2cd0 auth_method 0 2024-04-26T04:19:04.499 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:04.494+0000 7fde1183e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:19:04.499 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:04.494+0000 7fde1183e640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:19:04.499 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:04.494+0000 7fde1183e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:19:04.499 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:04.494+0000 7fde1183e640 1 -- 172.21.15.63:0/1754741848 >> v2:172.21.15.186:3300/0 conn(0x7fde0c0a2cd0 msgr2=0x7fde0c0a30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T04:19:04.499 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:04.494+0000 7fde1183e640 1 -- 172.21.15.63:0/1754741848 >> v2:172.21.15.186:3300/0 conn(0x7fde0c0a2cd0 msgr2=0x7fde0c0a30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:19:04.499 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:04.494+0000 7fde1183e640 1 --2- 172.21.15.63:0/1754741848 >> v2:172.21.15.186:3300/0 conn(0x7fde0c0a2cd0 0x7fde0c0a30b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:19:04.499 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:04.494+0000 7fde1183e640 1 --2- 172.21.15.63:0/1754741848 >> v2:172.21.15.186:3300/0 conn(0x7fde0c0a2cd0 0x7fde0c0a30b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T04:19:04.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:04.898+0000 7fde1203f640 1 -- 172.21.15.63:0/1754741848 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fde0c0a4c00 msgr2=0x7fde0c1373f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.63:3301/0 2024-04-26T04:19:04.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:04.898+0000 7fde1203f640 1 --2- 172.21.15.63:0/1754741848 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fde0c0a4c00 0x7fde0c1373f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T04:19:04.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:04.898+0000 7fde1183e640 1 --2- 172.21.15.63:0/1754741848 >> v2:172.21.15.186:3300/0 conn(0x7fde0c0a2cd0 0x7fde0c0a30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:19:04.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:04.898+0000 7fde1183e640 10 monclient(hunting): get_auth_request con 0x7fde0c0a2cd0 auth_method 0 2024-04-26T04:19:04.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:04.898+0000 7fde1183e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:19:04.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:04.898+0000 7fde1183e640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:19:04.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:04.898+0000 7fde1183e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:19:04.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:04.898+0000 7fde1183e640 1 -- 172.21.15.63:0/1754741848 >> v2:172.21.15.186:3300/0 conn(0x7fde0c0a2cd0 msgr2=0x7fde0c0a30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T04:19:04.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:04.898+0000 7fde1183e640 1 -- 172.21.15.63:0/1754741848 >> v2:172.21.15.186:3300/0 conn(0x7fde0c0a2cd0 msgr2=0x7fde0c0a30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:19:04.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:04.898+0000 7fde1183e640 1 --2- 172.21.15.63:0/1754741848 >> v2:172.21.15.186:3300/0 conn(0x7fde0c0a2cd0 0x7fde0c0a30b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:19:04.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:04.898+0000 7fde1183e640 1 --2- 172.21.15.63:0/1754741848 >> v2:172.21.15.186:3300/0 conn(0x7fde0c0a2cd0 0x7fde0c0a30b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T04:19:05.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:05.702+0000 7fde1203f640 1 -- 172.21.15.63:0/1754741848 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fde0c0a4c00 msgr2=0x7fde0c1373f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.63:3301/0 2024-04-26T04:19:05.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:05.702+0000 7fde1203f640 1 --2- 172.21.15.63:0/1754741848 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fde0c0a4c00 0x7fde0c1373f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T04:19:05.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:05.702+0000 7fde1183e640 1 --2- 172.21.15.63:0/1754741848 >> v2:172.21.15.186:3300/0 conn(0x7fde0c0a2cd0 0x7fde0c0a30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:19:05.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:05.702+0000 7fde1183e640 10 monclient(hunting): get_auth_request con 0x7fde0c0a2cd0 auth_method 0 2024-04-26T04:19:05.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:05.702+0000 7fde1183e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:19:05.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:05.702+0000 7fde1183e640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:19:05.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:05.702+0000 7fde1183e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:19:05.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:05.702+0000 7fde1183e640 1 -- 172.21.15.63:0/1754741848 >> v2:172.21.15.186:3300/0 conn(0x7fde0c0a2cd0 msgr2=0x7fde0c0a30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T04:19:05.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:05.702+0000 7fde1183e640 1 -- 172.21.15.63:0/1754741848 >> v2:172.21.15.186:3300/0 conn(0x7fde0c0a2cd0 msgr2=0x7fde0c0a30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:19:05.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:05.702+0000 7fde1183e640 1 --2- 172.21.15.63:0/1754741848 >> v2:172.21.15.186:3300/0 conn(0x7fde0c0a2cd0 0x7fde0c0a30b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:19:05.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:05.702+0000 7fde1183e640 1 --2- 172.21.15.63:0/1754741848 >> v2:172.21.15.186:3300/0 conn(0x7fde0c0a2cd0 0x7fde0c0a30b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T04:19:07.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:07.290+0000 7fde037fe640 10 monclient(hunting): tick 2024-04-26T04:19:07.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:07.290+0000 7fde037fe640 10 monclient(hunting): _check_auth_tickets 2024-04-26T04:19:07.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:07.290+0000 7fde037fe640 1 monclient(hunting): continuing hunt 2024-04-26T04:19:07.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:07.290+0000 7fde037fe640 10 monclient(hunting): _reopen_session rank -1 2024-04-26T04:19:07.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:07.290+0000 7fde037fe640 1 -- 172.21.15.63:0/1754741848 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fde0c0a4c00 msgr2=0x7fde0c1373f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T04:19:07.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:07.290+0000 7fde037fe640 1 --2- 172.21.15.63:0/1754741848 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fde0c0a4c00 0x7fde0c1373f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:19:07.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:07.290+0000 7fde037fe640 1 -- 172.21.15.63:0/1754741848 >> v2:172.21.15.186:3300/0 conn(0x7fde0c0a2cd0 msgr2=0x7fde0c0a30b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T04:19:07.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:07.290+0000 7fde037fe640 1 --2- 172.21.15.63:0/1754741848 >> v2:172.21.15.186:3300/0 conn(0x7fde0c0a2cd0 0x7fde0c0a30b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:19:07.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:07.290+0000 7fde037fe640 1 -- 172.21.15.63:0/1754741848 >> v1:172.21.15.186:6789/0 conn(0x7fde0c0a47d0 legacy=0x7fde0c136ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:19:07.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:07.290+0000 7fde037fe640 10 monclient: _add_conns ranks=[0,3] 2024-04-26T04:19:07.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:07.290+0000 7fde037fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fddf0003770 addr v1:172.21.15.63:6789/0 2024-04-26T04:19:07.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:07.290+0000 7fde037fe640 1 --2- 172.21.15.63:0/1754741848 >> v2:172.21.15.63:3300/0 conn(0x7fddf0007400 0x7fddf00098a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:19:07.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:07.290+0000 7fde037fe640 10 monclient(hunting): picked mon.noname-a con 0x7fddf0007400 addr v2:172.21.15.63:3300/0 2024-04-26T04:19:07.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:07.290+0000 7fde037fe640 1 -- 172.21.15.63:0/1754741848 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde0c0a1290 con 0x7fddf0003770 2024-04-26T04:19:07.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:07.290+0000 7fde037fe640 10 monclient(hunting): start opening mon connection 2024-04-26T04:19:07.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:07.290+0000 7fde037fe640 1 -- 172.21.15.63:0/1754741848 --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7fde0c0a35f0 con 0x7fddf0007400 2024-04-26T04:19:07.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:07.290+0000 7fde037fe640 10 monclient(hunting): _renew_subs 2024-04-26T04:19:07.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:07.290+0000 7fde1103d640 1 --2- 172.21.15.63:0/1754741848 >> v2:172.21.15.63:3300/0 conn(0x7fddf0007400 0x7fddf00098a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:19:07.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:07.290+0000 7fde1103d640 10 monclient(hunting): get_auth_request con 0x7fddf0007400 auth_method 0 2024-04-26T04:19:07.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:07.290+0000 7fde1103d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:19:07.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:07.290+0000 7fde1103d640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:19:07.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:07.290+0000 7fde1103d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:19:07.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:07.290+0000 7fde1103d640 1 -- 172.21.15.63:0/1754741848 >> v2:172.21.15.63:3300/0 conn(0x7fddf0007400 msgr2=0x7fddf00098a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T04:19:07.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:07.290+0000 7fde1103d640 1 -- 172.21.15.63:0/1754741848 >> v2:172.21.15.63:3300/0 conn(0x7fddf0007400 msgr2=0x7fddf00098a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:19:07.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:07.290+0000 7fde1103d640 1 --2- 172.21.15.63:0/1754741848 >> v2:172.21.15.63:3300/0 conn(0x7fddf0007400 0x7fddf00098a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:19:07.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:07.290+0000 7fde1103d640 1 --2- 172.21.15.63:0/1754741848 >> v2:172.21.15.63:3300/0 conn(0x7fddf0007400 0x7fddf00098a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:19:07.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:07.294+0000 7fde1183e640 1 -- 172.21.15.63:0/1754741848 >> v1:172.21.15.63:6789/0 conn(0x7fddf0003770 legacy=0x7fddf0005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:19:07.498 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:07.494+0000 7fde1103d640 1 --2- 172.21.15.63:0/1754741848 >> v2:172.21.15.63:3300/0 conn(0x7fddf0007400 0x7fddf00098a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:19:07.498 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:07.494+0000 7fde1103d640 10 monclient(hunting): get_auth_request con 0x7fddf0007400 auth_method 0 2024-04-26T04:19:07.499 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:07.494+0000 7fde1103d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:19:07.499 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:07.494+0000 7fde1103d640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:19:07.499 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:07.494+0000 7fde1103d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:19:07.499 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:07.494+0000 7fde1103d640 1 -- 172.21.15.63:0/1754741848 >> v2:172.21.15.63:3300/0 conn(0x7fddf0007400 msgr2=0x7fddf00098a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T04:19:07.499 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:07.494+0000 7fde1103d640 1 -- 172.21.15.63:0/1754741848 >> v2:172.21.15.63:3300/0 conn(0x7fddf0007400 msgr2=0x7fddf00098a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:19:07.499 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:07.494+0000 7fde1103d640 1 --2- 172.21.15.63:0/1754741848 >> v2:172.21.15.63:3300/0 conn(0x7fddf0007400 0x7fddf00098a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:19:07.499 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:07.494+0000 7fde1103d640 1 --2- 172.21.15.63:0/1754741848 >> v2:172.21.15.63:3300/0 conn(0x7fddf0007400 0x7fddf00098a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T04:19:07.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:07.898+0000 7fde1103d640 1 --2- 172.21.15.63:0/1754741848 >> v2:172.21.15.63:3300/0 conn(0x7fddf0007400 0x7fddf00098a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:19:07.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:07.898+0000 7fde1103d640 10 monclient(hunting): get_auth_request con 0x7fddf0007400 auth_method 0 2024-04-26T04:19:07.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:07.898+0000 7fde1103d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:19:07.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:07.898+0000 7fde1103d640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:19:07.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:07.898+0000 7fde1103d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:19:07.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:07.898+0000 7fde1103d640 1 -- 172.21.15.63:0/1754741848 >> v2:172.21.15.63:3300/0 conn(0x7fddf0007400 msgr2=0x7fddf00098a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T04:19:07.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:07.898+0000 7fde1103d640 1 -- 172.21.15.63:0/1754741848 >> v2:172.21.15.63:3300/0 conn(0x7fddf0007400 msgr2=0x7fddf00098a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:19:07.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:07.898+0000 7fde1103d640 1 --2- 172.21.15.63:0/1754741848 >> v2:172.21.15.63:3300/0 conn(0x7fddf0007400 0x7fddf00098a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:19:07.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:07.898+0000 7fde1103d640 1 --2- 172.21.15.63:0/1754741848 >> v2:172.21.15.63:3300/0 conn(0x7fddf0007400 0x7fddf00098a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T04:19:08.609 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.602+0000 7fde1083c640 1 -- 172.21.15.63:0/1754741848 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 200800018 0 0) 0x7fde0c0a1290 con 0x7fddf0003770 2024-04-26T04:19:08.609 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.602+0000 7fde1083c640 10 monclient(hunting): _negotiate 2024-04-26T04:19:08.609 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.602+0000 7fde1083c640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:19:08.609 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.602+0000 7fde1083c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:19:08.609 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.602+0000 7fde1083c640 10 monclient(hunting): my global_id is 4380 2024-04-26T04:19:08.609 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.602+0000 7fde1083c640 1 -- 172.21.15.63:0/1754741848 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fddf8003530 con 0x7fddf0003770 2024-04-26T04:19:08.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.626+0000 7fde1083c640 1 -- 172.21.15.63:0/1754741848 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2531714039 0 0) 0x7fddf8003530 con 0x7fddf0003770 2024-04-26T04:19:08.634 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.630+0000 7fde1083c640 1 -- 172.21.15.63:0/1754741848 >> v2:172.21.15.63:3300/0 conn(0x7fddf0007400 msgr2=0x7fddf00098a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T04:19:08.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.630+0000 7fde1083c640 1 --2- 172.21.15.63:0/1754741848 >> v2:172.21.15.63:3300/0 conn(0x7fddf0007400 0x7fddf00098a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:19:08.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.630+0000 7fde1083c640 10 monclient: _finish_hunting 0 2024-04-26T04:19:08.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.630+0000 7fde1083c640 1 monclient: found mon.noname-a-legacy 2024-04-26T04:19:08.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.630+0000 7fde1083c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:19:08.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.630+0000 7fde1083c640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.63:6789/0 2024-04-26T04:19:08.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.630+0000 7fde1083c640 1 -- 172.21.15.63:0/1754741848 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddf000afb0 con 0x7fddf0003770 2024-04-26T04:19:08.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.630+0000 7fde1083c640 10 monclient: _finish_auth 0 2024-04-26T04:19:08.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.630+0000 7fde1083c640 10 monclient: _check_auth_tickets 2024-04-26T04:19:08.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.630+0000 7fde1083c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:19:08.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.630+0000 7fde13aa2640 5 monclient: authenticate success, global_id 4380 2024-04-26T04:19:08.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.630+0000 7fde13aa2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T04:19:08.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.630+0000 7fde1083c640 1 -- 172.21.15.63:0/1754741848 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2612289768 0 0) 0x7fde0800f040 con 0x7fddf0003770 2024-04-26T04:19:08.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.630+0000 7fde1083c640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:19:08.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.630+0000 7fde1083c640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-26T04:19:08.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.630+0000 7fde1083c640 10 monclient: dump: 2024-04-26T04:19:08.638 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:19:08.638 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:19:08.638 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:19:08.638 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:19:08.638 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:19:08.638 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:19:08.638 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:19:08.638 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:19:08.639 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:19:08.639 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:19:08.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.630+0000 7fde1083c640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] but i'm connected to v1:172.21.15.63:6789/0, reconnecting 2024-04-26T04:19:08.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.630+0000 7fde1083c640 10 monclient: _reopen_session rank -1 2024-04-26T04:19:08.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.634+0000 7fde1083c640 1 -- 172.21.15.63:0/1754741848 >> v1:172.21.15.63:6789/0 conn(0x7fddf0003770 legacy=0x7fddf0005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:19:08.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.634+0000 7fde1203f640 1 -- 172.21.15.63:0/1754741848 reap_dead start 2024-04-26T04:19:08.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.634+0000 7fde1083c640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T04:19:08.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.634+0000 7fde1083c640 1 --2- 172.21.15.63:0/1754741848 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fddf8003aa0 0x7fddf8003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:19:08.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.634+0000 7fde1083c640 10 monclient(hunting): picked mon.c con 0x7fddf8003aa0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:19:08.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.634+0000 7fde1083c640 1 --2- 172.21.15.63:0/1754741848 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fddf8004810 0x7fddf8004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:19:08.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.634+0000 7fde1083c640 10 monclient(hunting): picked mon.b con 0x7fddf8004810 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:19:08.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.634+0000 7fde1183e640 1 -- 172.21.15.63:0/1754741848 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fddf8003aa0 msgr2=0x7fddf8003e80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.63:3301/0 2024-04-26T04:19:08.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.634+0000 7fde1083c640 1 --2- 172.21.15.63:0/1754741848 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fddf80052f0 0x7fddf800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:19:08.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.634+0000 7fde1183e640 1 --2- 172.21.15.63:0/1754741848 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fddf8003aa0 0x7fddf8003e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:19:08.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.634+0000 7fde1083c640 10 monclient(hunting): picked mon.a con 0x7fddf80052f0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:19:08.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.634+0000 7fde1103d640 1 --2- 172.21.15.63:0/1754741848 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fddf8004810 0x7fddf8004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:19:08.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.634+0000 7fde1203f640 1 --2- 172.21.15.63:0/1754741848 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fddf80052f0 0x7fddf800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:19:08.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.634+0000 7fde1083c640 10 monclient(hunting): start opening mon connection 2024-04-26T04:19:08.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.634+0000 7fde1083c640 1 -- 172.21.15.63:0/1754741848 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fde0c0a35f0 con 0x7fddf80052f0 2024-04-26T04:19:08.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.634+0000 7fde1083c640 10 monclient(hunting): start opening mon connection 2024-04-26T04:19:08.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.634+0000 7fde1083c640 1 -- 172.21.15.63:0/1754741848 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fddf800c1c0 con 0x7fddf8004810 2024-04-26T04:19:08.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.634+0000 7fde1083c640 10 monclient(hunting): start opening mon connection 2024-04-26T04:19:08.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.634+0000 7fde1083c640 1 -- 172.21.15.63:0/1754741848 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7fddf800c3c0 con 0x7fddf8003aa0 2024-04-26T04:19:08.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.634+0000 7fde1083c640 10 monclient(hunting): _renew_subs 2024-04-26T04:19:08.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.634+0000 7fde1083c640 10 monclient(hunting): _finish_auth 0 2024-04-26T04:19:08.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.634+0000 7fde13aa2640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-04-26T04:19:08.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.634+0000 7fde1203f640 10 monclient(hunting): get_auth_request con 0x7fddf80052f0 auth_method 0 2024-04-26T04:19:08.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.634+0000 7fde1203f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:19:08.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.634+0000 7fde1203f640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:19:08.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.634+0000 7fde1203f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:19:08.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.634+0000 7fde1103d640 10 monclient(hunting): get_auth_request con 0x7fddf8004810 auth_method 0 2024-04-26T04:19:08.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.634+0000 7fde1103d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:19:08.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.634+0000 7fde1103d640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:19:08.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.634+0000 7fde1103d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:19:08.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.634+0000 7fde1103d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:19:08.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.634+0000 7fde1103d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:19:08.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.634+0000 7fde1103d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T04:19:08.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.634+0000 7fde1203f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:19:08.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.634+0000 7fde1203f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:19:08.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.634+0000 7fde1203f640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T04:19:08.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.634+0000 7fde1103d640 10 monclient(hunting): handle_auth_done global_id 4380 payload 293 2024-04-26T04:19:08.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.634+0000 7fde1103d640 1 -- 172.21.15.63:0/1754741848 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fddf8003aa0 msgr2=0x7fddf8003e80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T04:19:08.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.634+0000 7fde1103d640 1 --2- 172.21.15.63:0/1754741848 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fddf8003aa0 0x7fddf8003e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:19:08.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.634+0000 7fde1103d640 1 -- 172.21.15.63:0/1754741848 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fddf80052f0 msgr2=0x7fddf800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:19:08.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.634+0000 7fde1103d640 1 --2- 172.21.15.63:0/1754741848 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fddf80052f0 0x7fddf800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:19:08.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.634+0000 7fde1103d640 10 monclient: _finish_hunting 0 2024-04-26T04:19:08.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.634+0000 7fde1103d640 1 monclient: found mon.b 2024-04-26T04:19:08.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.634+0000 7fde1103d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:19:08.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.634+0000 7fde1103d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-04-26T04:19:08.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.634+0000 7fde1103d640 1 -- 172.21.15.63:0/1754741848 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddf8003530 con 0x7fddf8004810 2024-04-26T04:19:08.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.634+0000 7fde1103d640 10 monclient: _finish_auth 0 2024-04-26T04:19:08.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.634+0000 7fde1103d640 10 monclient: _check_auth_tickets 2024-04-26T04:19:08.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.634+0000 7fde1103d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:19:08.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.634+0000 7fde1203f640 1 --2- 172.21.15.63:0/1754741848 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fddf80052f0 0x7fddf800b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T04:19:08.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.642+0000 7fde1103d640 1 -- 172.21.15.63:0/1754741848 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fddf8004810 msgr2=0x7fddf8004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:19:08.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.642+0000 7fde1103d640 1 --2- 172.21.15.63:0/1754741848 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fddf8004810 0x7fddf8004c90 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fddfc007b10 tx=0x7fddfc01cd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c8077212a189ce84 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:19:08.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.642+0000 7fde1083c640 1 -- 172.21.15.63:0/1754741848 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddfc0058c0 con 0x7fddf8004810 2024-04-26T04:19:08.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.642+0000 7fde1083c640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:19:08.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.642+0000 7fde1083c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T04:19:08.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.642+0000 7fde1083c640 10 monclient: dump: 2024-04-26T04:19:08.651 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:19:08.651 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:19:08.651 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:19:08.651 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:19:08.651 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:19:08.651 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:19:08.652 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:19:08.652 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:19:08.652 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:19:08.652 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:19:08.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.642+0000 7fde13aa2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T04:19:08.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.646+0000 7fde1083c640 1 -- 172.21.15.63:0/1754741848 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fddfc0a9070 con 0x7fddf8004810 2024-04-26T04:19:08.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.646+0000 7fde1083c640 10 monclient: handle_config config(0 keys) 2024-04-26T04:19:08.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.646+0000 7fde13aa2640 10 monclient: get_monmap_and_config success 2024-04-26T04:19:08.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.646+0000 7fde1083c640 1 -- 172.21.15.63:0/1754741848 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddfc0047f0 con 0x7fddf8004810 2024-04-26T04:19:08.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.646+0000 7fde13aa2640 4 set_mon_vals no callback set 2024-04-26T04:19:08.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.646+0000 7fde1083c640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:19:08.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.646+0000 7fde1083c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T04:19:08.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.646+0000 7fde1083c640 10 monclient: dump: 2024-04-26T04:19:08.653 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:19:08.653 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:19:08.653 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:19:08.654 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:19:08.654 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:19:08.654 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:19:08.654 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:19:08.654 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:19:08.654 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:19:08.654 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:19:08.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.646+0000 7fde13aa2640 10 monclient: shutdown 2024-04-26T04:19:08.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.646+0000 7fde13aa2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:19:08.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.646+0000 7fde13aa2640 1 -- 172.21.15.63:0/1754741848 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fddf8004810 msgr2=0x7fddf8004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:19:08.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.646+0000 7fde13aa2640 1 --2- 172.21.15.63:0/1754741848 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fddf8004810 0x7fddf8004c90 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fddfc007b10 tx=0x7fddfc01cd70 comp rx=0 tx=0).stop 2024-04-26T04:19:08.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.646+0000 7fde13aa2640 1 -- 172.21.15.63:0/1754741848 shutdown_connections 2024-04-26T04:19:08.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.646+0000 7fde13aa2640 1 --2- 172.21.15.63:0/1754741848 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fddf80052f0 0x7fddf800b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:19:08.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.646+0000 7fde13aa2640 1 --2- 172.21.15.63:0/1754741848 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fddf8004810 0x7fddf8004c90 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:19:08.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.646+0000 7fde13aa2640 1 --2- 172.21.15.63:0/1754741848 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fddf8003aa0 0x7fddf8003e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:19:08.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.646+0000 7fde13aa2640 1 -- 172.21.15.63:0/1754741848 >> 172.21.15.63:0/1754741848 conn(0x7fde0c0a5f40 msgr2=0x7fde0c0a6320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:19:08.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.646+0000 7fde13aa2640 1 -- 172.21.15.63:0/1754741848 shutdown_connections 2024-04-26T04:19:08.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.646+0000 7fde13aa2640 1 -- 172.21.15.63:0/1754741848 wait complete. 2024-04-26T04:19:08.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.646+0000 7fde13aa2640 10 monclient: build_initial_monmap 2024-04-26T04:19:08.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.646+0000 7fde13aa2640 10 monclient: monmap: 2024-04-26T04:19:08.664 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:19:08.664 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:19:08.664 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:19:08.664 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:19:08.664 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:19:08.664 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:19:08.664 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:19:08.664 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:19:08.665 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:19:08.665 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:19:08.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.646+0000 7fde13aa2640 1 Processor -- start 2024-04-26T04:19:08.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.646+0000 7fde13aa2640 1 -- start start 2024-04-26T04:19:08.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.646+0000 7fde13aa2640 10 monclient: init 2024-04-26T04:19:08.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.650+0000 7fde13aa2640 10 monclient: _reopen_session rank -1 2024-04-26T04:19:08.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.650+0000 7fde13aa2640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T04:19:08.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.650+0000 7fde13aa2640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fddf8003aa0 0x7fde0c11fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:19:08.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.650+0000 7fde13aa2640 10 monclient(hunting): picked mon.noname-a con 0x7fddf8003aa0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:19:08.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.650+0000 7fde13aa2640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fddf8004810 0x7fde0c11ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:19:08.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.650+0000 7fde13aa2640 10 monclient(hunting): picked mon.noname-c con 0x7fddf8004810 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:19:08.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.650+0000 7fde13aa2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fddf80052f0 0x7fde0c120510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:19:08.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.650+0000 7fde13aa2640 10 monclient(hunting): picked mon.noname-b con 0x7fddf80052f0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:19:08.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.650+0000 7fde13aa2640 10 monclient(hunting): start opening mon connection 2024-04-26T04:19:08.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.650+0000 7fde13aa2640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fde0c0a3230 con 0x7fddf8003aa0 2024-04-26T04:19:08.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.650+0000 7fde13aa2640 10 monclient(hunting): start opening mon connection 2024-04-26T04:19:08.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.650+0000 7fde13aa2640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fde0c0a30b0 con 0x7fddf80052f0 2024-04-26T04:19:08.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.650+0000 7fde13aa2640 10 monclient(hunting): start opening mon connection 2024-04-26T04:19:08.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.650+0000 7fde13aa2640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7fde0c0a33b0 con 0x7fddf8004810 2024-04-26T04:19:08.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.650+0000 7fde13aa2640 10 monclient(hunting): _renew_subs 2024-04-26T04:19:08.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.650+0000 7fde13aa2640 10 monclient(hunting): authenticate will time out at 980.360628s 2024-04-26T04:19:08.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.650+0000 7fde1103d640 1 -- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fddf8004810 msgr2=0x7fde0c11ffd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:3301/0 2024-04-26T04:19:08.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.650+0000 7fde1103d640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fddf8004810 0x7fde0c11ffd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:19:08.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.650+0000 7fde1183e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fddf8003aa0 0x7fde0c11fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:19:08.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.650+0000 7fde1183e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fddf8003aa0 0x7fde0c11fa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:39376/0 (socket says 172.21.15.63:39376) 2024-04-26T04:19:08.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.650+0000 7fde1183e640 1 -- 172.21.15.63:0/2742696054 learned_addr learned my addr 172.21.15.63:0/2742696054 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:19:08.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.650+0000 7fde1183e640 10 monclient(hunting): get_auth_request con 0x7fddf8003aa0 auth_method 0 2024-04-26T04:19:08.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.650+0000 7fde1183e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:19:08.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.650+0000 7fde1183e640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:19:08.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.650+0000 7fde1183e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:19:08.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.650+0000 7fde1183e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:19:08.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.650+0000 7fde1183e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:19:08.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.650+0000 7fde1203f640 1 --2- 172.21.15.63:0/2742696054 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fddf80052f0 0x7fde0c120510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:19:08.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.650+0000 7fde1183e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:19:08.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.650+0000 7fde1203f640 10 monclient(hunting): get_auth_request con 0x7fddf80052f0 auth_method 0 2024-04-26T04:19:08.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.650+0000 7fde1203f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:19:08.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.650+0000 7fde1203f640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:19:08.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.650+0000 7fde1203f640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:19:08.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.650+0000 7fde1203f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:19:08.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.650+0000 7fde1203f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:19:08.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.650+0000 7fde1203f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:19:08.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.650+0000 7fde1183e640 10 monclient(hunting): handle_auth_done global_id 4383 payload 824 2024-04-26T04:19:08.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.650+0000 7fde1183e640 1 -- 172.21.15.63:0/2742696054 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fddf8004810 msgr2=0x7fde0c11ffd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T04:19:08.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.650+0000 7fde1183e640 1 --2- 172.21.15.63:0/2742696054 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fddf8004810 0x7fde0c11ffd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:19:08.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.650+0000 7fde1183e640 1 -- 172.21.15.63:0/2742696054 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fddf80052f0 msgr2=0x7fde0c120510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:19:08.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.650+0000 7fde1183e640 1 --2- 172.21.15.63:0/2742696054 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fddf80052f0 0x7fde0c120510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:19:08.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.650+0000 7fde1183e640 10 monclient: _finish_hunting 0 2024-04-26T04:19:08.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.650+0000 7fde1183e640 1 monclient: found mon.noname-a 2024-04-26T04:19:08.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.650+0000 7fde1183e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:19:08.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.650+0000 7fde1183e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:19:08.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.650+0000 7fde1183e640 1 -- 172.21.15.63:0/2742696054 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde0c122d50 con 0x7fddf8003aa0 2024-04-26T04:19:08.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.650+0000 7fde1183e640 1 -- 172.21.15.63:0/2742696054 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fddf8003aa0 msgr2=0x7fde0c11fa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:19:08.672 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.650+0000 7fde1183e640 1 --2- 172.21.15.63:0/2742696054 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fddf8003aa0 0x7fde0c11fa90 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fde0800c300 tx=0x7fde08015850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:19:08.672 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.650+0000 7fde027fc640 1 -- 172.21.15.63:0/2742696054 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde0801d070 con 0x7fddf8003aa0 2024-04-26T04:19:08.672 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.650+0000 7fde027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:19:08.672 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.650+0000 7fde027fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:19:08.672 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.650+0000 7fde027fc640 10 monclient: dump: 2024-04-26T04:19:08.672 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:19:08.672 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:19:08.672 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:19:08.672 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:19:08.673 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:19:08.673 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:19:08.673 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:19:08.673 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:19:08.674 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:19:08.674 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:19:08.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.650+0000 7fde027fc640 10 monclient: _finish_auth 0 2024-04-26T04:19:08.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.650+0000 7fde027fc640 10 monclient: _check_auth_tickets 2024-04-26T04:19:08.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.650+0000 7fde027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:19:08.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.650+0000 7fde027fc640 1 -- 172.21.15.63:0/2742696054 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde08018950 con 0x7fddf8003aa0 2024-04-26T04:19:08.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.650+0000 7fde027fc640 10 monclient: handle_config config(0 keys) 2024-04-26T04:19:08.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.650+0000 7fde027fc640 1 -- 172.21.15.63:0/2742696054 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde08018d40 con 0x7fddf8003aa0 2024-04-26T04:19:08.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.650+0000 7fde027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:19:08.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.650+0000 7fde027fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:19:08.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.650+0000 7fde027fc640 10 monclient: dump: 2024-04-26T04:19:08.675 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:19:08.675 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:19:08.675 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:19:08.675 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:19:08.675 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:19:08.675 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:19:08.675 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:19:08.675 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:19:08.676 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:19:08.676 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:19:08.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.650+0000 7fde13aa2640 5 monclient: authenticate success, global_id 4383 2024-04-26T04:19:08.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.650+0000 7fde13aa2640 10 monclient: _renew_subs 2024-04-26T04:19:08.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.650+0000 7fde13aa2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:19:08.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.650+0000 7fde13aa2640 1 -- 172.21.15.63:0/2742696054 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde0c123030 con 0x7fddf8003aa0 2024-04-26T04:19:08.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.650+0000 7fde03fff640 4 set_mon_vals no callback set 2024-04-26T04:19:08.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.650+0000 7fde13aa2640 10 monclient: _renew_subs 2024-04-26T04:19:08.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.650+0000 7fde13aa2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:19:08.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.650+0000 7fde13aa2640 1 -- 172.21.15.63:0/2742696054 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde0c0a37d0 con 0x7fddf8003aa0 2024-04-26T04:19:08.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.654+0000 7fde027fc640 1 -- 172.21.15.63:0/2742696054 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fde08020070 con 0x7fddf8003aa0 2024-04-26T04:19:08.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.654+0000 7fde13aa2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:19:08.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.654+0000 7fde13aa2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:19:08.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.654+0000 7fde13aa2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:19:08.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.654+0000 7fde13aa2640 1 -- 172.21.15.63:0/2742696054 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde0c062e10 con 0x7fddf8003aa0 2024-04-26T04:19:08.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.654+0000 7fde027fc640 1 --2- 172.21.15.63:0/2742696054 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fddf0036770 0x7fddf0038c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:19:08.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.654+0000 7fde027fc640 1 -- 172.21.15.63:0/2742696054 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4738+0+0 (secure 0 0 0) 0x7fde0804bd50 con 0x7fddf8003aa0 2024-04-26T04:19:08.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.654+0000 7fde1103d640 1 --2- 172.21.15.63:0/2742696054 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fddf0036770 0x7fddf0038c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:19:08.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.654+0000 7fde1103d640 10 monclient: get_auth_request con 0x7fddf0036770 auth_method 0 2024-04-26T04:19:08.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.658+0000 7fde1103d640 1 --2- 172.21.15.63:0/2742696054 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fddf0036770 0x7fddf0038c30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fddf8004790 tx=0x7fddfc0b2040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:19:08.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.658+0000 7fde027fc640 1 -- 172.21.15.63:0/2742696054 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fde080213b0 con 0x7fddf8003aa0 2024-04-26T04:19:08.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.658+0000 7fde027fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:19:08.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.658+0000 7fde027fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:19:08.812 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[27717]: starting. 2024-04-26T04:19:08.813 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[27717]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-27716 2024-04-26T04:19:08.813 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[27717]: created object 0... 2024-04-26T04:19:08.813 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[27717]: created object 25... 2024-04-26T04:19:08.813 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[27717]: created object 49... 2024-04-26T04:19:08.813 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[27717]: finishing. 2024-04-26T04:19:08.813 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[27717]: shutting down. 2024-04-26T04:19:08.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.902+0000 7fde13aa2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T04:19:08.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.902+0000 7fde13aa2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T04:19:08.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.902+0000 7fde13aa2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:19:08.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.902+0000 7fde13aa2640 1 -- 172.21.15.63:0/2742696054 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fde0c0683d0 con 0x7fddf8003aa0 2024-04-26T04:19:08.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.902+0000 7fde027fc640 1 -- 172.21.15.63:0/2742696054 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1554 (secure 0 0 0) 0x7fde0804a770 con 0x7fddf8003aa0 2024-04-26T04:19:08.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.902+0000 7fde027fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T04:19:08.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.902+0000 7fde027fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:19:08.910 INFO:teuthology.orchestra.run.smithi063.stdout:{"election_epoch":12,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cd1c6a0c-4e4b-4014-89c0-2e535479ef39","modified":"2024-04-26T04:18:03.702339Z","created":"2024-04-26T04:18:03.702339Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3300","nonce":0},{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:3300","nonce":0},{"type":"v1","addr":"172.21.15.186:6789","nonce":0}]},"addr":"172.21.15.186:6789/0","public_addr":"172.21.15.186:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3301","nonce":0},{"type":"v1","addr":"172.21.15.63:6790","nonce":0}]},"addr":"172.21.15.63:6790/0","public_addr":"172.21.15.63:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T04:19:08.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.906+0000 7fde13aa2640 1 -- 172.21.15.63:0/2742696054 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fddf0036770 msgr2=0x7fddf0038c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:19:08.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.906+0000 7fde13aa2640 1 --2- 172.21.15.63:0/2742696054 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fddf0036770 0x7fddf0038c30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fddf8004790 tx=0x7fddfc0b2040 comp rx=0 tx=0).stop 2024-04-26T04:19:08.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.906+0000 7fde13aa2640 10 monclient: shutdown 2024-04-26T04:19:08.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.906+0000 7fde13aa2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:19:08.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.906+0000 7fde13aa2640 1 -- 172.21.15.63:0/2742696054 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fddf8003aa0 msgr2=0x7fde0c11fa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:19:08.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.906+0000 7fde13aa2640 1 --2- 172.21.15.63:0/2742696054 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fddf8003aa0 0x7fde0c11fa90 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fde0800c300 tx=0x7fde08015850 comp rx=0 tx=0).stop 2024-04-26T04:19:08.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.906+0000 7fde13aa2640 1 -- 172.21.15.63:0/2742696054 shutdown_connections 2024-04-26T04:19:08.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.906+0000 7fde13aa2640 1 --2- 172.21.15.63:0/2742696054 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fddf0036770 0x7fddf0038c30 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:19:08.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.906+0000 7fde13aa2640 1 --2- 172.21.15.63:0/2742696054 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fddf80052f0 0x7fde0c120510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:19:08.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.906+0000 7fde13aa2640 1 --2- 172.21.15.63:0/2742696054 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fddf8004810 0x7fde0c11ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:19:08.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.906+0000 7fde13aa2640 1 --2- 172.21.15.63:0/2742696054 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fddf8003aa0 0x7fde0c11fa90 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:19:08.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.906+0000 7fde13aa2640 1 -- 172.21.15.63:0/2742696054 >> 172.21.15.63:0/2742696054 conn(0x7fde0c0a5f40 msgr2=0x7fde0c135600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:19:08.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.906+0000 7fde13aa2640 1 -- 172.21.15.63:0/2742696054 shutdown_connections 2024-04-26T04:19:08.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:08.906+0000 7fde13aa2640 1 -- 172.21.15.63:0/2742696054 wait complete. 2024-04-26T04:19:08.931 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-04-26T04:19:08.931 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-04-26T04:19:08.932 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T04:19:09.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.054+0000 7fb27f680640 10 monclient: get_monmap_and_config 2024-04-26T04:19:09.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.054+0000 7fb27f680640 10 monclient: build_initial_monmap 2024-04-26T04:19:09.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.054+0000 7fb27f680640 10 monclient: monmap: 2024-04-26T04:19:09.058 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:19:09.058 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:19:09.058 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:19:09.057991+0000 2024-04-26T04:19:09.058 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:19:09.057991+0000 2024-04-26T04:19:09.058 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:19:09.058 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:19:09.059 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:19:09.059 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:19:09.059 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:19:09.059 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:19:09.059 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:19:09.059 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:19:09.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.054+0000 7fb27f680640 1 Processor -- start 2024-04-26T04:19:09.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.054+0000 7fb27f680640 1 -- start start 2024-04-26T04:19:09.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.054+0000 7fb27f680640 10 monclient: init 2024-04-26T04:19:09.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.054+0000 7fb27f680640 10 monclient: _reopen_session rank -1 2024-04-26T04:19:09.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.054+0000 7fb27f680640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-04-26T04:19:09.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.054+0000 7fb27f680640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb2780a2c60 0x7fb2780a3040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:19:09.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.054+0000 7fb27f680640 10 monclient(hunting): picked mon.noname-c con 0x7fb2780a2c60 addr v2:172.21.15.186:3300/0 2024-04-26T04:19:09.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.054+0000 7fb27f680640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fb2780a20c0 0x7fb2780a3db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:19:09.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.054+0000 7fb27f680640 10 monclient(hunting): picked mon.noname-b con 0x7fb2780a20c0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:19:09.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.054+0000 7fb27f680640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb2780a24a0 addr v1:172.21.15.186:6789/0 2024-04-26T04:19:09.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.054+0000 7fb27f680640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2780a0df0 con 0x7fb2780a24a0 2024-04-26T04:19:09.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.054+0000 7fb27f680640 10 monclient(hunting): start opening mon connection 2024-04-26T04:19:09.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.054+0000 7fb27f680640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fb2780a50a0 con 0x7fb2780a2c60 2024-04-26T04:19:09.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.054+0000 7fb27f680640 10 monclient(hunting): start opening mon connection 2024-04-26T04:19:09.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.054+0000 7fb27f680640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7fb2780a13b0 con 0x7fb2780a20c0 2024-04-26T04:19:09.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.054+0000 7fb27f680640 10 monclient(hunting): _renew_subs 2024-04-26T04:19:09.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.054+0000 7fb27f680640 10 monclient(hunting): authenticate will time out at 980.765918s 2024-04-26T04:19:09.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.054+0000 7fb27cc1b640 1 -- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fb2780a20c0 msgr2=0x7fb2780a3db0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.63:3301/0 2024-04-26T04:19:09.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.054+0000 7fb27cc1b640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fb2780a20c0 0x7fb2780a3db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:19:09.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.054+0000 7fb27d41c640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb2780a2c60 0x7fb2780a3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:19:09.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.054+0000 7fb27dc1d640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fb2780a24a0 0x7fb2780a42f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.63:49252/0 (socket says 172.21.15.63:49252) 2024-04-26T04:19:09.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.054+0000 7fb27dc1d640 1 -- 172.21.15.63:0/4049984581 learned_addr learned my addr 172.21.15.63:0/4049984581 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-04-26T04:19:09.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.054+0000 7fb27d41c640 10 monclient(hunting): get_auth_request con 0x7fb2780a2c60 auth_method 0 2024-04-26T04:19:09.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.054+0000 7fb27d41c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:19:09.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.054+0000 7fb27d41c640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:19:09.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.054+0000 7fb27d41c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:19:09.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.054+0000 7fb27d41c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:19:09.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.054+0000 7fb27d41c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:19:09.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.054+0000 7fb27d41c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:19:09.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.054+0000 7fb27d41c640 10 monclient(hunting): handle_auth_done global_id 4330 payload 274 2024-04-26T04:19:09.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.054+0000 7fb27d41c640 1 -- 172.21.15.63:0/4049984581 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fb2780a20c0 msgr2=0x7fb2780a3db0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T04:19:09.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.054+0000 7fb27d41c640 1 --2- 172.21.15.63:0/4049984581 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fb2780a20c0 0x7fb2780a3db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:19:09.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.054+0000 7fb27d41c640 1 -- 172.21.15.63:0/4049984581 >> v1:172.21.15.186:6789/0 conn(0x7fb2780a24a0 legacy=0x7fb2780a42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:19:09.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.054+0000 7fb27d41c640 10 monclient: _finish_hunting 0 2024-04-26T04:19:09.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.054+0000 7fb27d41c640 1 monclient: found mon.noname-c 2024-04-26T04:19:09.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.054+0000 7fb27d41c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:19:09.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.054+0000 7fb27d41c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.186:3300/0 2024-04-26T04:19:09.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.054+0000 7fb27d41c640 1 -- 172.21.15.63:0/4049984581 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2780a10d0 con 0x7fb2780a2c60 2024-04-26T04:19:09.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.058+0000 7fb27d41c640 1 -- 172.21.15.63:0/4049984581 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb2780a2c60 msgr2=0x7fb2780a3040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:19:09.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.058+0000 7fb27d41c640 1 --2- 172.21.15.63:0/4049984581 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb2780a2c60 0x7fb2780a3040 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb26c009d90 tx=0x7fb26c01c810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=72db136e4e46e780 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:19:09.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.058+0000 7fb26bfff640 1 -- 172.21.15.63:0/4049984581 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb26c0b2070 con 0x7fb2780a2c60 2024-04-26T04:19:09.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.058+0000 7fb26bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:19:09.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb26bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:19:09.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb26bfff640 10 monclient: dump: 2024-04-26T04:19:09.067 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:19:09.067 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:19:09.067 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:19:09.067 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:19:09.067 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:19:09.067 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:19:09.068 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:19:09.068 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:19:09.068 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:19:09.068 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:19:09.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb26bfff640 10 monclient: _finish_auth 0 2024-04-26T04:19:09.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb26bfff640 10 monclient: _check_auth_tickets 2024-04-26T04:19:09.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb26bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:19:09.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb27f680640 5 monclient: authenticate success, global_id 4330 2024-04-26T04:19:09.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb26bfff640 1 -- 172.21.15.63:0/4049984581 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb26c0a5990 con 0x7fb2780a2c60 2024-04-26T04:19:09.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb26bfff640 10 monclient: handle_config config(0 keys) 2024-04-26T04:19:09.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb27f680640 10 monclient: get_monmap_and_config success 2024-04-26T04:19:09.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb26bfff640 1 -- 172.21.15.63:0/4049984581 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb26c0a5d60 con 0x7fb2780a2c60 2024-04-26T04:19:09.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb26bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:19:09.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb27f680640 4 set_mon_vals no callback set 2024-04-26T04:19:09.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb26bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T04:19:09.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb26bfff640 10 monclient: dump: 2024-04-26T04:19:09.069 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:19:09.069 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:19:09.069 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:19:09.069 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:19:09.069 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:19:09.070 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:19:09.070 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:19:09.070 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:19:09.070 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:19:09.070 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:19:09.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb27f680640 10 monclient: shutdown 2024-04-26T04:19:09.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb27f680640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:19:09.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb27f680640 1 -- 172.21.15.63:0/4049984581 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb2780a2c60 msgr2=0x7fb2780a3040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:19:09.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb27f680640 1 --2- 172.21.15.63:0/4049984581 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb2780a2c60 0x7fb2780a3040 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb26c009d90 tx=0x7fb26c01c810 comp rx=0 tx=0).stop 2024-04-26T04:19:09.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb27f680640 1 -- 172.21.15.63:0/4049984581 shutdown_connections 2024-04-26T04:19:09.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb27f680640 1 --2- 172.21.15.63:0/4049984581 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fb2780a20c0 0x7fb2780a3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:19:09.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb27f680640 1 --2- 172.21.15.63:0/4049984581 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb2780a2c60 0x7fb2780a3040 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:19:09.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb27f680640 1 -- 172.21.15.63:0/4049984581 >> 172.21.15.63:0/4049984581 conn(0x7fb2780a5fb0 msgr2=0x7fb2780a63c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:19:09.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb27f680640 1 -- 172.21.15.63:0/4049984581 shutdown_connections 2024-04-26T04:19:09.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb27f680640 1 -- 172.21.15.63:0/4049984581 wait complete. 2024-04-26T04:19:09.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb27f680640 10 monclient: build_initial_monmap 2024-04-26T04:19:09.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb27f680640 10 monclient: monmap: 2024-04-26T04:19:09.071 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:19:09.072 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:19:09.072 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:19:09.072 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:19:09.072 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:19:09.072 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:19:09.072 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:19:09.072 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:19:09.072 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:19:09.072 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:19:09.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb27f680640 1 Processor -- start 2024-04-26T04:19:09.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb27f680640 1 -- start start 2024-04-26T04:19:09.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb27f680640 10 monclient: init 2024-04-26T04:19:09.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb27f680640 10 monclient: _reopen_session rank -1 2024-04-26T04:19:09.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb27f680640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T04:19:09.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb27f680640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb2780a20c0 0x7fb2781cee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:19:09.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb27f680640 10 monclient(hunting): picked mon.noname-a con 0x7fb2780a20c0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:19:09.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb27f680640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb2780a24a0 0x7fb27813e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:19:09.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb27f680640 10 monclient(hunting): picked mon.noname-b con 0x7fb2780a24a0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:19:09.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb27f680640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fb2780a2c60 0x7fb278124a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:19:09.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb27f680640 10 monclient(hunting): picked mon.noname-c con 0x7fb2780a2c60 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:19:09.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb27f680640 10 monclient(hunting): start opening mon connection 2024-04-26T04:19:09.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb27f680640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fb278141b60 con 0x7fb2780a20c0 2024-04-26T04:19:09.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb27f680640 10 monclient(hunting): start opening mon connection 2024-04-26T04:19:09.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb27f680640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb2781419e0 con 0x7fb2780a24a0 2024-04-26T04:19:09.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb27f680640 10 monclient(hunting): start opening mon connection 2024-04-26T04:19:09.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb27f680640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7fb278141ce0 con 0x7fb2780a2c60 2024-04-26T04:19:09.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb27f680640 10 monclient(hunting): _renew_subs 2024-04-26T04:19:09.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb27f680640 10 monclient(hunting): authenticate will time out at 980.774105s 2024-04-26T04:19:09.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb27dc1d640 1 -- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fb2780a2c60 msgr2=0x7fb278124a30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:3301/0 2024-04-26T04:19:09.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb27dc1d640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fb2780a2c60 0x7fb278124a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:19:09.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb27d41c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb2780a20c0 0x7fb2781cee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:19:09.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb27d41c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb2780a20c0 0x7fb2781cee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:39380/0 (socket says 172.21.15.63:39380) 2024-04-26T04:19:09.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb27d41c640 1 -- 172.21.15.63:0/2457060126 learned_addr learned my addr 172.21.15.63:0/2457060126 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:19:09.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb27d41c640 10 monclient(hunting): get_auth_request con 0x7fb2780a20c0 auth_method 0 2024-04-26T04:19:09.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb27d41c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:19:09.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb27d41c640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:19:09.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb27d41c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:19:09.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb27d41c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:19:09.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb27d41c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:19:09.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb27d41c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:19:09.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb27cc1b640 1 --2- 172.21.15.63:0/2457060126 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb2780a24a0 0x7fb27813e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:19:09.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb27cc1b640 10 monclient(hunting): get_auth_request con 0x7fb2780a24a0 auth_method 0 2024-04-26T04:19:09.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb27cc1b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:19:09.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb27cc1b640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:19:09.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb27cc1b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:19:09.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb27d41c640 10 monclient(hunting): handle_auth_done global_id 4386 payload 824 2024-04-26T04:19:09.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb27d41c640 1 -- 172.21.15.63:0/2457060126 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fb2780a2c60 msgr2=0x7fb278124a30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T04:19:09.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb27d41c640 1 --2- 172.21.15.63:0/2457060126 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fb2780a2c60 0x7fb278124a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:19:09.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb27d41c640 1 -- 172.21.15.63:0/2457060126 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb2780a24a0 msgr2=0x7fb27813e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:19:09.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb27d41c640 1 --2- 172.21.15.63:0/2457060126 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb2780a24a0 0x7fb27813e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:19:09.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb27d41c640 10 monclient: _finish_hunting 0 2024-04-26T04:19:09.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb27d41c640 1 monclient: found mon.noname-a 2024-04-26T04:19:09.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb27d41c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:19:09.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb27d41c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:19:09.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.062+0000 7fb27d41c640 1 -- 172.21.15.63:0/2457060126 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb278128120 con 0x7fb2780a20c0 2024-04-26T04:19:09.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.066+0000 7fb27d41c640 1 -- 172.21.15.63:0/2457060126 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb2780a20c0 msgr2=0x7fb2781cee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:19:09.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.066+0000 7fb27d41c640 1 --2- 172.21.15.63:0/2457060126 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb2780a20c0 0x7fb2781cee10 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb26c01aec0 tx=0x7fb26c0195b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:19:09.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.066+0000 7fb269ffb640 1 -- 172.21.15.63:0/2457060126 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb26c0bd020 con 0x7fb2780a20c0 2024-04-26T04:19:09.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.066+0000 7fb269ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:19:09.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.066+0000 7fb269ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:19:09.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.066+0000 7fb269ffb640 10 monclient: dump: 2024-04-26T04:19:09.081 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:19:09.081 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:19:09.081 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:19:09.082 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:19:09.082 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:19:09.082 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:19:09.082 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:19:09.082 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:19:09.082 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:19:09.082 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:19:09.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.066+0000 7fb269ffb640 10 monclient: _finish_auth 0 2024-04-26T04:19:09.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.066+0000 7fb269ffb640 10 monclient: _check_auth_tickets 2024-04-26T04:19:09.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.066+0000 7fb269ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:19:09.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.066+0000 7fb269ffb640 1 -- 172.21.15.63:0/2457060126 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb26c0b3040 con 0x7fb2780a20c0 2024-04-26T04:19:09.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.066+0000 7fb269ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T04:19:09.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.066+0000 7fb269ffb640 1 -- 172.21.15.63:0/2457060126 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb26c0b2070 con 0x7fb2780a20c0 2024-04-26T04:19:09.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.066+0000 7fb27f680640 5 monclient: authenticate success, global_id 4386 2024-04-26T04:19:09.083 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.066+0000 7fb27f680640 10 monclient: _renew_subs 2024-04-26T04:19:09.083 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.066+0000 7fb27f680640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:19:09.083 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.066+0000 7fb27f680640 1 -- 172.21.15.63:0/2457060126 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2780a38d0 con 0x7fb2780a20c0 2024-04-26T04:19:09.083 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.066+0000 7fb269ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:19:09.083 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.066+0000 7fb26b7fe640 4 set_mon_vals no callback set 2024-04-26T04:19:09.083 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.066+0000 7fb269ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:19:09.083 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.066+0000 7fb269ffb640 10 monclient: dump: 2024-04-26T04:19:09.083 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:19:09.083 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:19:09.083 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:19:09.083 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:19:09.083 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:19:09.084 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:19:09.084 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:19:09.084 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:19:09.084 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:19:09.084 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:19:09.084 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.066+0000 7fb27f680640 10 monclient: _renew_subs 2024-04-26T04:19:09.084 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.066+0000 7fb27f680640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:19:09.084 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.066+0000 7fb27f680640 1 -- 172.21.15.63:0/2457060126 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2781289c0 con 0x7fb2780a20c0 2024-04-26T04:19:09.084 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.066+0000 7fb27f680640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:19:09.084 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.066+0000 7fb27f680640 1 --2- 172.21.15.63:0/2457060126 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb278125730 0x7fb27802bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:19:09.084 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.066+0000 7fb27f680640 10 monclient(hunting): start opening mon connection 2024-04-26T04:19:09.084 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.066+0000 7fb27f680640 1 -- 172.21.15.63:0/2457060126 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fb278062e10 con 0x7fb278125730 2024-04-26T04:19:09.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.066+0000 7fb269ffb640 1 -- 172.21.15.63:0/2457060126 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fb26c0c0020 con 0x7fb2780a20c0 2024-04-26T04:19:09.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.066+0000 7fb27cc1b640 1 --2- 172.21.15.63:0/2457060126 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb278125730 0x7fb27802bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:19:09.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.066+0000 7fb27cc1b640 10 monclient: get_auth_request con 0x7fb278125730 auth_method 0 2024-04-26T04:19:09.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.066+0000 7fb27cc1b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:19:09.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.066+0000 7fb27cc1b640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:19:09.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.066+0000 7fb27cc1b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:19:09.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.066+0000 7fb27cc1b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:19:09.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.066+0000 7fb27cc1b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:19:09.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.066+0000 7fb27cc1b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:19:09.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.066+0000 7fb269ffb640 1 --2- 172.21.15.63:0/2457060126 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fb25c035e70 0x7fb25c038330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:19:09.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.066+0000 7fb269ffb640 1 -- 172.21.15.63:0/2457060126 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4738+0+0 (secure 0 0 0) 0x7fb26c0e3de0 con 0x7fb2780a20c0 2024-04-26T04:19:09.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.066+0000 7fb27cc1b640 10 monclient(hunting): handle_auth_done global_id 4389 payload 824 2024-04-26T04:19:09.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.066+0000 7fb27cc1b640 1 -- 172.21.15.63:0/2457060126 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb278121210 con 0x7fb278125730 2024-04-26T04:19:09.086 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.066+0000 7fb27dc1d640 1 --2- 172.21.15.63:0/2457060126 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fb25c035e70 0x7fb25c038330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:19:09.086 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.066+0000 7fb27dc1d640 10 monclient: get_auth_request con 0x7fb25c035e70 auth_method 0 2024-04-26T04:19:09.086 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.070+0000 7fb27cc1b640 1 -- 172.21.15.63:0/2457060126 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb278125730 msgr2=0x7fb27802bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:19:09.086 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.070+0000 7fb27cc1b640 1 --2- 172.21.15.63:0/2457060126 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb278125730 0x7fb27802bfc0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb278126f50 tx=0x7fb27400db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:19:09.086 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.070+0000 7fb269ffb640 1 -- 172.21.15.63:0/2457060126 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb274018070 con 0x7fb278125730 2024-04-26T04:19:09.086 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.070+0000 7fb269ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:19:09.086 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.070+0000 7fb269ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:19:09.086 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.070+0000 7fb269ffb640 10 monclient: dump: 2024-04-26T04:19:09.086 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:19:09.086 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:19:09.086 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:19:09.086 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:19:09.087 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:19:09.087 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:19:09.087 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:19:09.087 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:19:09.087 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:19:09.087 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:19:09.087 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.070+0000 7fb269ffb640 1 -- 172.21.15.63:0/2457060126 <== mon.0 v2:172.21.15.63:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb274005c60 con 0x7fb278125730 2024-04-26T04:19:09.087 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.070+0000 7fb269ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:19:09.087 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.070+0000 7fb269ffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:19:09.087 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.070+0000 7fb269ffb640 1 -- 172.21.15.63:0/2457060126 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb278125730 msgr2=0x7fb27802bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:19:09.087 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.070+0000 7fb269ffb640 1 --2- 172.21.15.63:0/2457060126 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb278125730 0x7fb27802bfc0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb278126f50 tx=0x7fb27400db90 comp rx=0 tx=0).stop 2024-04-26T04:19:09.087 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.070+0000 7fb269ffb640 1 -- 172.21.15.63:0/2457060126 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb278125730 msgr2=0x7fb27802bfc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:19:09.088 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.070+0000 7fb269ffb640 1 --2- 172.21.15.63:0/2457060126 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb278125730 0x7fb27802bfc0 secure :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb278126f50 tx=0x7fb27400db90 comp rx=0 tx=0).stop 2024-04-26T04:19:09.088 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.070+0000 7fb27dc1d640 1 --2- 172.21.15.63:0/2457060126 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fb25c035e70 0x7fb25c038330 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb270004820 tx=0x7fb270007ab0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:19:09.090 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.074+0000 7fb27f680640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T04:19:09.090 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.074+0000 7fb27f680640 1 --2- 172.21.15.63:0/2457060126 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb278125b10 0x7fb27802c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:19:09.090 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.074+0000 7fb27f680640 10 monclient(hunting): start opening mon connection 2024-04-26T04:19:09.090 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.074+0000 7fb27f680640 1 -- 172.21.15.63:0/2457060126 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fb2780683f0 con 0x7fb278125b10 2024-04-26T04:19:09.090 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.074+0000 7fb27cc1b640 1 --2- 172.21.15.63:0/2457060126 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb278125b10 0x7fb27802c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:19:09.090 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.074+0000 7fb27cc1b640 10 monclient: get_auth_request con 0x7fb278125b10 auth_method 0 2024-04-26T04:19:09.090 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.074+0000 7fb27cc1b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:19:09.090 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.074+0000 7fb27cc1b640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:19:09.090 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.074+0000 7fb27cc1b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:19:09.090 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.074+0000 7fb27cc1b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:19:09.091 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.074+0000 7fb27cc1b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:19:09.091 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.074+0000 7fb27cc1b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:19:09.091 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.074+0000 7fb27cc1b640 10 monclient(hunting): handle_auth_done global_id 4392 payload 824 2024-04-26T04:19:09.091 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.074+0000 7fb27cc1b640 1 -- 172.21.15.63:0/2457060126 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb2781cfc90 con 0x7fb278125b10 2024-04-26T04:19:09.091 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.078+0000 7fb27cc1b640 1 -- 172.21.15.63:0/2457060126 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb278125b10 msgr2=0x7fb27802c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:19:09.091 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.078+0000 7fb27cc1b640 1 --2- 172.21.15.63:0/2457060126 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb278125b10 0x7fb27802c500 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb274012000 tx=0x7fb2740059e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:19:09.091 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.078+0000 7fb269ffb640 1 -- 172.21.15.63:0/2457060126 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb274018030 con 0x7fb278125b10 2024-04-26T04:19:09.091 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.078+0000 7fb269ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:19:09.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.078+0000 7fb269ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:19:09.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.078+0000 7fb269ffb640 10 monclient: dump: 2024-04-26T04:19:09.092 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:19:09.092 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:19:09.092 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:19:09.092 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:19:09.092 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:19:09.092 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:19:09.092 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:19:09.092 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:19:09.092 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:19:09.093 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:19:09.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.078+0000 7fb269ffb640 1 -- 172.21.15.63:0/2457060126 <== mon.0 v2:172.21.15.63:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4711 (secure 0 0 0) 0x7fb274005320 con 0x7fb278125b10 2024-04-26T04:19:09.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.078+0000 7fb269ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T04:19:09.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.078+0000 7fb269ffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:19:09.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.078+0000 7fb269ffb640 1 -- 172.21.15.63:0/2457060126 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb278125b10 msgr2=0x7fb27802c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:19:09.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.078+0000 7fb269ffb640 1 --2- 172.21.15.63:0/2457060126 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb278125b10 0x7fb27802c500 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb274012000 tx=0x7fb2740059e0 comp rx=0 tx=0).stop 2024-04-26T04:19:09.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.078+0000 7fb269ffb640 1 -- 172.21.15.63:0/2457060126 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb278125b10 msgr2=0x7fb27802c500 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:19:09.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.078+0000 7fb269ffb640 1 --2- 172.21.15.63:0/2457060126 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb278125b10 0x7fb27802c500 secure :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb274012000 tx=0x7fb2740059e0 comp rx=0 tx=0).stop 2024-04-26T04:19:09.094 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-26T04:19:09.094 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a", 2024-04-26T04:19:09.094 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 0, 2024-04-26T04:19:09.094 INFO:teuthology.orchestra.run.smithi063.stdout: "state": "leader", 2024-04-26T04:19:09.094 INFO:teuthology.orchestra.run.smithi063.stdout: "election_epoch": 12, 2024-04-26T04:19:09.094 INFO:teuthology.orchestra.run.smithi063.stdout: "uptime": 40231, 2024-04-26T04:19:09.094 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum": [ 2024-04-26T04:19:09.094 INFO:teuthology.orchestra.run.smithi063.stdout: 0, 2024-04-26T04:19:09.094 INFO:teuthology.orchestra.run.smithi063.stdout: 1 2024-04-26T04:19:09.095 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:19:09.095 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_age": 0, 2024-04-26T04:19:09.095 INFO:teuthology.orchestra.run.smithi063.stdout: "features": { 2024-04-26T04:19:09.095 INFO:teuthology.orchestra.run.smithi063.stdout: "required_con": "2450521708007866372", 2024-04-26T04:19:09.095 INFO:teuthology.orchestra.run.smithi063.stdout: "required_mon": [ 2024-04-26T04:19:09.095 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:19:09.095 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:19:09.095 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:19:09.095 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:19:09.095 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:19:09.095 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:19:09.095 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:19:09.096 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:19:09.096 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:19:09.096 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:19:09.096 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:19:09.096 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:19:09.096 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_con": "4540701547738038271", 2024-04-26T04:19:09.097 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_mon": [ 2024-04-26T04:19:09.097 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:19:09.097 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:19:09.097 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:19:09.097 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:19:09.097 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:19:09.097 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:19:09.097 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:19:09.097 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:19:09.097 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:19:09.098 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:19:09.098 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:19:09.098 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:19:09.098 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:19:09.098 INFO:teuthology.orchestra.run.smithi063.stdout: "outside_quorum": [], 2024-04-26T04:19:09.098 INFO:teuthology.orchestra.run.smithi063.stdout: "extra_probe_peers": [], 2024-04-26T04:19:09.099 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_provider": [], 2024-04-26T04:19:09.099 INFO:teuthology.orchestra.run.smithi063.stdout: "monmap": { 2024-04-26T04:19:09.099 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 1, 2024-04-26T04:19:09.099 INFO:teuthology.orchestra.run.smithi063.stdout: "fsid": "cd1c6a0c-4e4b-4014-89c0-2e535479ef39", 2024-04-26T04:19:09.099 INFO:teuthology.orchestra.run.smithi063.stdout: "modified": "2024-04-26T04:18:03.702339Z", 2024-04-26T04:19:09.099 INFO:teuthology.orchestra.run.smithi063.stdout: "created": "2024-04-26T04:18:03.702339Z", 2024-04-26T04:19:09.099 INFO:teuthology.orchestra.run.smithi063.stdout: "min_mon_release": 19, 2024-04-26T04:19:09.099 INFO:teuthology.orchestra.run.smithi063.stdout: "min_mon_release_name": "squid", 2024-04-26T04:19:09.099 INFO:teuthology.orchestra.run.smithi063.stdout: "election_strategy": 1, 2024-04-26T04:19:09.099 INFO:teuthology.orchestra.run.smithi063.stdout: "disallowed_leaders: ": "", 2024-04-26T04:19:09.100 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": false, 2024-04-26T04:19:09.100 INFO:teuthology.orchestra.run.smithi063.stdout: "tiebreaker_mon": "", 2024-04-26T04:19:09.100 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_ranks: ": "", 2024-04-26T04:19:09.100 INFO:teuthology.orchestra.run.smithi063.stdout: "features": { 2024-04-26T04:19:09.100 INFO:teuthology.orchestra.run.smithi063.stdout: "persistent": [ 2024-04-26T04:19:09.100 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:19:09.100 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:19:09.100 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:19:09.100 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:19:09.100 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:19:09.100 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:19:09.100 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:19:09.101 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:19:09.101 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:19:09.101 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:19:09.101 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:19:09.101 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:19:09.101 INFO:teuthology.orchestra.run.smithi063.stdout: "optional": [] 2024-04-26T04:19:09.101 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:19:09.101 INFO:teuthology.orchestra.run.smithi063.stdout: "mons": [ 2024-04-26T04:19:09.101 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:19:09.101 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 0, 2024-04-26T04:19:09.102 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a", 2024-04-26T04:19:09.102 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:19:09.102 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:19:09.102 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:19:09.102 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:19:09.102 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:3300", 2024-04-26T04:19:09.103 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:19:09.103 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:19:09.103 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:19:09.103 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:19:09.103 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6789", 2024-04-26T04:19:09.103 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:19:09.103 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:19:09.103 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:19:09.103 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:19:09.103 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6789/0", 2024-04-26T04:19:09.104 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6789/0", 2024-04-26T04:19:09.104 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:19:09.104 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:19:09.104 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:19:09.104 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:19:09.104 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:19:09.104 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 1, 2024-04-26T04:19:09.104 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b", 2024-04-26T04:19:09.104 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:19:09.104 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:19:09.104 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:19:09.104 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:19:09.105 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:3300", 2024-04-26T04:19:09.105 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:19:09.105 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:19:09.105 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:19:09.105 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:19:09.105 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:6789", 2024-04-26T04:19:09.105 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:19:09.105 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:19:09.105 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:19:09.105 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:19:09.105 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:6789/0", 2024-04-26T04:19:09.105 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.186:6789/0", 2024-04-26T04:19:09.106 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:19:09.106 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:19:09.106 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:19:09.106 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:19:09.106 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:19:09.106 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 2, 2024-04-26T04:19:09.106 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "c", 2024-04-26T04:19:09.106 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:19:09.106 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:19:09.106 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:19:09.106 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:19:09.107 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:3301", 2024-04-26T04:19:09.107 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:19:09.107 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:19:09.107 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:19:09.107 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:19:09.107 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6790", 2024-04-26T04:19:09.107 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:19:09.107 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:19:09.107 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:19:09.107 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:19:09.108 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6790/0", 2024-04-26T04:19:09.108 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6790/0", 2024-04-26T04:19:09.108 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:19:09.108 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:19:09.108 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:19:09.108 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:19:09.108 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:19:09.108 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:19:09.109 INFO:teuthology.orchestra.run.smithi063.stdout: "feature_map": { 2024-04-26T04:19:09.109 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": [ 2024-04-26T04:19:09.109 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:19:09.109 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:19:09.109 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:19:09.109 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 1 2024-04-26T04:19:09.109 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:19:09.109 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:19:09.109 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": [ 2024-04-26T04:19:09.109 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:19:09.109 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:19:09.109 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:19:09.110 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 3 2024-04-26T04:19:09.110 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:19:09.110 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:19:09.110 INFO:teuthology.orchestra.run.smithi063.stdout: "client": [ 2024-04-26T04:19:09.110 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:19:09.110 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:19:09.110 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:19:09.110 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 2 2024-04-26T04:19:09.110 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:19:09.110 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:19:09.110 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:19:09.110 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": false 2024-04-26T04:19:09.110 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-26T04:19:09.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.078+0000 7fb27f680640 1 -- 172.21.15.63:0/2457060126 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fb25c035e70 msgr2=0x7fb25c038330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:19:09.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.078+0000 7fb27f680640 1 --2- 172.21.15.63:0/2457060126 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fb25c035e70 0x7fb25c038330 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb270004820 tx=0x7fb270007ab0 comp rx=0 tx=0).stop 2024-04-26T04:19:09.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.078+0000 7fb27f680640 10 monclient: shutdown 2024-04-26T04:19:09.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.078+0000 7fb27f680640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:19:09.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.078+0000 7fb27f680640 1 -- 172.21.15.63:0/2457060126 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb2780a20c0 msgr2=0x7fb2781cee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:19:09.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.078+0000 7fb27f680640 1 --2- 172.21.15.63:0/2457060126 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb2780a20c0 0x7fb2781cee10 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb26c01aec0 tx=0x7fb26c0195b0 comp rx=0 tx=0).stop 2024-04-26T04:19:09.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.078+0000 7fb27dc1d640 1 -- 172.21.15.63:0/2457060126 reap_dead start 2024-04-26T04:19:09.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.078+0000 7fb27dc1d640 1 -- 172.21.15.63:0/2457060126 reap_dead start 2024-04-26T04:19:09.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.078+0000 7fb27f680640 1 -- 172.21.15.63:0/2457060126 shutdown_connections 2024-04-26T04:19:09.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.078+0000 7fb27f680640 1 -- 172.21.15.63:0/2457060126 >> 172.21.15.63:0/2457060126 conn(0x7fb2780a5fb0 msgr2=0x7fb2780710c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:19:09.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.078+0000 7fb27f680640 1 -- 172.21.15.63:0/2457060126 shutdown_connections 2024-04-26T04:19:09.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.078+0000 7fb27f680640 1 -- 172.21.15.63:0/2457060126 wait complete. 2024-04-26T04:19:09.113 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-26T04:19:09.114 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-04-26T04:19:09.114 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T04:19:09.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.234+0000 7fe89f0f7640 10 monclient: get_monmap_and_config 2024-04-26T04:19:09.239 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.234+0000 7fe89f0f7640 10 monclient: build_initial_monmap 2024-04-26T04:19:09.239 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.234+0000 7fe89f0f7640 10 monclient: monmap: 2024-04-26T04:19:09.239 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:19:09.239 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:19:09.239 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:19:09.238828+0000 2024-04-26T04:19:09.239 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:19:09.238828+0000 2024-04-26T04:19:09.239 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:19:09.239 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:19:09.239 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:19:09.239 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:19:09.239 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:19:09.239 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:19:09.239 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:19:09.239 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:19:09.242 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.234+0000 7fe89f0f7640 1 Processor -- start 2024-04-26T04:19:09.242 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.234+0000 7fe89f0f7640 1 -- start start 2024-04-26T04:19:09.242 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.234+0000 7fe89f0f7640 10 monclient: init 2024-04-26T04:19:09.242 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.234+0000 7fe89f0f7640 10 monclient: _reopen_session rank -1 2024-04-26T04:19:09.242 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.234+0000 7fe89f0f7640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-04-26T04:19:09.242 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.234+0000 7fe89f0f7640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fe8980a2cd0 0x7fe8980a30b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:19:09.242 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.234+0000 7fe89f0f7640 10 monclient(hunting): picked mon.noname-b con 0x7fe8980a2cd0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:19:09.242 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.234+0000 7fe89f0f7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe8980a47d0 addr v1:172.21.15.186:6789/0 2024-04-26T04:19:09.242 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.234+0000 7fe89f0f7640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe8980a4bd0 0x7fe8981373f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:19:09.242 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.234+0000 7fe89f0f7640 10 monclient(hunting): picked mon.noname-c con 0x7fe8980a4bd0 addr v2:172.21.15.186:3300/0 2024-04-26T04:19:09.242 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.234+0000 7fe89f0f7640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8980a0fb0 con 0x7fe8980a47d0 2024-04-26T04:19:09.242 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.234+0000 7fe89f0f7640 10 monclient(hunting): start opening mon connection 2024-04-26T04:19:09.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.234+0000 7fe89f0f7640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fe8980a35f0 con 0x7fe8980a4bd0 2024-04-26T04:19:09.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.234+0000 7fe89f0f7640 10 monclient(hunting): start opening mon connection 2024-04-26T04:19:09.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.234+0000 7fe89f0f7640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7fe8980a5380 con 0x7fe8980a2cd0 2024-04-26T04:19:09.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.234+0000 7fe89f0f7640 10 monclient(hunting): _renew_subs 2024-04-26T04:19:09.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.234+0000 7fe89f0f7640 10 monclient(hunting): authenticate will time out at 980.946564s 2024-04-26T04:19:09.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.234+0000 7fe89ce93640 1 -- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fe8980a2cd0 msgr2=0x7fe8980a30b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.63:3301/0 2024-04-26T04:19:09.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.234+0000 7fe89ce93640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fe8980a2cd0 0x7fe8980a30b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:19:09.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.234+0000 7fe89d694640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe8980a4bd0 0x7fe8981373f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:19:09.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.234+0000 7fe897fff640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fe8980a47d0 0x7fe898136ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.63:49258/0 (socket says 172.21.15.63:49258) 2024-04-26T04:19:09.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.234+0000 7fe897fff640 1 -- 172.21.15.63:0/4282596371 learned_addr learned my addr 172.21.15.63:0/4282596371 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-04-26T04:19:09.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.234+0000 7fe89d694640 10 monclient(hunting): get_auth_request con 0x7fe8980a4bd0 auth_method 0 2024-04-26T04:19:09.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.234+0000 7fe89d694640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:19:09.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.234+0000 7fe89d694640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:19:09.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.234+0000 7fe89d694640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:19:09.244 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.234+0000 7fe89d694640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:19:09.244 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.234+0000 7fe89d694640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:19:09.244 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.234+0000 7fe89d694640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:19:09.244 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.234+0000 7fe89d694640 10 monclient(hunting): handle_auth_done global_id 4336 payload 274 2024-04-26T04:19:09.244 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.234+0000 7fe89d694640 1 -- 172.21.15.63:0/4282596371 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fe8980a2cd0 msgr2=0x7fe8980a30b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T04:19:09.244 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.234+0000 7fe89d694640 1 --2- 172.21.15.63:0/4282596371 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fe8980a2cd0 0x7fe8980a30b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:19:09.244 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.234+0000 7fe89d694640 1 -- 172.21.15.63:0/4282596371 >> v1:172.21.15.186:6789/0 conn(0x7fe8980a47d0 legacy=0x7fe898136ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:19:09.244 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.234+0000 7fe89d694640 10 monclient: _finish_hunting 0 2024-04-26T04:19:09.244 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.234+0000 7fe89d694640 1 monclient: found mon.noname-c 2024-04-26T04:19:09.244 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.234+0000 7fe89d694640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:19:09.244 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.234+0000 7fe89d694640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.186:3300/0 2024-04-26T04:19:09.244 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.234+0000 7fe89d694640 1 -- 172.21.15.63:0/4282596371 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8980a1290 con 0x7fe8980a4bd0 2024-04-26T04:19:09.245 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe89d694640 1 -- 172.21.15.63:0/4282596371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8980a4bd0 msgr2=0x7fe8981373f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:19:09.245 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe89d694640 1 --2- 172.21.15.63:0/4282596371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8980a4bd0 0x7fe8981373f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe8880097c0 tx=0x7fe88801c3d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe8d2534cedab790 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:19:09.247 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe8977fe640 1 -- 172.21.15.63:0/4282596371 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8880b0070 con 0x7fe8980a4bd0 2024-04-26T04:19:09.247 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe8977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:19:09.247 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe8977fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:19:09.247 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe8977fe640 10 monclient: dump: 2024-04-26T04:19:09.247 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:19:09.247 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:19:09.247 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:19:09.247 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:19:09.247 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:19:09.247 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:19:09.248 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:19:09.248 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:19:09.248 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:19:09.248 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:19:09.248 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe8977fe640 10 monclient: _finish_auth 0 2024-04-26T04:19:09.248 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe8977fe640 10 monclient: _check_auth_tickets 2024-04-26T04:19:09.248 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe8977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:19:09.248 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe8977fe640 1 -- 172.21.15.63:0/4282596371 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8880ab600 con 0x7fe8980a4bd0 2024-04-26T04:19:09.248 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe8977fe640 10 monclient: handle_config config(0 keys) 2024-04-26T04:19:09.248 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe8977fe640 1 -- 172.21.15.63:0/4282596371 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8880ab9d0 con 0x7fe8980a4bd0 2024-04-26T04:19:09.248 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe8977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:19:09.248 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe8977fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T04:19:09.248 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe8977fe640 10 monclient: dump: 2024-04-26T04:19:09.248 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:19:09.249 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:19:09.249 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:19:09.249 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:19:09.249 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:19:09.249 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:19:09.249 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:19:09.249 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:19:09.249 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:19:09.249 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:19:09.249 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe89f0f7640 5 monclient: authenticate success, global_id 4336 2024-04-26T04:19:09.249 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe89f0f7640 10 monclient: get_monmap_and_config success 2024-04-26T04:19:09.249 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe89f0f7640 4 set_mon_vals no callback set 2024-04-26T04:19:09.249 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe89f0f7640 10 monclient: shutdown 2024-04-26T04:19:09.249 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe89f0f7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:19:09.249 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe89f0f7640 1 -- 172.21.15.63:0/4282596371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8980a4bd0 msgr2=0x7fe8981373f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:19:09.250 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe89f0f7640 1 --2- 172.21.15.63:0/4282596371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8980a4bd0 0x7fe8981373f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe8880097c0 tx=0x7fe88801c3d0 comp rx=0 tx=0).stop 2024-04-26T04:19:09.250 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe89f0f7640 1 -- 172.21.15.63:0/4282596371 shutdown_connections 2024-04-26T04:19:09.250 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe89f0f7640 1 --2- 172.21.15.63:0/4282596371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8980a4bd0 0x7fe8981373f0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:19:09.250 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe89f0f7640 1 --2- 172.21.15.63:0/4282596371 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fe8980a2cd0 0x7fe8980a30b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:19:09.250 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe89f0f7640 1 -- 172.21.15.63:0/4282596371 >> 172.21.15.63:0/4282596371 conn(0x7fe8980a5f40 msgr2=0x7fe8980a6320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:19:09.250 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe89f0f7640 1 -- 172.21.15.63:0/4282596371 shutdown_connections 2024-04-26T04:19:09.250 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe89f0f7640 1 -- 172.21.15.63:0/4282596371 wait complete. 2024-04-26T04:19:09.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe89f0f7640 10 monclient: build_initial_monmap 2024-04-26T04:19:09.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe89f0f7640 10 monclient: monmap: 2024-04-26T04:19:09.254 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:19:09.254 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:19:09.254 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:19:09.254 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:19:09.254 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:19:09.254 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:19:09.255 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:19:09.255 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:19:09.255 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:19:09.255 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:19:09.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe89f0f7640 1 Processor -- start 2024-04-26T04:19:09.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe89f0f7640 1 -- start start 2024-04-26T04:19:09.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe89f0f7640 10 monclient: init 2024-04-26T04:19:09.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe89f0f7640 10 monclient: _reopen_session rank -1 2024-04-26T04:19:09.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe89f0f7640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T04:19:09.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe89f0f7640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe8980a2cd0 0x7fe8981355a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:19:09.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe89f0f7640 10 monclient(hunting): picked mon.noname-a con 0x7fe8980a2cd0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:19:09.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe89f0f7640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fe8980a47d0 0x7fe898135ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:19:09.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe89f0f7640 10 monclient(hunting): picked mon.noname-c con 0x7fe8980a47d0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:19:09.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe89f0f7640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8980a4bd0 0x7fe89813b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:19:09.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe89f0f7640 10 monclient(hunting): picked mon.noname-b con 0x7fe8980a4bd0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:19:09.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe89f0f7640 10 monclient(hunting): start opening mon connection 2024-04-26T04:19:09.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe89f0f7640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe898141a50 con 0x7fe8980a2cd0 2024-04-26T04:19:09.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe89f0f7640 10 monclient(hunting): start opening mon connection 2024-04-26T04:19:09.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe89f0f7640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe898137b40 con 0x7fe8980a4bd0 2024-04-26T04:19:09.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe89f0f7640 10 monclient(hunting): start opening mon connection 2024-04-26T04:19:09.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe89f0f7640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7fe898141bd0 con 0x7fe8980a47d0 2024-04-26T04:19:09.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe89f0f7640 10 monclient(hunting): _renew_subs 2024-04-26T04:19:09.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe89f0f7640 10 monclient(hunting): authenticate will time out at 980.955328s 2024-04-26T04:19:09.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe897fff640 1 -- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fe8980a47d0 msgr2=0x7fe898135ae0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:3301/0 2024-04-26T04:19:09.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe897fff640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fe8980a47d0 0x7fe898135ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:19:09.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe89ce93640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe8980a2cd0 0x7fe8981355a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:19:09.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe89ce93640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe8980a2cd0 0x7fe8981355a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:39408/0 (socket says 172.21.15.63:39408) 2024-04-26T04:19:09.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe89ce93640 1 -- 172.21.15.63:0/3068918447 learned_addr learned my addr 172.21.15.63:0/3068918447 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:19:09.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe89ce93640 10 monclient(hunting): get_auth_request con 0x7fe8980a2cd0 auth_method 0 2024-04-26T04:19:09.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe89ce93640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:19:09.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe89ce93640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:19:09.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe89ce93640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:19:09.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe89ce93640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:19:09.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe89ce93640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:19:09.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe89ce93640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:19:09.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe89ce93640 10 monclient(hunting): handle_auth_done global_id 4395 payload 824 2024-04-26T04:19:09.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe89d694640 1 --2- 172.21.15.63:0/3068918447 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8980a4bd0 0x7fe89813b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:19:09.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe89ce93640 1 -- 172.21.15.63:0/3068918447 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fe8980a47d0 msgr2=0x7fe898135ae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T04:19:09.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe89ce93640 1 --2- 172.21.15.63:0/3068918447 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fe8980a47d0 0x7fe898135ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:19:09.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe89ce93640 1 -- 172.21.15.63:0/3068918447 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8980a4bd0 msgr2=0x7fe89813b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:19:09.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe89ce93640 1 --2- 172.21.15.63:0/3068918447 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8980a4bd0 0x7fe89813b860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:19:09.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe89ce93640 10 monclient: _finish_hunting 0 2024-04-26T04:19:09.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe89ce93640 1 monclient: found mon.noname-a 2024-04-26T04:19:09.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe89ce93640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:19:09.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe89ce93640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:19:09.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.242+0000 7fe89ce93640 1 -- 172.21.15.63:0/3068918447 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8980a37d0 con 0x7fe8980a2cd0 2024-04-26T04:19:09.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.246+0000 7fe89ce93640 1 -- 172.21.15.63:0/3068918447 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe8980a2cd0 msgr2=0x7fe8981355a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:19:09.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.246+0000 7fe89ce93640 1 --2- 172.21.15.63:0/3068918447 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe8980a2cd0 0x7fe8981355a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe89000b9e0 tx=0x7fe89000bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:19:09.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.246+0000 7fe8957fa640 1 -- 172.21.15.63:0/3068918447 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe89000cdb0 con 0x7fe8980a2cd0 2024-04-26T04:19:09.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.246+0000 7fe8957fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:19:09.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.246+0000 7fe8957fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:19:09.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.246+0000 7fe8957fa640 10 monclient: dump: 2024-04-26T04:19:09.258 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:19:09.258 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:19:09.258 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:19:09.258 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:19:09.258 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:19:09.259 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:19:09.259 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:19:09.259 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:19:09.259 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:19:09.259 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:19:09.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.246+0000 7fe8957fa640 10 monclient: _finish_auth 0 2024-04-26T04:19:09.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.246+0000 7fe8957fa640 10 monclient: _check_auth_tickets 2024-04-26T04:19:09.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.246+0000 7fe8957fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:19:09.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.246+0000 7fe8957fa640 1 -- 172.21.15.63:0/3068918447 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe890010070 con 0x7fe8980a2cd0 2024-04-26T04:19:09.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.246+0000 7fe89f0f7640 5 monclient: authenticate success, global_id 4395 2024-04-26T04:19:09.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.246+0000 7fe89f0f7640 10 monclient: _renew_subs 2024-04-26T04:19:09.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.246+0000 7fe89f0f7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:19:09.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.246+0000 7fe89f0f7640 1 -- 172.21.15.63:0/3068918447 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8980729b0 con 0x7fe8980a2cd0 2024-04-26T04:19:09.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.246+0000 7fe8957fa640 10 monclient: handle_config config(0 keys) 2024-04-26T04:19:09.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.246+0000 7fe8957fa640 1 -- 172.21.15.63:0/3068918447 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8900156b0 con 0x7fe8980a2cd0 2024-04-26T04:19:09.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.246+0000 7fe8957fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:19:09.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.246+0000 7fe8957fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:19:09.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.246+0000 7fe8957fa640 10 monclient: dump: 2024-04-26T04:19:09.260 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:19:09.260 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:19:09.260 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:19:09.260 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:19:09.260 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:19:09.260 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:19:09.260 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:19:09.260 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:19:09.260 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:19:09.260 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:19:09.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.246+0000 7fe89f0f7640 10 monclient: _renew_subs 2024-04-26T04:19:09.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.246+0000 7fe89f0f7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:19:09.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.246+0000 7fe89f0f7640 1 -- 172.21.15.63:0/3068918447 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe89813dde0 con 0x7fe8980a2cd0 2024-04-26T04:19:09.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.246+0000 7fe896ffd640 4 set_mon_vals no callback set 2024-04-26T04:19:09.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.246+0000 7fe89f0f7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:19:09.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.246+0000 7fe89f0f7640 1 --2- 172.21.15.63:0/3068918447 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe89813bda0 0x7fe89813c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:19:09.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.246+0000 7fe89f0f7640 10 monclient(hunting): start opening mon connection 2024-04-26T04:19:09.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.246+0000 7fe89f0f7640 1 -- 172.21.15.63:0/3068918447 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe898062e10 con 0x7fe89813bda0 2024-04-26T04:19:09.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.246+0000 7fe8957fa640 1 -- 172.21.15.63:0/3068918447 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fe89001e3f0 con 0x7fe8980a2cd0 2024-04-26T04:19:09.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.246+0000 7fe897fff640 1 --2- 172.21.15.63:0/3068918447 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe89813bda0 0x7fe89813c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:19:09.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.246+0000 7fe8957fa640 1 --2- 172.21.15.63:0/3068918447 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fe884035e70 0x7fe884038330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:19:09.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.246+0000 7fe897fff640 10 monclient: get_auth_request con 0x7fe89813bda0 auth_method 0 2024-04-26T04:19:09.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.246+0000 7fe897fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:19:09.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.246+0000 7fe897fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:19:09.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.246+0000 7fe897fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:19:09.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.246+0000 7fe8957fa640 1 -- 172.21.15.63:0/3068918447 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4738+0+0 (secure 0 0 0) 0x7fe890049160 con 0x7fe8980a2cd0 2024-04-26T04:19:09.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.246+0000 7fe897fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:19:09.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.246+0000 7fe897fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:19:09.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.246+0000 7fe897fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:19:09.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.246+0000 7fe89d694640 1 --2- 172.21.15.63:0/3068918447 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fe884035e70 0x7fe884038330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:19:09.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.246+0000 7fe89d694640 10 monclient: get_auth_request con 0x7fe884035e70 auth_method 0 2024-04-26T04:19:09.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.246+0000 7fe897fff640 10 monclient(hunting): handle_auth_done global_id 4339 payload 824 2024-04-26T04:19:09.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.246+0000 7fe897fff640 1 -- 172.21.15.63:0/3068918447 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe898136020 con 0x7fe89813bda0 2024-04-26T04:19:09.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.250+0000 7fe897fff640 1 -- 172.21.15.63:0/3068918447 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe89813bda0 msgr2=0x7fe89813c180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:19:09.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.250+0000 7fe897fff640 1 --2- 172.21.15.63:0/3068918447 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe89813bda0 0x7fe89813c180 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe898136bd0 tx=0x7fe88c00c8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:19:09.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.250+0000 7fe89d694640 1 --2- 172.21.15.63:0/3068918447 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fe884035e70 0x7fe884038330 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe8880ab000 tx=0x7fe888008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:19:09.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.250+0000 7fe8957fa640 1 -- 172.21.15.63:0/3068918447 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe88c017070 con 0x7fe89813bda0 2024-04-26T04:19:09.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.250+0000 7fe8957fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:19:09.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.250+0000 7fe8957fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T04:19:09.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.250+0000 7fe8957fa640 10 monclient: dump: 2024-04-26T04:19:09.264 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:19:09.264 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:19:09.265 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:19:09.265 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:19:09.265 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:19:09.265 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:19:09.265 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:19:09.265 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:19:09.265 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:19:09.265 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:19:09.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.250+0000 7fe8957fa640 1 -- 172.21.15.63:0/3068918447 <== mon.1 v2:172.21.15.186:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe88c0070d0 con 0x7fe89813bda0 2024-04-26T04:19:09.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.250+0000 7fe8957fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:19:09.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.250+0000 7fe8957fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:19:09.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.250+0000 7fe8957fa640 1 -- 172.21.15.63:0/3068918447 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe89813bda0 msgr2=0x7fe89813c180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:19:09.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.250+0000 7fe8957fa640 1 --2- 172.21.15.63:0/3068918447 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe89813bda0 0x7fe89813c180 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe898136bd0 tx=0x7fe88c00c8e0 comp rx=0 tx=0).stop 2024-04-26T04:19:09.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.250+0000 7fe8957fa640 1 -- 172.21.15.63:0/3068918447 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe89813bda0 msgr2=0x7fe89813c180 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:19:09.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.250+0000 7fe8957fa640 1 --2- 172.21.15.63:0/3068918447 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe89813bda0 0x7fe89813c180 secure :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe898136bd0 tx=0x7fe88c00c8e0 comp rx=0 tx=0).stop 2024-04-26T04:19:09.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.258+0000 7fe89f0f7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T04:19:09.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.258+0000 7fe89f0f7640 1 --2- 172.21.15.63:0/3068918447 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe89813c6c0 0x7fe898137cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:19:09.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.258+0000 7fe89f0f7640 10 monclient(hunting): start opening mon connection 2024-04-26T04:19:09.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.258+0000 7fe89f0f7640 1 -- 172.21.15.63:0/3068918447 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe8980683d0 con 0x7fe89813c6c0 2024-04-26T04:19:09.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.258+0000 7fe897fff640 1 --2- 172.21.15.63:0/3068918447 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe89813c6c0 0x7fe898137cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:19:09.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.258+0000 7fe897fff640 10 monclient: get_auth_request con 0x7fe89813c6c0 auth_method 0 2024-04-26T04:19:09.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.258+0000 7fe897fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:19:09.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.258+0000 7fe897fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:19:09.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.258+0000 7fe897fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:19:09.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.258+0000 7fe897fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:19:09.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.258+0000 7fe897fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:19:09.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.258+0000 7fe897fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:19:09.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.258+0000 7fe897fff640 10 monclient(hunting): handle_auth_done global_id 4342 payload 824 2024-04-26T04:19:09.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.258+0000 7fe897fff640 1 -- 172.21.15.63:0/3068918447 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe898120ab0 con 0x7fe89813c6c0 2024-04-26T04:19:09.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.262+0000 7fe897fff640 1 -- 172.21.15.63:0/3068918447 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe89813c6c0 msgr2=0x7fe898137cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:19:09.268 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.262+0000 7fe897fff640 1 --2- 172.21.15.63:0/3068918447 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe89813c6c0 0x7fe898137cf0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe88c011000 tx=0x7fe88c004a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:19:09.268 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.262+0000 7fe8957fa640 1 -- 172.21.15.63:0/3068918447 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe88c017030 con 0x7fe89813c6c0 2024-04-26T04:19:09.268 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.262+0000 7fe8957fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:19:09.268 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.262+0000 7fe8957fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T04:19:09.268 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.262+0000 7fe8957fa640 10 monclient: dump: 2024-04-26T04:19:09.268 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:19:09.268 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:19:09.268 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:19:09.268 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:19:09.268 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:19:09.268 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:19:09.268 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:19:09.268 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:19:09.269 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:19:09.269 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:19:09.269 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.262+0000 7fe8957fa640 1 -- 172.21.15.63:0/3068918447 <== mon.1 v2:172.21.15.186:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4879 (secure 0 0 0) 0x7fe88c0192c0 con 0x7fe89813c6c0 2024-04-26T04:19:09.269 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.262+0000 7fe8957fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T04:19:09.269 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.262+0000 7fe8957fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:19:09.269 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.262+0000 7fe8957fa640 1 -- 172.21.15.63:0/3068918447 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe89813c6c0 msgr2=0x7fe898137cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:19:09.269 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.262+0000 7fe8957fa640 1 --2- 172.21.15.63:0/3068918447 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe89813c6c0 0x7fe898137cf0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe88c011000 tx=0x7fe88c004a20 comp rx=0 tx=0).stop 2024-04-26T04:19:09.269 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.262+0000 7fe8957fa640 1 -- 172.21.15.63:0/3068918447 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe89813c6c0 msgr2=0x7fe898137cf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:19:09.269 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.262+0000 7fe8957fa640 1 --2- 172.21.15.63:0/3068918447 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe89813c6c0 0x7fe898137cf0 secure :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe88c011000 tx=0x7fe88c004a20 comp rx=0 tx=0).stop 2024-04-26T04:19:09.269 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-26T04:19:09.269 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b", 2024-04-26T04:19:09.270 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 1, 2024-04-26T04:19:09.270 INFO:teuthology.orchestra.run.smithi063.stdout: "state": "peon", 2024-04-26T04:19:09.270 INFO:teuthology.orchestra.run.smithi063.stdout: "election_epoch": 12, 2024-04-26T04:19:09.270 INFO:teuthology.orchestra.run.smithi063.stdout: "uptime": 40379, 2024-04-26T04:19:09.270 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum": [ 2024-04-26T04:19:09.270 INFO:teuthology.orchestra.run.smithi063.stdout: 0, 2024-04-26T04:19:09.270 INFO:teuthology.orchestra.run.smithi063.stdout: 1 2024-04-26T04:19:09.270 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:19:09.270 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_age": 0, 2024-04-26T04:19:09.270 INFO:teuthology.orchestra.run.smithi063.stdout: "features": { 2024-04-26T04:19:09.270 INFO:teuthology.orchestra.run.smithi063.stdout: "required_con": "2450521708007866372", 2024-04-26T04:19:09.270 INFO:teuthology.orchestra.run.smithi063.stdout: "required_mon": [ 2024-04-26T04:19:09.270 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:19:09.271 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:19:09.271 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:19:09.271 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:19:09.271 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:19:09.271 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:19:09.271 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:19:09.271 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:19:09.271 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:19:09.271 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:19:09.271 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:19:09.271 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:19:09.271 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_con": "4540701547738038271", 2024-04-26T04:19:09.271 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_mon": [ 2024-04-26T04:19:09.271 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:19:09.271 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:19:09.272 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:19:09.272 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:19:09.272 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:19:09.272 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:19:09.272 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:19:09.272 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:19:09.272 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:19:09.272 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:19:09.272 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:19:09.272 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:19:09.272 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:19:09.272 INFO:teuthology.orchestra.run.smithi063.stdout: "outside_quorum": [], 2024-04-26T04:19:09.273 INFO:teuthology.orchestra.run.smithi063.stdout: "extra_probe_peers": [], 2024-04-26T04:19:09.273 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_provider": [], 2024-04-26T04:19:09.273 INFO:teuthology.orchestra.run.smithi063.stdout: "monmap": { 2024-04-26T04:19:09.273 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 1, 2024-04-26T04:19:09.273 INFO:teuthology.orchestra.run.smithi063.stdout: "fsid": "cd1c6a0c-4e4b-4014-89c0-2e535479ef39", 2024-04-26T04:19:09.273 INFO:teuthology.orchestra.run.smithi063.stdout: "modified": "2024-04-26T04:18:03.702339Z", 2024-04-26T04:19:09.273 INFO:teuthology.orchestra.run.smithi063.stdout: "created": "2024-04-26T04:18:03.702339Z", 2024-04-26T04:19:09.273 INFO:teuthology.orchestra.run.smithi063.stdout: "min_mon_release": 19, 2024-04-26T04:19:09.273 INFO:teuthology.orchestra.run.smithi063.stdout: "min_mon_release_name": "squid", 2024-04-26T04:19:09.273 INFO:teuthology.orchestra.run.smithi063.stdout: "election_strategy": 1, 2024-04-26T04:19:09.273 INFO:teuthology.orchestra.run.smithi063.stdout: "disallowed_leaders: ": "", 2024-04-26T04:19:09.273 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": false, 2024-04-26T04:19:09.273 INFO:teuthology.orchestra.run.smithi063.stdout: "tiebreaker_mon": "", 2024-04-26T04:19:09.273 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_ranks: ": "", 2024-04-26T04:19:09.273 INFO:teuthology.orchestra.run.smithi063.stdout: "features": { 2024-04-26T04:19:09.274 INFO:teuthology.orchestra.run.smithi063.stdout: "persistent": [ 2024-04-26T04:19:09.274 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:19:09.274 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:19:09.274 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:19:09.274 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:19:09.274 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:19:09.274 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:19:09.274 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:19:09.274 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:19:09.274 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:19:09.274 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:19:09.274 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:19:09.274 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:19:09.274 INFO:teuthology.orchestra.run.smithi063.stdout: "optional": [] 2024-04-26T04:19:09.275 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:19:09.275 INFO:teuthology.orchestra.run.smithi063.stdout: "mons": [ 2024-04-26T04:19:09.275 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:19:09.275 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 0, 2024-04-26T04:19:09.275 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a", 2024-04-26T04:19:09.275 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:19:09.275 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:19:09.275 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:19:09.275 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:19:09.275 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:3300", 2024-04-26T04:19:09.275 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:19:09.275 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:19:09.275 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:19:09.275 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:19:09.276 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6789", 2024-04-26T04:19:09.276 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:19:09.276 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:19:09.276 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:19:09.276 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:19:09.276 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6789/0", 2024-04-26T04:19:09.276 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6789/0", 2024-04-26T04:19:09.276 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:19:09.276 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:19:09.276 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:19:09.276 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:19:09.276 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:19:09.276 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 1, 2024-04-26T04:19:09.276 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b", 2024-04-26T04:19:09.276 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:19:09.277 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:19:09.277 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:19:09.277 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:19:09.277 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:3300", 2024-04-26T04:19:09.277 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:19:09.277 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:19:09.277 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:19:09.277 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:19:09.277 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:6789", 2024-04-26T04:19:09.277 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:19:09.277 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:19:09.277 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:19:09.277 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:19:09.277 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:6789/0", 2024-04-26T04:19:09.278 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.186:6789/0", 2024-04-26T04:19:09.278 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:19:09.278 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:19:09.278 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:19:09.278 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:19:09.278 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:19:09.278 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 2, 2024-04-26T04:19:09.278 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "c", 2024-04-26T04:19:09.278 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:19:09.278 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:19:09.278 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:19:09.278 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:19:09.278 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:3301", 2024-04-26T04:19:09.278 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:19:09.279 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:19:09.279 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:19:09.279 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:19:09.279 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6790", 2024-04-26T04:19:09.279 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:19:09.279 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:19:09.279 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:19:09.279 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:19:09.279 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6790/0", 2024-04-26T04:19:09.279 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6790/0", 2024-04-26T04:19:09.279 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:19:09.279 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:19:09.279 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:19:09.279 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:19:09.279 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:19:09.280 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:19:09.280 INFO:teuthology.orchestra.run.smithi063.stdout: "feature_map": { 2024-04-26T04:19:09.280 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": [ 2024-04-26T04:19:09.280 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:19:09.280 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:19:09.280 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:19:09.280 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 1 2024-04-26T04:19:09.280 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:19:09.280 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:19:09.280 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": [ 2024-04-26T04:19:09.280 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:19:09.280 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:19:09.280 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:19:09.280 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 3 2024-04-26T04:19:09.281 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:19:09.281 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:19:09.281 INFO:teuthology.orchestra.run.smithi063.stdout: "client": [ 2024-04-26T04:19:09.281 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:19:09.281 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:19:09.281 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:19:09.281 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 6 2024-04-26T04:19:09.281 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:19:09.281 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:19:09.281 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": [ 2024-04-26T04:19:09.281 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:19:09.281 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:19:09.281 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:19:09.281 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 1 2024-04-26T04:19:09.281 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:19:09.282 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:19:09.282 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:19:09.282 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": false 2024-04-26T04:19:09.282 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-26T04:19:09.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.262+0000 7fe89f0f7640 1 -- 172.21.15.63:0/3068918447 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fe884035e70 msgr2=0x7fe884038330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:19:09.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.262+0000 7fe89f0f7640 1 --2- 172.21.15.63:0/3068918447 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fe884035e70 0x7fe884038330 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe8880ab000 tx=0x7fe888008040 comp rx=0 tx=0).stop 2024-04-26T04:19:09.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.262+0000 7fe89f0f7640 10 monclient: shutdown 2024-04-26T04:19:09.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.262+0000 7fe89f0f7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:19:09.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.262+0000 7fe89f0f7640 1 -- 172.21.15.63:0/3068918447 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe8980a2cd0 msgr2=0x7fe8981355a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:19:09.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.262+0000 7fe89f0f7640 1 --2- 172.21.15.63:0/3068918447 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe8980a2cd0 0x7fe8981355a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe89000b9e0 tx=0x7fe89000bea0 comp rx=0 tx=0).stop 2024-04-26T04:19:09.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.262+0000 7fe89d694640 1 -- 172.21.15.63:0/3068918447 reap_dead start 2024-04-26T04:19:09.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.262+0000 7fe89d694640 1 -- 172.21.15.63:0/3068918447 reap_dead start 2024-04-26T04:19:09.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.262+0000 7fe89f0f7640 1 -- 172.21.15.63:0/3068918447 shutdown_connections 2024-04-26T04:19:09.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.262+0000 7fe89f0f7640 1 -- 172.21.15.63:0/3068918447 >> 172.21.15.63:0/3068918447 conn(0x7fe8980a5f40 msgr2=0x7fe8980a1a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:19:09.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.262+0000 7fe89f0f7640 1 -- 172.21.15.63:0/3068918447 shutdown_connections 2024-04-26T04:19:09.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:19:09.262+0000 7fe89f0f7640 1 -- 172.21.15.63:0/3068918447 wait complete. 2024-04-26T04:19:09.285 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-26T04:19:09.285 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-04-26T04:19:09.606 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[27718]: starting. 2024-04-26T04:19:09.606 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[27718]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-27716 2024-04-26T04:19:09.606 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[27718]: shutting down. 2024-04-26T04:19:11.974 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:19:11.974 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[27751]: starting. 2024-04-26T04:19:11.974 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[27751]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-27716 2024-04-26T04:19:11.974 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[27751]: created object 0... 2024-04-26T04:19:11.974 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[27751]: created object 25... 2024-04-26T04:19:11.974 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[27751]: created object 49... 2024-04-26T04:19:11.974 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[27751]: finishing. 2024-04-26T04:19:11.974 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[27751]: shutting down. 2024-04-26T04:19:12.245 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:19:12.245 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[27753]: starting. 2024-04-26T04:19:12.245 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[27753]: listing objects. 2024-04-26T04:19:12.245 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[27753]: listed object 0... 2024-04-26T04:19:12.245 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[27753]: listed object 25... 2024-04-26T04:19:12.245 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[27753]: saw 50 objects 2024-04-26T04:19:12.245 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[27753]: shutting down. 2024-04-26T04:19:12.759 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:19:12.759 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[27752]: starting. 2024-04-26T04:19:12.759 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[27752]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-27716 2024-04-26T04:19:12.759 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[27752]: shutting down. 2024-04-26T04:19:12.762 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:19:12.762 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:19:12.762 INFO:teuthology.orchestra.run.smithi186.stdout:******* SUCCESS ********** 2024-04-26T04:19:12.765 DEBUG:teuthology.orchestra.run.smithi186:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-26T04:19:14.997 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[27804]: starting. 2024-04-26T04:19:14.997 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[27804]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-27803 2024-04-26T04:19:14.997 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[27804]: created object 0... 2024-04-26T04:19:14.997 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[27804]: created object 25... 2024-04-26T04:19:14.997 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[27804]: created object 49... 2024-04-26T04:19:14.997 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[27804]: finishing. 2024-04-26T04:19:14.997 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[27804]: shutting down. 2024-04-26T04:19:15.783 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[27805]: starting. 2024-04-26T04:19:15.783 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[27805]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-27803 2024-04-26T04:19:15.783 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[27805]: shutting down. 2024-04-26T04:19:18.022 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:19:18.022 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[27838]: starting. 2024-04-26T04:19:18.022 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[27838]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-27803 2024-04-26T04:19:18.023 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[27838]: created object 0... 2024-04-26T04:19:18.023 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[27838]: created object 25... 2024-04-26T04:19:18.023 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[27838]: created object 49... 2024-04-26T04:19:18.023 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[27838]: finishing. 2024-04-26T04:19:18.023 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[27838]: shutting down. 2024-04-26T04:19:18.112 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:19:18.113 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[27840]: starting. 2024-04-26T04:19:18.113 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[27840]: listing objects. 2024-04-26T04:19:18.113 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[27840]: listed object 0... 2024-04-26T04:19:18.113 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[27840]: listed object 25... 2024-04-26T04:19:18.113 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[27840]: saw 50 objects 2024-04-26T04:19:18.113 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[27840]: shutting down. 2024-04-26T04:19:18.804 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:19:18.804 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[27839]: starting. 2024-04-26T04:19:18.804 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[27839]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-27803 2024-04-26T04:19:18.805 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[27839]: shutting down. 2024-04-26T04:19:18.808 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:19:18.808 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:19:18.808 INFO:teuthology.orchestra.run.smithi186.stdout:******* SUCCESS ********** 2024-04-26T04:19:18.812 DEBUG:teuthology.orchestra.run.smithi186:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-26T04:19:21.101 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[27891]: starting. 2024-04-26T04:19:21.101 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[27891]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-27890 2024-04-26T04:19:21.101 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[27891]: created object 0... 2024-04-26T04:19:21.101 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[27891]: created object 25... 2024-04-26T04:19:21.101 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[27891]: created object 49... 2024-04-26T04:19:21.101 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[27891]: finishing. 2024-04-26T04:19:21.101 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[27891]: shutting down. 2024-04-26T04:19:21.894 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[27892]: starting. 2024-04-26T04:19:21.894 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[27892]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-27890 2024-04-26T04:19:21.894 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[27892]: shutting down. 2024-04-26T04:19:24.200 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:19:24.200 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[27925]: starting. 2024-04-26T04:19:24.200 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[27925]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-27890 2024-04-26T04:19:24.201 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[27925]: created object 0... 2024-04-26T04:19:24.201 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[27925]: created object 25... 2024-04-26T04:19:24.201 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[27925]: created object 49... 2024-04-26T04:19:24.201 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[27925]: finishing. 2024-04-26T04:19:24.201 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[27925]: shutting down. 2024-04-26T04:19:24.253 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:19:24.254 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[27927]: starting. 2024-04-26T04:19:24.254 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[27927]: listing objects. 2024-04-26T04:19:24.254 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[27927]: listed object 0... 2024-04-26T04:19:24.254 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[27927]: listed object 25... 2024-04-26T04:19:24.254 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[27927]: saw 50 objects 2024-04-26T04:19:24.254 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[27927]: shutting down. 2024-04-26T04:19:24.948 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:19:24.948 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[27926]: starting. 2024-04-26T04:19:24.948 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[27926]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-27890 2024-04-26T04:19:24.948 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[27926]: shutting down. 2024-04-26T04:19:24.951 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:19:24.952 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:19:24.952 INFO:teuthology.orchestra.run.smithi186.stdout:******* SUCCESS ********** 2024-04-26T04:19:24.955 DEBUG:teuthology.orchestra.run.smithi186:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-26T04:19:27.181 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[27978]: starting. 2024-04-26T04:19:27.181 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[27978]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-27977 2024-04-26T04:19:27.181 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[27978]: created object 0... 2024-04-26T04:19:27.181 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[27978]: created object 25... 2024-04-26T04:19:27.182 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[27978]: created object 49... 2024-04-26T04:19:27.182 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[27978]: finishing. 2024-04-26T04:19:27.182 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[27978]: shutting down. 2024-04-26T04:19:27.972 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[27979]: starting. 2024-04-26T04:19:27.972 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[27979]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-27977 2024-04-26T04:19:27.972 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[27979]: shutting down. 2024-04-26T04:19:30.227 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:19:30.227 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28012]: starting. 2024-04-26T04:19:30.227 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28012]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-27977 2024-04-26T04:19:30.227 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28012]: created object 0... 2024-04-26T04:19:30.227 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28012]: created object 25... 2024-04-26T04:19:30.227 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28012]: created object 49... 2024-04-26T04:19:30.227 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28012]: finishing. 2024-04-26T04:19:30.227 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28012]: shutting down. 2024-04-26T04:19:30.284 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:19:30.284 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28014]: starting. 2024-04-26T04:19:30.284 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28014]: listing objects. 2024-04-26T04:19:30.285 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28014]: listed object 0... 2024-04-26T04:19:30.285 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28014]: listed object 25... 2024-04-26T04:19:30.285 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28014]: saw 50 objects 2024-04-26T04:19:30.285 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28014]: shutting down. 2024-04-26T04:19:31.024 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:19:31.025 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[28013]: starting. 2024-04-26T04:19:31.025 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[28013]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-27977 2024-04-26T04:19:31.025 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[28013]: shutting down. 2024-04-26T04:19:31.028 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:19:31.028 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:19:31.028 INFO:teuthology.orchestra.run.smithi186.stdout:******* SUCCESS ********** 2024-04-26T04:19:31.030 DEBUG:teuthology.orchestra.run.smithi186:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-26T04:19:33.367 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28065]: starting. 2024-04-26T04:19:33.367 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28065]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-28064 2024-04-26T04:19:33.367 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28065]: created object 0... 2024-04-26T04:19:33.367 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28065]: created object 25... 2024-04-26T04:19:33.367 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28065]: created object 49... 2024-04-26T04:19:33.367 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28065]: finishing. 2024-04-26T04:19:33.367 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28065]: shutting down. 2024-04-26T04:19:33.882 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[28066]: starting. 2024-04-26T04:19:33.883 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[28066]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-28064 2024-04-26T04:19:33.883 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[28066]: shutting down. 2024-04-26T04:19:36.108 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:19:36.108 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28099]: starting. 2024-04-26T04:19:36.108 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28099]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-28064 2024-04-26T04:19:36.108 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28099]: created object 0... 2024-04-26T04:19:36.108 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28099]: created object 25... 2024-04-26T04:19:36.109 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28099]: created object 49... 2024-04-26T04:19:36.109 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28099]: finishing. 2024-04-26T04:19:36.109 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28099]: shutting down. 2024-04-26T04:19:36.172 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:19:36.172 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28101]: starting. 2024-04-26T04:19:36.172 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28101]: listing objects. 2024-04-26T04:19:36.172 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28101]: listed object 0... 2024-04-26T04:19:36.172 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28101]: listed object 25... 2024-04-26T04:19:36.172 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28101]: saw 50 objects 2024-04-26T04:19:36.172 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28101]: shutting down. 2024-04-26T04:19:36.901 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:19:36.901 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[28100]: starting. 2024-04-26T04:19:36.902 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[28100]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-28064 2024-04-26T04:19:36.902 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[28100]: shutting down. 2024-04-26T04:19:36.904 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:19:36.905 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:19:36.905 INFO:teuthology.orchestra.run.smithi186.stdout:******* SUCCESS ********** 2024-04-26T04:19:36.908 DEBUG:teuthology.orchestra.run.smithi186:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-26T04:19:39.315 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28152]: starting. 2024-04-26T04:19:39.315 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28152]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-28151 2024-04-26T04:19:39.315 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28152]: created object 0... 2024-04-26T04:19:39.315 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28152]: created object 25... 2024-04-26T04:19:39.315 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28152]: created object 49... 2024-04-26T04:19:39.315 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28152]: finishing. 2024-04-26T04:19:39.315 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28152]: shutting down. 2024-04-26T04:19:39.895 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[28153]: starting. 2024-04-26T04:19:39.895 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[28153]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-28151 2024-04-26T04:19:39.895 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[28153]: shutting down. 2024-04-26T04:19:42.162 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:19:42.162 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28186]: starting. 2024-04-26T04:19:42.163 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28186]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-28151 2024-04-26T04:19:42.163 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28186]: created object 0... 2024-04-26T04:19:42.163 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28186]: created object 25... 2024-04-26T04:19:42.163 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28186]: created object 49... 2024-04-26T04:19:42.163 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28186]: finishing. 2024-04-26T04:19:42.163 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28186]: shutting down. 2024-04-26T04:19:42.226 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:19:42.227 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28188]: starting. 2024-04-26T04:19:42.227 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28188]: listing objects. 2024-04-26T04:19:42.227 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28188]: listed object 0... 2024-04-26T04:19:42.227 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28188]: listed object 25... 2024-04-26T04:19:42.227 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28188]: saw 50 objects 2024-04-26T04:19:42.227 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28188]: shutting down. 2024-04-26T04:19:42.947 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:19:42.947 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[28187]: starting. 2024-04-26T04:19:42.947 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[28187]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-28151 2024-04-26T04:19:42.947 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[28187]: shutting down. 2024-04-26T04:19:42.949 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:19:42.949 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:19:42.949 INFO:teuthology.orchestra.run.smithi186.stdout:******* SUCCESS ********** 2024-04-26T04:19:42.952 DEBUG:teuthology.orchestra.run.smithi186:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-26T04:19:45.192 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28239]: starting. 2024-04-26T04:19:45.192 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28239]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-28238 2024-04-26T04:19:45.192 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28239]: created object 0... 2024-04-26T04:19:45.192 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28239]: created object 25... 2024-04-26T04:19:45.192 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28239]: created object 49... 2024-04-26T04:19:45.193 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28239]: finishing. 2024-04-26T04:19:45.193 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28239]: shutting down. 2024-04-26T04:19:45.980 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[28240]: starting. 2024-04-26T04:19:45.980 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[28240]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-28238 2024-04-26T04:19:45.980 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[28240]: shutting down. 2024-04-26T04:19:48.206 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:19:48.206 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28273]: starting. 2024-04-26T04:19:48.206 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28273]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-28238 2024-04-26T04:19:48.206 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28273]: created object 0... 2024-04-26T04:19:48.206 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28273]: created object 25... 2024-04-26T04:19:48.206 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28273]: created object 49... 2024-04-26T04:19:48.206 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28273]: finishing. 2024-04-26T04:19:48.207 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28273]: shutting down. 2024-04-26T04:19:48.270 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:19:48.270 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28275]: starting. 2024-04-26T04:19:48.270 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28275]: listing objects. 2024-04-26T04:19:48.270 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28275]: listed object 0... 2024-04-26T04:19:48.271 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28275]: listed object 25... 2024-04-26T04:19:48.271 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28275]: saw 50 objects 2024-04-26T04:19:48.271 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28275]: shutting down. 2024-04-26T04:19:48.885 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:19:48.886 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[28274]: starting. 2024-04-26T04:19:48.886 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[28274]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-28238 2024-04-26T04:19:48.886 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[28274]: shutting down. 2024-04-26T04:19:48.889 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:19:48.889 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:19:48.889 INFO:teuthology.orchestra.run.smithi186.stdout:******* SUCCESS ********** 2024-04-26T04:19:48.893 DEBUG:teuthology.orchestra.run.smithi186:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-26T04:19:51.126 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28326]: starting. 2024-04-26T04:19:51.126 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28326]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-28325 2024-04-26T04:19:51.127 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28326]: created object 0... 2024-04-26T04:19:51.127 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28326]: created object 25... 2024-04-26T04:19:51.127 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28326]: created object 49... 2024-04-26T04:19:51.127 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28326]: finishing. 2024-04-26T04:19:51.127 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28326]: shutting down. 2024-04-26T04:19:51.928 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[28327]: starting. 2024-04-26T04:19:51.928 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[28327]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-28325 2024-04-26T04:19:51.928 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[28327]: shutting down. 2024-04-26T04:19:54.161 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:19:54.161 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28360]: starting. 2024-04-26T04:19:54.161 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28360]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-28325 2024-04-26T04:19:54.161 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28360]: created object 0... 2024-04-26T04:19:54.161 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28360]: created object 25... 2024-04-26T04:19:54.161 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28360]: created object 49... 2024-04-26T04:19:54.161 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28360]: finishing. 2024-04-26T04:19:54.162 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28360]: shutting down. 2024-04-26T04:19:54.219 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:19:54.219 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28362]: starting. 2024-04-26T04:19:54.219 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28362]: listing objects. 2024-04-26T04:19:54.219 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28362]: listed object 0... 2024-04-26T04:19:54.219 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28362]: listed object 25... 2024-04-26T04:19:54.220 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28362]: saw 50 objects 2024-04-26T04:19:54.220 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28362]: shutting down. 2024-04-26T04:19:54.943 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:19:54.943 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[28361]: starting. 2024-04-26T04:19:54.943 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[28361]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-28325 2024-04-26T04:19:54.943 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[28361]: shutting down. 2024-04-26T04:19:54.945 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:19:54.945 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:19:54.946 INFO:teuthology.orchestra.run.smithi186.stdout:******* SUCCESS ********** 2024-04-26T04:19:54.949 DEBUG:teuthology.orchestra.run.smithi186:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-26T04:19:57.180 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28413]: starting. 2024-04-26T04:19:57.180 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28413]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-28412 2024-04-26T04:19:57.180 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28413]: created object 0... 2024-04-26T04:19:57.180 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28413]: created object 25... 2024-04-26T04:19:57.180 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28413]: created object 49... 2024-04-26T04:19:57.180 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28413]: finishing. 2024-04-26T04:19:57.180 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28413]: shutting down. 2024-04-26T04:19:57.984 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[28414]: starting. 2024-04-26T04:19:57.984 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[28414]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-28412 2024-04-26T04:19:57.984 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[28414]: shutting down. 2024-04-26T04:20:00.106 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:20:00.106 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28447]: starting. 2024-04-26T04:20:00.106 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28447]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-28412 2024-04-26T04:20:00.106 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28447]: created object 0... 2024-04-26T04:20:00.107 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28447]: created object 25... 2024-04-26T04:20:00.107 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28447]: created object 49... 2024-04-26T04:20:00.107 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28447]: finishing. 2024-04-26T04:20:00.107 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28447]: shutting down. 2024-04-26T04:20:00.169 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:20:00.169 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28449]: starting. 2024-04-26T04:20:00.169 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28449]: listing objects. 2024-04-26T04:20:00.170 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28449]: listed object 0... 2024-04-26T04:20:00.170 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28449]: listed object 25... 2024-04-26T04:20:00.170 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28449]: saw 50 objects 2024-04-26T04:20:00.170 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28449]: shutting down. 2024-04-26T04:20:00.920 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:20:00.920 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[28448]: starting. 2024-04-26T04:20:00.920 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[28448]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-28412 2024-04-26T04:20:00.920 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[28448]: shutting down. 2024-04-26T04:20:00.924 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:20:00.924 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:20:00.924 INFO:teuthology.orchestra.run.smithi186.stdout:******* SUCCESS ********** 2024-04-26T04:20:00.927 DEBUG:teuthology.orchestra.run.smithi186:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-26T04:20:03.149 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28500]: starting. 2024-04-26T04:20:03.149 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28500]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-28499 2024-04-26T04:20:03.149 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28500]: created object 0... 2024-04-26T04:20:03.149 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28500]: created object 25... 2024-04-26T04:20:03.149 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28500]: created object 49... 2024-04-26T04:20:03.149 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28500]: finishing. 2024-04-26T04:20:03.149 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28500]: shutting down. 2024-04-26T04:20:03.959 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[28501]: starting. 2024-04-26T04:20:03.959 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[28501]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-28499 2024-04-26T04:20:03.959 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[28501]: shutting down. 2024-04-26T04:20:06.180 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:20:06.180 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28534]: starting. 2024-04-26T04:20:06.180 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28534]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-28499 2024-04-26T04:20:06.180 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28534]: created object 0... 2024-04-26T04:20:06.180 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28534]: created object 25... 2024-04-26T04:20:06.180 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28534]: created object 49... 2024-04-26T04:20:06.180 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28534]: finishing. 2024-04-26T04:20:06.180 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28534]: shutting down. 2024-04-26T04:20:07.082 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:20:07.082 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28536]: starting. 2024-04-26T04:20:07.082 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28536]: listing objects. 2024-04-26T04:20:07.082 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28536]: listed object 0... 2024-04-26T04:20:07.082 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28536]: listed object 25... 2024-04-26T04:20:07.082 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28536]: saw 50 objects 2024-04-26T04:20:07.082 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28536]: shutting down. 2024-04-26T04:20:07.991 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:20:07.991 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[28535]: starting. 2024-04-26T04:20:07.991 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[28535]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-28499 2024-04-26T04:20:07.992 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[28535]: shutting down. 2024-04-26T04:20:07.994 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:20:07.995 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:20:07.995 INFO:teuthology.orchestra.run.smithi186.stdout:******* SUCCESS ********** 2024-04-26T04:20:07.999 DEBUG:teuthology.orchestra.run.smithi186:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-26T04:20:10.104 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28587]: starting. 2024-04-26T04:20:10.104 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28587]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-28586 2024-04-26T04:20:10.104 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28587]: created object 0... 2024-04-26T04:20:10.104 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28587]: created object 25... 2024-04-26T04:20:10.104 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28587]: created object 49... 2024-04-26T04:20:10.104 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28587]: finishing. 2024-04-26T04:20:10.104 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28587]: shutting down. 2024-04-26T04:20:10.911 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[28588]: starting. 2024-04-26T04:20:10.911 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[28588]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-28586 2024-04-26T04:20:10.911 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[28588]: shutting down. 2024-04-26T04:20:13.129 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:20:13.129 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28621]: starting. 2024-04-26T04:20:13.129 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28621]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-28586 2024-04-26T04:20:13.129 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28621]: created object 0... 2024-04-26T04:20:13.129 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28621]: created object 25... 2024-04-26T04:20:13.129 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28621]: created object 49... 2024-04-26T04:20:13.129 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28621]: finishing. 2024-04-26T04:20:13.129 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28621]: shutting down. 2024-04-26T04:20:13.187 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:20:13.187 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28623]: starting. 2024-04-26T04:20:13.187 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28623]: listing objects. 2024-04-26T04:20:13.187 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28623]: listed object 0... 2024-04-26T04:20:13.187 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28623]: listed object 25... 2024-04-26T04:20:13.187 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28623]: saw 50 objects 2024-04-26T04:20:13.187 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28623]: shutting down. 2024-04-26T04:20:13.932 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:20:13.933 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[28622]: starting. 2024-04-26T04:20:13.933 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[28622]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-28586 2024-04-26T04:20:13.933 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[28622]: shutting down. 2024-04-26T04:20:13.935 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:20:13.935 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:20:13.935 INFO:teuthology.orchestra.run.smithi186.stdout:******* SUCCESS ********** 2024-04-26T04:20:13.938 DEBUG:teuthology.orchestra.run.smithi186:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-26T04:20:16.171 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28674]: starting. 2024-04-26T04:20:16.171 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28674]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-28673 2024-04-26T04:20:16.171 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28674]: created object 0... 2024-04-26T04:20:16.171 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28674]: created object 25... 2024-04-26T04:20:16.172 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28674]: created object 49... 2024-04-26T04:20:16.172 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28674]: finishing. 2024-04-26T04:20:16.172 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28674]: shutting down. 2024-04-26T04:20:17.479 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[28675]: starting. 2024-04-26T04:20:17.479 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[28675]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-28673 2024-04-26T04:20:17.479 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[28675]: shutting down. 2024-04-26T04:20:19.111 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:20:19.111 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28708]: starting. 2024-04-26T04:20:19.112 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28708]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-28673 2024-04-26T04:20:19.112 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28708]: created object 0... 2024-04-26T04:20:19.112 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28708]: created object 25... 2024-04-26T04:20:19.112 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28708]: created object 49... 2024-04-26T04:20:19.112 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28708]: finishing. 2024-04-26T04:20:19.112 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28708]: shutting down. 2024-04-26T04:20:19.168 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:20:19.169 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28710]: starting. 2024-04-26T04:20:19.169 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28710]: listing objects. 2024-04-26T04:20:19.169 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28710]: listed object 0... 2024-04-26T04:20:19.169 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28710]: listed object 25... 2024-04-26T04:20:19.169 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28710]: saw 50 objects 2024-04-26T04:20:19.169 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28710]: shutting down. 2024-04-26T04:20:19.908 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:20:19.908 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[28709]: starting. 2024-04-26T04:20:19.909 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[28709]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-28673 2024-04-26T04:20:19.909 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[28709]: shutting down. 2024-04-26T04:20:19.910 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:20:19.910 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:20:19.910 INFO:teuthology.orchestra.run.smithi186.stdout:******* SUCCESS ********** 2024-04-26T04:20:19.913 DEBUG:teuthology.orchestra.run.smithi186:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-26T04:20:22.121 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28761]: starting. 2024-04-26T04:20:22.121 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28761]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-28760 2024-04-26T04:20:22.121 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28761]: created object 0... 2024-04-26T04:20:22.121 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28761]: created object 25... 2024-04-26T04:20:22.121 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28761]: created object 49... 2024-04-26T04:20:22.121 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28761]: finishing. 2024-04-26T04:20:22.121 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28761]: shutting down. 2024-04-26T04:20:22.921 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[28762]: starting. 2024-04-26T04:20:22.921 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[28762]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-28760 2024-04-26T04:20:22.921 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[28762]: shutting down. 2024-04-26T04:20:25.187 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:20:25.187 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28795]: starting. 2024-04-26T04:20:25.187 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28795]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-28760 2024-04-26T04:20:25.187 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28795]: created object 0... 2024-04-26T04:20:25.187 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28795]: created object 25... 2024-04-26T04:20:25.187 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28795]: created object 49... 2024-04-26T04:20:25.188 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28795]: finishing. 2024-04-26T04:20:25.188 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28795]: shutting down. 2024-04-26T04:20:25.250 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:20:25.250 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28797]: starting. 2024-04-26T04:20:25.250 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28797]: listing objects. 2024-04-26T04:20:25.250 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28797]: listed object 0... 2024-04-26T04:20:25.250 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28797]: listed object 25... 2024-04-26T04:20:25.250 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28797]: saw 50 objects 2024-04-26T04:20:25.250 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28797]: shutting down. 2024-04-26T04:20:25.989 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:20:25.989 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[28796]: starting. 2024-04-26T04:20:25.989 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[28796]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-28760 2024-04-26T04:20:25.989 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[28796]: shutting down. 2024-04-26T04:20:25.991 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:20:25.991 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:20:25.991 INFO:teuthology.orchestra.run.smithi186.stdout:******* SUCCESS ********** 2024-04-26T04:20:25.994 DEBUG:teuthology.orchestra.run.smithi186:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-26T04:20:28.359 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28848]: starting. 2024-04-26T04:20:28.360 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28848]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-28847 2024-04-26T04:20:28.360 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28848]: created object 0... 2024-04-26T04:20:28.360 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28848]: created object 25... 2024-04-26T04:20:28.360 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28848]: created object 49... 2024-04-26T04:20:28.360 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28848]: finishing. 2024-04-26T04:20:28.360 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28848]: shutting down. 2024-04-26T04:20:28.918 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[28849]: starting. 2024-04-26T04:20:28.918 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[28849]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-28847 2024-04-26T04:20:28.918 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[28849]: shutting down. 2024-04-26T04:20:31.121 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:20:31.122 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28882]: starting. 2024-04-26T04:20:31.122 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28882]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-28847 2024-04-26T04:20:31.122 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28882]: created object 0... 2024-04-26T04:20:31.122 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28882]: created object 25... 2024-04-26T04:20:31.122 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28882]: created object 49... 2024-04-26T04:20:31.122 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28882]: finishing. 2024-04-26T04:20:31.122 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28882]: shutting down. 2024-04-26T04:20:31.183 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:20:31.183 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28884]: starting. 2024-04-26T04:20:31.183 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28884]: listing objects. 2024-04-26T04:20:31.183 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28884]: listed object 0... 2024-04-26T04:20:31.184 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28884]: listed object 25... 2024-04-26T04:20:31.184 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28884]: saw 50 objects 2024-04-26T04:20:31.184 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28884]: shutting down. 2024-04-26T04:20:32.088 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:20:32.088 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[28883]: starting. 2024-04-26T04:20:32.088 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[28883]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-28847 2024-04-26T04:20:32.088 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[28883]: shutting down. 2024-04-26T04:20:32.091 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:20:32.091 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:20:32.091 INFO:teuthology.orchestra.run.smithi186.stdout:******* SUCCESS ********** 2024-04-26T04:20:32.094 DEBUG:teuthology.orchestra.run.smithi186:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-26T04:20:34.323 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28935]: starting. 2024-04-26T04:20:34.324 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28935]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-28934 2024-04-26T04:20:34.324 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28935]: created object 0... 2024-04-26T04:20:34.324 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28935]: created object 25... 2024-04-26T04:20:34.324 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28935]: created object 49... 2024-04-26T04:20:34.324 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28935]: finishing. 2024-04-26T04:20:34.324 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[28935]: shutting down. 2024-04-26T04:20:35.115 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[28936]: starting. 2024-04-26T04:20:35.115 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[28936]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-28934 2024-04-26T04:20:35.116 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[28936]: shutting down. 2024-04-26T04:20:37.359 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:20:37.359 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28969]: starting. 2024-04-26T04:20:37.359 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28969]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-28934 2024-04-26T04:20:37.359 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28969]: created object 0... 2024-04-26T04:20:37.359 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28969]: created object 25... 2024-04-26T04:20:37.359 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28969]: created object 49... 2024-04-26T04:20:37.359 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28969]: finishing. 2024-04-26T04:20:37.359 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[28969]: shutting down. 2024-04-26T04:20:37.418 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:20:37.418 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28971]: starting. 2024-04-26T04:20:37.418 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28971]: listing objects. 2024-04-26T04:20:37.418 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28971]: listed object 0... 2024-04-26T04:20:37.418 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28971]: listed object 25... 2024-04-26T04:20:37.419 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28971]: saw 50 objects 2024-04-26T04:20:37.419 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[28971]: shutting down. 2024-04-26T04:20:38.147 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:20:38.148 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[28970]: starting. 2024-04-26T04:20:38.148 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[28970]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-28934 2024-04-26T04:20:38.148 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[28970]: shutting down. 2024-04-26T04:20:38.150 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:20:38.150 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:20:38.150 INFO:teuthology.orchestra.run.smithi186.stdout:******* SUCCESS ********** 2024-04-26T04:20:38.155 DEBUG:teuthology.orchestra.run.smithi186:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-26T04:20:39.286 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-04-26T04:20:39.286 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-04-26T04:20:39.286 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-26T04:20:39.286 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-04-26T04:20:39.289 INFO:tasks.ceph.mon.c:Started 2024-04-26T04:20:39.289 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-26T04:20:39.289 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T04:20:39.414 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.409+0000 7f07d71a6640 10 monclient: get_monmap_and_config 2024-04-26T04:20:39.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.409+0000 7f07d71a6640 10 monclient: build_initial_monmap 2024-04-26T04:20:39.415 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.409+0000 7f07d71a6640 10 monclient: monmap: 2024-04-26T04:20:39.415 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:20:39.415 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:39.415 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:20:39.410969+0000 2024-04-26T04:20:39.415 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:20:39.410969+0000 2024-04-26T04:20:39.415 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:20:39.415 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:39.415 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:20:39.416 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:20:39.416 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:20:39.416 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:20:39.416 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:20:39.416 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:39.416 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.409+0000 7f07d71a6640 1 Processor -- start 2024-04-26T04:20:39.416 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.409+0000 7f07d71a6640 1 -- start start 2024-04-26T04:20:39.416 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.409+0000 7f07d71a6640 10 monclient: init 2024-04-26T04:20:39.417 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.409+0000 7f07d71a6640 10 monclient: _reopen_session rank -1 2024-04-26T04:20:39.417 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.409+0000 7f07d71a6640 10 monclient: _add_conns ranks=[0,2,3,1,4] 2024-04-26T04:20:39.417 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.409+0000 7f07d71a6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f07d006be90 addr v1:172.21.15.63:6789/0 2024-04-26T04:20:39.417 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.409+0000 7f07d71a6640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f07d0072910 0x7f07d013c050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:39.417 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.409+0000 7f07d71a6640 10 monclient(hunting): picked mon.noname-b con 0x7f07d0072910 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:20:39.417 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.409+0000 7f07d71a6640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f07d013c590 0x7f07d013e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:39.417 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.409+0000 7f07d71a6640 10 monclient(hunting): picked mon.noname-a con 0x7f07d013c590 addr v2:172.21.15.63:3300/0 2024-04-26T04:20:39.417 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.409+0000 7f07d71a6640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07d006e760 con 0x7f07d006be90 2024-04-26T04:20:39.418 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.409+0000 7f07d71a6640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:39.418 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.409+0000 7f07d71a6640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7f07d0134070 con 0x7f07d013c590 2024-04-26T04:20:39.418 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.409+0000 7f07d71a6640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:39.418 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.409+0000 7f07d71a6640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f07d006e9d0 con 0x7f07d0072910 2024-04-26T04:20:39.418 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.409+0000 7f07d71a6640 10 monclient(hunting): _renew_subs 2024-04-26T04:20:39.418 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.409+0000 7f07d71a6640 10 monclient(hunting): authenticate will time out at 1071.118403s 2024-04-26T04:20:39.418 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.409+0000 7f07cffff640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f07d0072910 0x7f07d013c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:39.418 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.409+0000 7f07d5743640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f07d013c590 0x7f07d013e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:39.418 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.409+0000 7f07d4f42640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f07d006be90 0x7f07d0072200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:48762/0 (socket says 172.21.15.63:48762) 2024-04-26T04:20:39.418 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.409+0000 7f07d4f42640 1 -- 172.21.15.63:0/1521992370 learned_addr learned my addr 172.21.15.63:0/1521992370 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-04-26T04:20:39.418 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.409+0000 7f07cffff640 10 monclient(hunting): get_auth_request con 0x7f07d0072910 auth_method 0 2024-04-26T04:20:39.418 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.409+0000 7f07cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:39.418 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.409+0000 7f07cffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:39.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.409+0000 7f07cffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:20:39.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.409+0000 7f07d5743640 10 monclient(hunting): get_auth_request con 0x7f07d013c590 auth_method 0 2024-04-26T04:20:39.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.409+0000 7f07d5743640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:39.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.409+0000 7f07d5743640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:39.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.409+0000 7f07d5743640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:20:39.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.409+0000 7f07cffff640 1 -- 172.21.15.63:0/1521992370 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f07d0072910 msgr2=0x7f07d013c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T04:20:39.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.409+0000 7f07cffff640 1 -- 172.21.15.63:0/1521992370 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f07d0072910 msgr2=0x7f07d013c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:20:39.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.409+0000 7f07cffff640 1 --2- 172.21.15.63:0/1521992370 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f07d0072910 0x7f07d013c050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:20:39.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.409+0000 7f07d5743640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:20:39.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.409+0000 7f07d5743640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:20:39.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.409+0000 7f07d5743640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:20:39.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.409+0000 7f07cffff640 1 --2- 172.21.15.63:0/1521992370 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f07d0072910 0x7f07d013c050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:20:39.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.409+0000 7f07d5743640 10 monclient(hunting): handle_auth_done global_id 4752 payload 274 2024-04-26T04:20:39.419 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.409+0000 7f07d5743640 1 -- 172.21.15.63:0/1521992370 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f07d0072910 msgr2=0x7f07d013c050 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T04:20:39.420 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.409+0000 7f07d5743640 1 --2- 172.21.15.63:0/1521992370 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f07d0072910 0x7f07d013c050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:39.420 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.409+0000 7f07d5743640 1 -- 172.21.15.63:0/1521992370 >> v1:172.21.15.63:6789/0 conn(0x7f07d006be90 legacy=0x7f07d0072200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:20:39.420 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.409+0000 7f07d5743640 10 monclient: _finish_hunting 0 2024-04-26T04:20:39.420 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.409+0000 7f07d5743640 1 monclient: found mon.noname-a 2024-04-26T04:20:39.420 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.409+0000 7f07d5743640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:20:39.420 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.409+0000 7f07d5743640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:20:39.420 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.409+0000 7f07d5743640 1 -- 172.21.15.63:0/1521992370 --> v2:172.21.15.63:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07d006ecb0 con 0x7f07d013c590 2024-04-26T04:20:39.425 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.413+0000 7f07d5743640 1 -- 172.21.15.63:0/1521992370 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07d013c590 msgr2=0x7f07d013e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:20:39.425 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.413+0000 7f07d5743640 1 --2- 172.21.15.63:0/1521992370 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07d013c590 0x7f07d013e980 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f07c8009d90 tx=0x7f07c801c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa6a57edadbeaf25 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:39.425 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.413+0000 7f07cf7fe640 1 -- 172.21.15.63:0/1521992370 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07c80b2070 con 0x7f07d013c590 2024-04-26T04:20:39.425 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.413+0000 7f07cf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:20:39.425 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.413+0000 7f07cf7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:20:39.425 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.413+0000 7f07cf7fe640 10 monclient: dump: 2024-04-26T04:20:39.425 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:20:39.425 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:39.425 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:39.425 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:39.426 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:20:39.426 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:39.426 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:20:39.426 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:20:39.426 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:20:39.426 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:39.426 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.413+0000 7f07cf7fe640 10 monclient: _finish_auth 0 2024-04-26T04:20:39.426 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.413+0000 7f07cf7fe640 10 monclient: _check_auth_tickets 2024-04-26T04:20:39.426 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.413+0000 7f07cf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:20:39.426 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.413+0000 7f07cf7fe640 1 -- 172.21.15.63:0/1521992370 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07c80a57a0 con 0x7f07d013c590 2024-04-26T04:20:39.426 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.413+0000 7f07cf7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T04:20:39.426 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.413+0000 7f07cf7fe640 1 -- 172.21.15.63:0/1521992370 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07c80a5b70 con 0x7f07d013c590 2024-04-26T04:20:39.426 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.413+0000 7f07cf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:20:39.426 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.413+0000 7f07cf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:20:39.427 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.413+0000 7f07cf7fe640 10 monclient: dump: 2024-04-26T04:20:39.427 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:20:39.427 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:39.427 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:39.427 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:39.427 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:20:39.427 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:39.427 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:20:39.427 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:20:39.427 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:20:39.427 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:39.427 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.413+0000 7f07d71a6640 5 monclient: authenticate success, global_id 4752 2024-04-26T04:20:39.427 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.413+0000 7f07d71a6640 10 monclient: get_monmap_and_config success 2024-04-26T04:20:39.427 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.413+0000 7f07d71a6640 4 set_mon_vals no callback set 2024-04-26T04:20:39.428 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.417+0000 7f07d71a6640 10 monclient: shutdown 2024-04-26T04:20:39.428 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.417+0000 7f07d71a6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:20:39.428 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.417+0000 7f07d71a6640 1 -- 172.21.15.63:0/1521992370 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07d013c590 msgr2=0x7f07d013e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:39.428 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.417+0000 7f07d71a6640 1 --2- 172.21.15.63:0/1521992370 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07d013c590 0x7f07d013e980 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f07c8009d90 tx=0x7f07c801c7f0 comp rx=0 tx=0).stop 2024-04-26T04:20:39.428 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.417+0000 7f07d71a6640 1 -- 172.21.15.63:0/1521992370 shutdown_connections 2024-04-26T04:20:39.428 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.417+0000 7f07d71a6640 1 --2- 172.21.15.63:0/1521992370 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07d013c590 0x7f07d013e980 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:39.428 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.417+0000 7f07d71a6640 1 --2- 172.21.15.63:0/1521992370 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f07d0072910 0x7f07d013c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:39.428 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.417+0000 7f07d71a6640 1 -- 172.21.15.63:0/1521992370 >> 172.21.15.63:0/1521992370 conn(0x7f07d00a60a0 msgr2=0x7f07d00a24b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:20:39.428 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.417+0000 7f07d71a6640 1 -- 172.21.15.63:0/1521992370 shutdown_connections 2024-04-26T04:20:39.428 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.417+0000 7f07d71a6640 1 -- 172.21.15.63:0/1521992370 wait complete. 2024-04-26T04:20:39.428 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.417+0000 7f07d71a6640 10 monclient: build_initial_monmap 2024-04-26T04:20:39.428 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.417+0000 7f07d71a6640 10 monclient: monmap: 2024-04-26T04:20:39.428 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:20:39.428 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:20:39.428 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:20:39.429 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:20:39.429 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:20:39.429 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:39.429 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:20:39.429 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:20:39.429 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:20:39.429 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:39.429 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.417+0000 7f07d71a6640 1 Processor -- start 2024-04-26T04:20:39.429 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.417+0000 7f07d71a6640 1 -- start start 2024-04-26T04:20:39.429 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.417+0000 7f07d71a6640 10 monclient: init 2024-04-26T04:20:39.429 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.417+0000 7f07d71a6640 10 monclient: _reopen_session rank -1 2024-04-26T04:20:39.429 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.417+0000 7f07d71a6640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T04:20:39.429 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.417+0000 7f07d71a6640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f07d006be90 0x7f07d0130500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:39.429 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.417+0000 7f07d71a6640 10 monclient(hunting): picked mon.noname-c con 0x7f07d006be90 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:20:39.429 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.417+0000 7f07d71a6640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07d0072910 0x7f07d0130a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:39.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.417+0000 7f07d71a6640 10 monclient(hunting): picked mon.noname-a con 0x7f07d0072910 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:20:39.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.417+0000 7f07d71a6640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f07d013c590 0x7f07d0137e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:39.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.417+0000 7f07d71a6640 10 monclient(hunting): picked mon.noname-b con 0x7f07d013c590 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:20:39.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.417+0000 7f07d71a6640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:39.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.417+0000 7f07d71a6640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f07d0141860 con 0x7f07d0072910 2024-04-26T04:20:39.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.417+0000 7f07d71a6640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:39.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.417+0000 7f07d71a6640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f07d01416e0 con 0x7f07d013c590 2024-04-26T04:20:39.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.417+0000 7f07d71a6640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:39.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.417+0000 7f07d71a6640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f07d01419e0 con 0x7f07d006be90 2024-04-26T04:20:39.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.417+0000 7f07d71a6640 10 monclient(hunting): _renew_subs 2024-04-26T04:20:39.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.417+0000 7f07d71a6640 10 monclient(hunting): authenticate will time out at 1071.126046s 2024-04-26T04:20:39.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.417+0000 7f07d4f42640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f07d006be90 0x7f07d0130500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:39.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.417+0000 7f07cffff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07d0072910 0x7f07d0130a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:39.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.417+0000 7f07cffff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07d0072910 0x7f07d0130a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:36438/0 (socket says 172.21.15.63:36438) 2024-04-26T04:20:39.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.417+0000 7f07d4f42640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f07d006be90 0x7f07d0130500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:37056/0 (socket says 172.21.15.63:37056) 2024-04-26T04:20:39.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.417+0000 7f07cffff640 1 -- 172.21.15.63:0/3341868972 learned_addr learned my addr 172.21.15.63:0/3341868972 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:20:39.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.417+0000 7f07cffff640 10 monclient(hunting): get_auth_request con 0x7f07d0072910 auth_method 0 2024-04-26T04:20:39.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.417+0000 7f07cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:39.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.417+0000 7f07cffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:39.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.417+0000 7f07cffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:20:39.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.417+0000 7f07d4f42640 10 monclient(hunting): get_auth_request con 0x7f07d006be90 auth_method 0 2024-04-26T04:20:39.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.417+0000 7f07d4f42640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:39.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.417+0000 7f07d4f42640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:39.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.417+0000 7f07d4f42640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:20:39.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.417+0000 7f07cffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:20:39.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.417+0000 7f07cffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:20:39.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.417+0000 7f07cffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:20:39.431 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.417+0000 7f07d4f42640 1 -- 172.21.15.63:0/3341868972 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f07d006be90 msgr2=0x7f07d0130500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T04:20:39.432 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.417+0000 7f07d4f42640 1 -- 172.21.15.63:0/3341868972 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f07d006be90 msgr2=0x7f07d0130500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T04:20:39.432 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.417+0000 7f07d4f42640 1 --2- 172.21.15.63:0/3341868972 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f07d006be90 0x7f07d0130500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:20:39.432 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.417+0000 7f07d4f42640 1 --2- 172.21.15.63:0/3341868972 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f07d006be90 0x7f07d0130500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:20:39.432 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.417+0000 7f07cffff640 10 monclient(hunting): handle_auth_done global_id 4755 payload 824 2024-04-26T04:20:39.432 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.417+0000 7f07cffff640 1 -- 172.21.15.63:0/3341868972 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f07d006be90 msgr2=0x7f07d0130500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T04:20:39.432 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.417+0000 7f07cffff640 1 --2- 172.21.15.63:0/3341868972 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f07d006be90 0x7f07d0130500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:39.432 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.417+0000 7f07cffff640 1 -- 172.21.15.63:0/3341868972 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f07d013c590 msgr2=0x7f07d0137e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:39.432 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.417+0000 7f07cffff640 1 --2- 172.21.15.63:0/3341868972 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f07d013c590 0x7f07d0137e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:39.432 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.417+0000 7f07cffff640 10 monclient: _finish_hunting 0 2024-04-26T04:20:39.432 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.417+0000 7f07cffff640 1 monclient: found mon.noname-a 2024-04-26T04:20:39.432 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.417+0000 7f07cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:20:39.432 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.417+0000 7f07cffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:20:39.432 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.417+0000 7f07cffff640 1 -- 172.21.15.63:0/3341868972 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07d00a4da0 con 0x7f07d0072910 2024-04-26T04:20:39.432 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.421+0000 7f07cffff640 1 -- 172.21.15.63:0/3341868972 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07d0072910 msgr2=0x7f07d0130a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:20:39.433 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.421+0000 7f07cffff640 1 --2- 172.21.15.63:0/3341868972 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07d0072910 0x7f07d0130a40 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f07c400e7e0 tx=0x7f07c400eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:39.433 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.421+0000 7f07cd7fa640 1 -- 172.21.15.63:0/3341868972 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07c4017020 con 0x7f07d0072910 2024-04-26T04:20:39.433 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.421+0000 7f07cd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:20:39.433 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.421+0000 7f07cd7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:20:39.433 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.421+0000 7f07cd7fa640 10 monclient: dump: 2024-04-26T04:20:39.433 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:20:39.433 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:39.433 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:39.433 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:39.433 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:20:39.433 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:39.434 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:20:39.434 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:20:39.434 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:20:39.434 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:39.434 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.421+0000 7f07cd7fa640 10 monclient: _finish_auth 0 2024-04-26T04:20:39.434 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.421+0000 7f07cd7fa640 10 monclient: _check_auth_tickets 2024-04-26T04:20:39.434 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.421+0000 7f07cd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:20:39.434 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.421+0000 7f07cd7fa640 1 -- 172.21.15.63:0/3341868972 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07c400c870 con 0x7f07d0072910 2024-04-26T04:20:39.434 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.421+0000 7f07cd7fa640 10 monclient: handle_config config(0 keys) 2024-04-26T04:20:39.434 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.421+0000 7f07cd7fa640 1 -- 172.21.15.63:0/3341868972 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07c400f040 con 0x7f07d0072910 2024-04-26T04:20:39.434 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.421+0000 7f07d71a6640 5 monclient: authenticate success, global_id 4755 2024-04-26T04:20:39.434 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.421+0000 7f07d71a6640 10 monclient: _renew_subs 2024-04-26T04:20:39.434 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.421+0000 7f07d71a6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:20:39.435 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.421+0000 7f07d71a6640 1 -- 172.21.15.63:0/3341868972 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07d009fd90 con 0x7f07d0072910 2024-04-26T04:20:39.435 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.421+0000 7f07cd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:20:39.435 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.421+0000 7f07cd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:20:39.435 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.421+0000 7f07cd7fa640 10 monclient: dump: 2024-04-26T04:20:39.435 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:20:39.435 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:39.435 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:39.435 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:39.435 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:20:39.435 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:39.435 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:20:39.436 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:20:39.436 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:20:39.436 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:39.436 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.421+0000 7f07d71a6640 10 monclient: _renew_subs 2024-04-26T04:20:39.436 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.421+0000 7f07d71a6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:20:39.436 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.421+0000 7f07d71a6640 1 -- 172.21.15.63:0/3341868972 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07d013acc0 con 0x7f07d0072910 2024-04-26T04:20:39.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.421+0000 7f07ceffd640 4 set_mon_vals no callback set 2024-04-26T04:20:39.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.421+0000 7f07d71a6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:20:39.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.421+0000 7f07d71a6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:20:39.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.421+0000 7f07d71a6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:20:39.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.421+0000 7f07cd7fa640 1 -- 172.21.15.63:0/3341868972 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f07c401a020 con 0x7f07d0072910 2024-04-26T04:20:39.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.421+0000 7f07d71a6640 1 -- 172.21.15.63:0/3341868972 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07d0062e10 con 0x7f07d0072910 2024-04-26T04:20:39.438 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.421+0000 7f07cd7fa640 1 --2- 172.21.15.63:0/3341868972 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f07bc035ec0 0x7f07bc038380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:39.438 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.421+0000 7f07cd7fa640 1 -- 172.21.15.63:0/3341868972 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 4738+0+0 (secure 0 0 0) 0x7f07c4048c50 con 0x7f07d0072910 2024-04-26T04:20:39.438 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.421+0000 7f07d4f42640 1 --2- 172.21.15.63:0/3341868972 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f07bc035ec0 0x7f07bc038380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:39.438 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.421+0000 7f07d4f42640 10 monclient: get_auth_request con 0x7f07bc035ec0 auth_method 0 2024-04-26T04:20:39.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.425+0000 7f07d4f42640 1 --2- 172.21.15.63:0/3341868972 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f07bc035ec0 0x7f07bc038380 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f07c0006e10 tx=0x7f07c0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:39.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.425+0000 7f07cd7fa640 1 -- 172.21.15.63:0/3341868972 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f07c4021020 con 0x7f07d0072910 2024-04-26T04:20:39.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.425+0000 7f07cd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:20:39.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.425+0000 7f07cd7fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:20:39.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.649+0000 7f07d71a6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T04:20:39.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.649+0000 7f07d71a6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T04:20:39.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.649+0000 7f07d71a6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:20:39.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.649+0000 7f07d71a6640 1 -- 172.21.15.63:0/3341868972 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f07d013eca0 con 0x7f07d0072910 2024-04-26T04:20:39.654 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.653+0000 7f07cd7fa640 1 -- 172.21.15.63:0/3341868972 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f07c4047670 con 0x7f07d0072910 2024-04-26T04:20:39.654 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.653+0000 7f07cd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T04:20:39.654 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.653+0000 7f07cd7fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:20:39.654 INFO:teuthology.orchestra.run.smithi063.stdout:{"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cd1c6a0c-4e4b-4014-89c0-2e535479ef39","modified":"2024-04-26T04:18:03.702339Z","created":"2024-04-26T04:18:03.702339Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3300","nonce":0},{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:3300","nonce":0},{"type":"v1","addr":"172.21.15.186:6789","nonce":0}]},"addr":"172.21.15.186:6789/0","public_addr":"172.21.15.186:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3301","nonce":0},{"type":"v1","addr":"172.21.15.63:6790","nonce":0}]},"addr":"172.21.15.63:6790/0","public_addr":"172.21.15.63:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T04:20:39.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.653+0000 7f07d71a6640 1 -- 172.21.15.63:0/3341868972 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f07bc035ec0 msgr2=0x7f07bc038380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:39.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.653+0000 7f07d71a6640 1 --2- 172.21.15.63:0/3341868972 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f07bc035ec0 0x7f07bc038380 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f07c0006e10 tx=0x7f07c0008040 comp rx=0 tx=0).stop 2024-04-26T04:20:39.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.653+0000 7f07d71a6640 10 monclient: shutdown 2024-04-26T04:20:39.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.653+0000 7f07d71a6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:20:39.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.653+0000 7f07d71a6640 1 -- 172.21.15.63:0/3341868972 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07d0072910 msgr2=0x7f07d0130a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:39.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.653+0000 7f07d71a6640 1 --2- 172.21.15.63:0/3341868972 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07d0072910 0x7f07d0130a40 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f07c400e7e0 tx=0x7f07c400eca0 comp rx=0 tx=0).stop 2024-04-26T04:20:39.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.653+0000 7f07d71a6640 1 -- 172.21.15.63:0/3341868972 shutdown_connections 2024-04-26T04:20:39.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.653+0000 7f07d71a6640 1 --2- 172.21.15.63:0/3341868972 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f07bc035ec0 0x7f07bc038380 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:39.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.653+0000 7f07d71a6640 1 --2- 172.21.15.63:0/3341868972 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f07d013c590 0x7f07d0137e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:39.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.653+0000 7f07d71a6640 1 --2- 172.21.15.63:0/3341868972 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07d0072910 0x7f07d0130a40 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:39.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.653+0000 7f07d71a6640 1 --2- 172.21.15.63:0/3341868972 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f07d006be90 0x7f07d0130500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:39.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.653+0000 7f07d71a6640 1 -- 172.21.15.63:0/3341868972 >> 172.21.15.63:0/3341868972 conn(0x7f07d00a60a0 msgr2=0x7f07d013d5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:20:39.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.653+0000 7f07d71a6640 1 -- 172.21.15.63:0/3341868972 shutdown_connections 2024-04-26T04:20:39.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.653+0000 7f07d71a6640 1 -- 172.21.15.63:0/3341868972 wait complete. 2024-04-26T04:20:39.673 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-26T04:20:39.673 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-26T04:20:39.673 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T04:20:39.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.793+0000 7f90a2682640 10 monclient: get_monmap_and_config 2024-04-26T04:20:39.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.793+0000 7f90a2682640 10 monclient: build_initial_monmap 2024-04-26T04:20:39.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.793+0000 7f90a2682640 10 monclient: monmap: 2024-04-26T04:20:39.795 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:20:39.795 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:39.795 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:20:39.794994+0000 2024-04-26T04:20:39.795 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:20:39.794994+0000 2024-04-26T04:20:39.795 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:20:39.795 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:39.795 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:20:39.795 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:20:39.796 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:20:39.796 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:20:39.796 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:20:39.796 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:39.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.793+0000 7f90a2682640 1 Processor -- start 2024-04-26T04:20:39.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.793+0000 7f90a2682640 1 -- start start 2024-04-26T04:20:39.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.793+0000 7f90a2682640 10 monclient: init 2024-04-26T04:20:39.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.793+0000 7f90a2682640 10 monclient: _reopen_session rank -1 2024-04-26T04:20:39.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.793+0000 7f90a2682640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2024-04-26T04:20:39.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.793+0000 7f90a2682640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f909c09b3b0 addr v1:172.21.15.63:6789/0 2024-04-26T04:20:39.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.793+0000 7f90a2682640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f909c137320 0x7f909c139710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:39.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.793+0000 7f90a2682640 10 monclient(hunting): picked mon.noname-b con 0x7f909c137320 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:20:39.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.793+0000 7f90a2682640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f909c139c50 0x7f909c13c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:39.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.793+0000 7f90a2682640 10 monclient(hunting): picked mon.noname-a con 0x7f909c139c50 addr v2:172.21.15.63:3300/0 2024-04-26T04:20:39.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.793+0000 7f90a2682640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f909c09dde0 con 0x7f909c09b3b0 2024-04-26T04:20:39.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.793+0000 7f90a2682640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:39.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.793+0000 7f90a2682640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7f909c09c100 con 0x7f909c139c50 2024-04-26T04:20:39.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.793+0000 7f90a2682640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:39.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.793+0000 7f90a2682640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f909c09b790 con 0x7f909c137320 2024-04-26T04:20:39.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.793+0000 7f90a2682640 10 monclient(hunting): _renew_subs 2024-04-26T04:20:39.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.793+0000 7f90a2682640 10 monclient(hunting): authenticate will time out at 1071.502649s 2024-04-26T04:20:39.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.793+0000 7f90a0c1f640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f909c139c50 0x7f909c13c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:39.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.793+0000 7f90937fe640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f909c137320 0x7f909c139710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:39.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.793+0000 7f90a0c1f640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f909c139c50 0x7f909c13c040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:36458/0 (socket says 172.21.15.63:36458) 2024-04-26T04:20:39.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.793+0000 7f90a0c1f640 1 -- 172.21.15.63:0/3435885999 learned_addr learned my addr 172.21.15.63:0/3435885999 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:20:39.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.793+0000 7f90937fe640 10 monclient(hunting): get_auth_request con 0x7f909c137320 auth_method 0 2024-04-26T04:20:39.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.793+0000 7f90937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:39.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.793+0000 7f90937fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:39.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.793+0000 7f90937fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:20:39.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.793+0000 7f90a0c1f640 10 monclient(hunting): get_auth_request con 0x7f909c139c50 auth_method 0 2024-04-26T04:20:39.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.793+0000 7f90a0c1f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:39.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.793+0000 7f90a0c1f640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:39.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.793+0000 7f90a0c1f640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:20:39.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.793+0000 7f90937fe640 1 -- 172.21.15.63:0/3435885999 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f909c137320 msgr2=0x7f909c139710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T04:20:39.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.793+0000 7f90937fe640 1 -- 172.21.15.63:0/3435885999 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f909c137320 msgr2=0x7f909c139710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:20:39.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.793+0000 7f90937fe640 1 --2- 172.21.15.63:0/3435885999 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f909c137320 0x7f909c139710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:20:39.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.793+0000 7f90a0c1f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:20:39.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.793+0000 7f90a0c1f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:20:39.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.793+0000 7f90a0c1f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:20:39.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.793+0000 7f90937fe640 1 --2- 172.21.15.63:0/3435885999 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f909c137320 0x7f909c139710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:20:39.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.797+0000 7f909bfff640 1 -- 172.21.15.63:0/3435885999 >> v1:172.21.15.63:6789/0 conn(0x7f909c09b3b0 legacy=0x7f909c09e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:20:39.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.797+0000 7f90a0c1f640 10 monclient(hunting): handle_auth_done global_id 4758 payload 274 2024-04-26T04:20:39.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.797+0000 7f90a0c1f640 1 -- 172.21.15.63:0/3435885999 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f909c137320 msgr2=0x7f909c139710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T04:20:39.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.797+0000 7f90a0c1f640 1 --2- 172.21.15.63:0/3435885999 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f909c137320 0x7f909c139710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:39.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.797+0000 7f90a0c1f640 1 -- 172.21.15.63:0/3435885999 >> v1:172.21.15.63:6789/0 conn(0x7f909c09b3b0 legacy=0x7f909c09e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:39.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.797+0000 7f90a0c1f640 10 monclient: _finish_hunting 0 2024-04-26T04:20:39.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.797+0000 7f90a0c1f640 1 monclient: found mon.noname-a 2024-04-26T04:20:39.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.797+0000 7f90a0c1f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:20:39.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.797+0000 7f90a0c1f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:20:39.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.797+0000 7f90a0c1f640 1 -- 172.21.15.63:0/3435885999 --> v2:172.21.15.63:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f909c09c9b0 con 0x7f909c139c50 2024-04-26T04:20:39.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.801+0000 7f90a0c1f640 1 -- 172.21.15.63:0/3435885999 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f909c139c50 msgr2=0x7f909c13c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:20:39.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.801+0000 7f90a0c1f640 1 --2- 172.21.15.63:0/3435885999 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f909c139c50 0x7f909c13c040 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f9094009f00 tx=0x7f909401c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=586a66fd01769d6c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:39.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.801+0000 7f909b7fe640 1 -- 172.21.15.63:0/3435885999 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90940a5720 con 0x7f909c139c50 2024-04-26T04:20:39.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.801+0000 7f909b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:20:39.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.801+0000 7f909b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:20:39.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.801+0000 7f909b7fe640 10 monclient: dump: 2024-04-26T04:20:39.807 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:20:39.807 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:39.807 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:39.807 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:39.807 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:20:39.807 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:39.807 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:20:39.807 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:20:39.807 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:20:39.807 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:39.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.801+0000 7f909b7fe640 10 monclient: _finish_auth 0 2024-04-26T04:20:39.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.801+0000 7f909b7fe640 10 monclient: _check_auth_tickets 2024-04-26T04:20:39.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.801+0000 7f909b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:20:39.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.801+0000 7f90a2682640 5 monclient: authenticate success, global_id 4758 2024-04-26T04:20:39.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.801+0000 7f90a2682640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T04:20:39.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.801+0000 7f909b7fe640 1 -- 172.21.15.63:0/3435885999 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90940a58c0 con 0x7f909c139c50 2024-04-26T04:20:39.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.801+0000 7f909b7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T04:20:39.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.801+0000 7f90a2682640 10 monclient: get_monmap_and_config success 2024-04-26T04:20:39.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.801+0000 7f909b7fe640 1 -- 172.21.15.63:0/3435885999 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90940a5c90 con 0x7f909c139c50 2024-04-26T04:20:39.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.801+0000 7f90a2682640 4 set_mon_vals no callback set 2024-04-26T04:20:39.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.801+0000 7f909b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:20:39.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.801+0000 7f909b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:20:39.808 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.801+0000 7f909b7fe640 10 monclient: dump: 2024-04-26T04:20:39.808 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:20:39.808 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:39.808 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:39.808 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:39.808 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:20:39.808 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:39.808 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:20:39.809 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:20:39.809 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:20:39.809 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:39.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.801+0000 7f90a2682640 10 monclient: shutdown 2024-04-26T04:20:39.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.801+0000 7f90a2682640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:20:39.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.801+0000 7f90a2682640 1 -- 172.21.15.63:0/3435885999 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f909c139c50 msgr2=0x7f909c13c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:39.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.801+0000 7f90a2682640 1 --2- 172.21.15.63:0/3435885999 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f909c139c50 0x7f909c13c040 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f9094009f00 tx=0x7f909401c950 comp rx=0 tx=0).stop 2024-04-26T04:20:39.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.801+0000 7f90a2682640 1 -- 172.21.15.63:0/3435885999 shutdown_connections 2024-04-26T04:20:39.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.801+0000 7f90a2682640 1 --2- 172.21.15.63:0/3435885999 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f909c139c50 0x7f909c13c040 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:39.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.801+0000 7f90a2682640 1 --2- 172.21.15.63:0/3435885999 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f909c137320 0x7f909c139710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:39.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.801+0000 7f90a2682640 1 -- 172.21.15.63:0/3435885999 >> 172.21.15.63:0/3435885999 conn(0x7f909c0a1e70 msgr2=0x7f909c0a2280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:20:39.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.801+0000 7f90a2682640 1 -- 172.21.15.63:0/3435885999 shutdown_connections 2024-04-26T04:20:39.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.801+0000 7f90a2682640 1 -- 172.21.15.63:0/3435885999 wait complete. 2024-04-26T04:20:39.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.801+0000 7f90a2682640 10 monclient: build_initial_monmap 2024-04-26T04:20:39.809 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.801+0000 7f90a2682640 10 monclient: monmap: 2024-04-26T04:20:39.810 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:20:39.810 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:20:39.810 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:20:39.810 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:20:39.810 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:20:39.810 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:39.810 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:20:39.810 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:20:39.810 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:20:39.810 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:39.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.801+0000 7f90a2682640 1 Processor -- start 2024-04-26T04:20:39.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.801+0000 7f90a2682640 1 -- start start 2024-04-26T04:20:39.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.801+0000 7f90a2682640 10 monclient: init 2024-04-26T04:20:39.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.805+0000 7f90a2682640 10 monclient: _reopen_session rank -1 2024-04-26T04:20:39.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.805+0000 7f90a2682640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T04:20:39.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.805+0000 7f90a2682640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f909c09b3b0 0x7f909c1cc9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:39.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.805+0000 7f90a2682640 10 monclient(hunting): picked mon.noname-a con 0x7f909c09b3b0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:20:39.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.805+0000 7f90a2682640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f909c137320 0x7f909c136540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:39.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.805+0000 7f90a2682640 10 monclient(hunting): picked mon.noname-b con 0x7f909c137320 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:20:39.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.805+0000 7f90a2682640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f909c139c50 0x7f909c136a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:39.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.805+0000 7f90a2682640 10 monclient(hunting): picked mon.noname-c con 0x7f909c139c50 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:20:39.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.805+0000 7f90a2682640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:39.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.805+0000 7f90a2682640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f909c13f130 con 0x7f909c09b3b0 2024-04-26T04:20:39.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.805+0000 7f90a2682640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:39.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.805+0000 7f90a2682640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f909c13efb0 con 0x7f909c137320 2024-04-26T04:20:39.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.805+0000 7f90a2682640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:39.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.805+0000 7f90a2682640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f909c13f2b0 con 0x7f909c139c50 2024-04-26T04:20:39.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.805+0000 7f90a2682640 10 monclient(hunting): _renew_subs 2024-04-26T04:20:39.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.805+0000 7f90a2682640 10 monclient(hunting): authenticate will time out at 1071.512546s 2024-04-26T04:20:39.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.805+0000 7f909bfff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f909c09b3b0 0x7f909c1cc9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:39.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.805+0000 7f90a0c1f640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f909c139c50 0x7f909c136a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:39.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.805+0000 7f909bfff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f909c09b3b0 0x7f909c1cc9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:36470/0 (socket says 172.21.15.63:36470) 2024-04-26T04:20:39.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.805+0000 7f90a0c1f640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f909c139c50 0x7f909c136a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:37070/0 (socket says 172.21.15.63:37070) 2024-04-26T04:20:39.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.805+0000 7f909bfff640 1 -- 172.21.15.63:0/3792814066 learned_addr learned my addr 172.21.15.63:0/3792814066 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:20:39.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.805+0000 7f909bfff640 10 monclient(hunting): get_auth_request con 0x7f909c09b3b0 auth_method 0 2024-04-26T04:20:39.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.805+0000 7f909bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:39.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.805+0000 7f909bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:39.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.805+0000 7f909bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:20:39.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.805+0000 7f90a0c1f640 10 monclient(hunting): get_auth_request con 0x7f909c139c50 auth_method 0 2024-04-26T04:20:39.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.805+0000 7f90a0c1f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:39.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.805+0000 7f90a0c1f640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:39.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.805+0000 7f90a0c1f640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:20:39.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.805+0000 7f90a0c1f640 1 -- 172.21.15.63:0/3792814066 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f909c139c50 msgr2=0x7f909c136a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-26T04:20:39.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.805+0000 7f90a0c1f640 1 -- 172.21.15.63:0/3792814066 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f909c139c50 msgr2=0x7f909c136a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T04:20:39.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.805+0000 7f909bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:20:39.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.805+0000 7f90a0c1f640 1 --2- 172.21.15.63:0/3792814066 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f909c139c50 0x7f909c136a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:20:39.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.805+0000 7f909bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:20:39.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.805+0000 7f909bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:20:39.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.805+0000 7f90a0c1f640 1 --2- 172.21.15.63:0/3792814066 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f909c139c50 0x7f909c136a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:20:39.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.805+0000 7f90937fe640 1 --2- 172.21.15.63:0/3792814066 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f909c137320 0x7f909c136540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:39.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.805+0000 7f90937fe640 10 monclient(hunting): get_auth_request con 0x7f909c137320 auth_method 0 2024-04-26T04:20:39.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.805+0000 7f90937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:39.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.805+0000 7f90937fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:39.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.805+0000 7f90937fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:20:39.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.805+0000 7f909bfff640 10 monclient(hunting): handle_auth_done global_id 4761 payload 824 2024-04-26T04:20:39.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.805+0000 7f909bfff640 1 -- 172.21.15.63:0/3792814066 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f909c139c50 msgr2=0x7f909c136a80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T04:20:39.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.805+0000 7f909bfff640 1 --2- 172.21.15.63:0/3792814066 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f909c139c50 0x7f909c136a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:39.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.805+0000 7f909bfff640 1 -- 172.21.15.63:0/3792814066 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f909c137320 msgr2=0x7f909c136540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:39.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.805+0000 7f909bfff640 1 --2- 172.21.15.63:0/3792814066 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f909c137320 0x7f909c136540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:39.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.805+0000 7f909bfff640 10 monclient: _finish_hunting 0 2024-04-26T04:20:39.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.805+0000 7f909bfff640 1 monclient: found mon.noname-a 2024-04-26T04:20:39.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.805+0000 7f909bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:20:39.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.805+0000 7f909bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:20:39.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.805+0000 7f909bfff640 1 -- 172.21.15.63:0/3792814066 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f909c12c150 con 0x7f909c09b3b0 2024-04-26T04:20:39.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.805+0000 7f909bfff640 1 -- 172.21.15.63:0/3792814066 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f909c09b3b0 msgr2=0x7f909c1cc9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:20:39.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.805+0000 7f909bfff640 1 --2- 172.21.15.63:0/3792814066 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f909c09b3b0 0x7f909c1cc9a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f908c00c430 tx=0x7f908c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:39.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.805+0000 7f90997fa640 1 -- 172.21.15.63:0/3792814066 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f908c017070 con 0x7f909c09b3b0 2024-04-26T04:20:39.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.805+0000 7f90997fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:20:39.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.805+0000 7f90997fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:20:39.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.805+0000 7f90997fa640 10 monclient: dump: 2024-04-26T04:20:39.820 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:20:39.820 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:39.820 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:39.820 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:39.820 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:20:39.820 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:39.820 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:20:39.820 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:20:39.820 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:20:39.820 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:39.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.805+0000 7f90997fa640 10 monclient: _finish_auth 0 2024-04-26T04:20:39.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.805+0000 7f90997fa640 10 monclient: _check_auth_tickets 2024-04-26T04:20:39.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.805+0000 7f90997fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:20:39.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.805+0000 7f90997fa640 1 -- 172.21.15.63:0/3792814066 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f908c0057d0 con 0x7f909c09b3b0 2024-04-26T04:20:39.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.805+0000 7f90997fa640 10 monclient: handle_config config(0 keys) 2024-04-26T04:20:39.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.805+0000 7f90997fa640 1 -- 172.21.15.63:0/3792814066 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f908c005bc0 con 0x7f909c09b3b0 2024-04-26T04:20:39.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.805+0000 7f90997fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:20:39.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.805+0000 7f90997fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:20:39.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.805+0000 7f90997fa640 10 monclient: dump: 2024-04-26T04:20:39.821 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:20:39.821 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:39.821 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:39.821 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:39.821 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:20:39.821 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:39.822 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:20:39.822 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:20:39.822 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:20:39.822 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:39.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.805+0000 7f909affd640 4 set_mon_vals no callback set 2024-04-26T04:20:39.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.809+0000 7f90a2682640 5 monclient: authenticate success, global_id 4761 2024-04-26T04:20:39.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.809+0000 7f90a2682640 10 monclient: _renew_subs 2024-04-26T04:20:39.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.809+0000 7f90a2682640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:20:39.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.809+0000 7f90a2682640 1 -- 172.21.15.63:0/3792814066 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f909c139360 con 0x7f909c09b3b0 2024-04-26T04:20:39.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.809+0000 7f90a2682640 10 monclient: _renew_subs 2024-04-26T04:20:39.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.809+0000 7f90a2682640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:20:39.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.809+0000 7f90a2682640 1 -- 172.21.15.63:0/3792814066 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f909c12dfa0 con 0x7f909c09b3b0 2024-04-26T04:20:39.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.809+0000 7f90a2682640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:20:39.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.809+0000 7f90a2682640 1 --2- 172.21.15.63:0/3792814066 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f909c12aa20 0x7f909c02bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:39.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.809+0000 7f90a2682640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:39.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.809+0000 7f90a2682640 1 -- 172.21.15.63:0/3792814066 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f909c0504d0 con 0x7f909c12aa20 2024-04-26T04:20:39.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.809+0000 7f90937fe640 1 --2- 172.21.15.63:0/3792814066 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f909c12aa20 0x7f909c02bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:39.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.809+0000 7f90937fe640 10 monclient: get_auth_request con 0x7f909c12aa20 auth_method 0 2024-04-26T04:20:39.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.809+0000 7f90937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:39.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.809+0000 7f90937fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:39.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.809+0000 7f90937fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:20:39.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.809+0000 7f90997fa640 1 -- 172.21.15.63:0/3792814066 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f908c01a070 con 0x7f909c09b3b0 2024-04-26T04:20:39.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.809+0000 7f90937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:20:39.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.809+0000 7f90937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:20:39.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.809+0000 7f90937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:20:39.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.809+0000 7f90937fe640 10 monclient(hunting): handle_auth_done global_id 4764 payload 824 2024-04-26T04:20:39.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.809+0000 7f90997fa640 1 --2- 172.21.15.63:0/3792814066 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f907c035ec0 0x7f907c038380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:39.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.809+0000 7f90937fe640 1 -- 172.21.15.63:0/3792814066 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f909c12d890 con 0x7f909c12aa20 2024-04-26T04:20:39.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.809+0000 7f90997fa640 1 -- 172.21.15.63:0/3792814066 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 4738+0+0 (secure 0 0 0) 0x7f908c048280 con 0x7f909c09b3b0 2024-04-26T04:20:39.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.809+0000 7f90a0c1f640 1 --2- 172.21.15.63:0/3792814066 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f907c035ec0 0x7f907c038380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:39.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.809+0000 7f90a0c1f640 10 monclient: get_auth_request con 0x7f907c035ec0 auth_method 0 2024-04-26T04:20:39.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.809+0000 7f90937fe640 1 -- 172.21.15.63:0/3792814066 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f909c12aa20 msgr2=0x7f909c02bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:20:39.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.809+0000 7f90937fe640 1 --2- 172.21.15.63:0/3792814066 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f909c12aa20 0x7f909c02bfc0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f909c12bfc0 tx=0x7f9084010c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:39.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.809+0000 7f90997fa640 1 -- 172.21.15.63:0/3792814066 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9084019020 con 0x7f909c12aa20 2024-04-26T04:20:39.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.809+0000 7f90997fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:20:39.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.809+0000 7f90997fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:20:39.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.809+0000 7f90997fa640 10 monclient: dump: 2024-04-26T04:20:39.824 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:20:39.824 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:39.824 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:39.824 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:39.824 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:20:39.824 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:39.824 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:20:39.825 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:20:39.825 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:20:39.825 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:39.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.813+0000 7f90997fa640 1 -- 172.21.15.63:0/3792814066 <== mon.0 v2:172.21.15.63:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f908400edb0 con 0x7f909c12aa20 2024-04-26T04:20:39.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.813+0000 7f90997fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:20:39.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.813+0000 7f90997fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:20:39.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.813+0000 7f90997fa640 1 -- 172.21.15.63:0/3792814066 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f909c12aa20 msgr2=0x7f909c02bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:39.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.813+0000 7f90997fa640 1 --2- 172.21.15.63:0/3792814066 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f909c12aa20 0x7f909c02bfc0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f909c12bfc0 tx=0x7f9084010c20 comp rx=0 tx=0).stop 2024-04-26T04:20:39.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.813+0000 7f90997fa640 1 -- 172.21.15.63:0/3792814066 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f909c12aa20 msgr2=0x7f909c02bfc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:20:39.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.813+0000 7f90997fa640 1 --2- 172.21.15.63:0/3792814066 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f909c12aa20 0x7f909c02bfc0 secure :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f909c12bfc0 tx=0x7f9084010c20 comp rx=0 tx=0).stop 2024-04-26T04:20:39.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.813+0000 7f90a0c1f640 1 --2- 172.21.15.63:0/3792814066 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f907c035ec0 0x7f907c038380 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f909c06b260 tx=0x7f90940a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:39.827 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.817+0000 7f90a2682640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T04:20:39.827 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.817+0000 7f90a2682640 1 --2- 172.21.15.63:0/3792814066 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f909c12ae00 0x7f909c02c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:39.827 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.817+0000 7f90a2682640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:39.827 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.817+0000 7f90a2682640 1 -- 172.21.15.63:0/3792814066 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f909c09cc30 con 0x7f909c12ae00 2024-04-26T04:20:39.827 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.817+0000 7f90937fe640 1 --2- 172.21.15.63:0/3792814066 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f909c12ae00 0x7f909c02c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:39.827 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.817+0000 7f90937fe640 10 monclient: get_auth_request con 0x7f909c12ae00 auth_method 0 2024-04-26T04:20:39.827 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.817+0000 7f90937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:39.827 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.817+0000 7f90937fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:39.827 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.817+0000 7f90937fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:20:39.828 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.817+0000 7f90937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:20:39.828 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.817+0000 7f90937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:20:39.828 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.817+0000 7f90937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:20:39.828 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.817+0000 7f90937fe640 10 monclient(hunting): handle_auth_done global_id 4767 payload 824 2024-04-26T04:20:39.828 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.817+0000 7f90937fe640 1 -- 172.21.15.63:0/3792814066 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f909c062e10 con 0x7f909c12ae00 2024-04-26T04:20:39.828 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.821+0000 7f90937fe640 1 -- 172.21.15.63:0/3792814066 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f909c12ae00 msgr2=0x7f909c02c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:20:39.828 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.821+0000 7f90937fe640 1 --2- 172.21.15.63:0/3792814066 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f909c12ae00 0x7f909c02c500 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f908401a000 tx=0x7f9084004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:39.828 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.821+0000 7f90997fa640 1 -- 172.21.15.63:0/3792814066 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9084019030 con 0x7f909c12ae00 2024-04-26T04:20:39.828 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.821+0000 7f90997fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:20:39.828 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.821+0000 7f90997fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:20:39.828 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.821+0000 7f90997fa640 10 monclient: dump: 2024-04-26T04:20:39.828 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:20:39.828 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:39.828 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:39.828 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:39.829 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:20:39.829 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:39.829 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:20:39.829 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:20:39.829 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:20:39.829 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:39.829 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.821+0000 7f90997fa640 1 -- 172.21.15.63:0/3792814066 <== mon.0 v2:172.21.15.63:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4723 (secure 0 0 0) 0x7f9084015b80 con 0x7f909c12ae00 2024-04-26T04:20:39.829 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.821+0000 7f90997fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T04:20:39.829 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.821+0000 7f90997fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:20:39.829 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.821+0000 7f90997fa640 1 -- 172.21.15.63:0/3792814066 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f909c12ae00 msgr2=0x7f909c02c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:39.829 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.821+0000 7f90997fa640 1 --2- 172.21.15.63:0/3792814066 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f909c12ae00 0x7f909c02c500 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f908401a000 tx=0x7f9084004740 comp rx=0 tx=0).stop 2024-04-26T04:20:39.829 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.821+0000 7f90997fa640 1 -- 172.21.15.63:0/3792814066 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f909c12ae00 msgr2=0x7f909c02c500 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:20:39.829 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.821+0000 7f90997fa640 1 --2- 172.21.15.63:0/3792814066 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f909c12ae00 0x7f909c02c500 secure :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f908401a000 tx=0x7f9084004740 comp rx=0 tx=0).stop 2024-04-26T04:20:39.830 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-26T04:20:39.830 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a", 2024-04-26T04:20:39.830 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 0, 2024-04-26T04:20:39.830 INFO:teuthology.orchestra.run.smithi063.stdout: "state": "leader", 2024-04-26T04:20:39.830 INFO:teuthology.orchestra.run.smithi063.stdout: "election_epoch": 14, 2024-04-26T04:20:39.830 INFO:teuthology.orchestra.run.smithi063.stdout: "uptime": 130974, 2024-04-26T04:20:39.830 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum": [ 2024-04-26T04:20:39.830 INFO:teuthology.orchestra.run.smithi063.stdout: 0, 2024-04-26T04:20:39.830 INFO:teuthology.orchestra.run.smithi063.stdout: 1, 2024-04-26T04:20:39.830 INFO:teuthology.orchestra.run.smithi063.stdout: 2 2024-04-26T04:20:39.830 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:20:39.830 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_age": 0, 2024-04-26T04:20:39.831 INFO:teuthology.orchestra.run.smithi063.stdout: "features": { 2024-04-26T04:20:39.831 INFO:teuthology.orchestra.run.smithi063.stdout: "required_con": "2450521708007866372", 2024-04-26T04:20:39.831 INFO:teuthology.orchestra.run.smithi063.stdout: "required_mon": [ 2024-04-26T04:20:39.831 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:20:39.831 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:20:39.831 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:20:39.831 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:20:39.831 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:20:39.831 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:20:39.831 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:20:39.831 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:20:39.831 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:20:39.831 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:20:39.831 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:20:39.831 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:20:39.832 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_con": "4540701547738038271", 2024-04-26T04:20:39.832 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_mon": [ 2024-04-26T04:20:39.832 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:20:39.832 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:20:39.832 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:20:39.832 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:20:39.832 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:20:39.832 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:20:39.832 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:20:39.832 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:20:39.832 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:20:39.832 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:20:39.832 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:20:39.832 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:20:39.832 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:39.832 INFO:teuthology.orchestra.run.smithi063.stdout: "outside_quorum": [], 2024-04-26T04:20:39.833 INFO:teuthology.orchestra.run.smithi063.stdout: "extra_probe_peers": [], 2024-04-26T04:20:39.833 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_provider": [], 2024-04-26T04:20:39.833 INFO:teuthology.orchestra.run.smithi063.stdout: "monmap": { 2024-04-26T04:20:39.833 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 1, 2024-04-26T04:20:39.833 INFO:teuthology.orchestra.run.smithi063.stdout: "fsid": "cd1c6a0c-4e4b-4014-89c0-2e535479ef39", 2024-04-26T04:20:39.833 INFO:teuthology.orchestra.run.smithi063.stdout: "modified": "2024-04-26T04:18:03.702339Z", 2024-04-26T04:20:39.833 INFO:teuthology.orchestra.run.smithi063.stdout: "created": "2024-04-26T04:18:03.702339Z", 2024-04-26T04:20:39.833 INFO:teuthology.orchestra.run.smithi063.stdout: "min_mon_release": 19, 2024-04-26T04:20:39.833 INFO:teuthology.orchestra.run.smithi063.stdout: "min_mon_release_name": "squid", 2024-04-26T04:20:39.833 INFO:teuthology.orchestra.run.smithi063.stdout: "election_strategy": 1, 2024-04-26T04:20:39.833 INFO:teuthology.orchestra.run.smithi063.stdout: "disallowed_leaders: ": "", 2024-04-26T04:20:39.833 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": false, 2024-04-26T04:20:39.833 INFO:teuthology.orchestra.run.smithi063.stdout: "tiebreaker_mon": "", 2024-04-26T04:20:39.833 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_ranks: ": "", 2024-04-26T04:20:39.834 INFO:teuthology.orchestra.run.smithi063.stdout: "features": { 2024-04-26T04:20:39.834 INFO:teuthology.orchestra.run.smithi063.stdout: "persistent": [ 2024-04-26T04:20:39.834 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:20:39.834 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:20:39.834 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:20:39.834 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:20:39.834 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:20:39.834 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:20:39.834 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:20:39.834 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:20:39.834 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:20:39.834 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:20:39.834 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:20:39.834 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:20:39.834 INFO:teuthology.orchestra.run.smithi063.stdout: "optional": [] 2024-04-26T04:20:39.835 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:39.835 INFO:teuthology.orchestra.run.smithi063.stdout: "mons": [ 2024-04-26T04:20:39.835 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:39.835 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 0, 2024-04-26T04:20:39.835 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a", 2024-04-26T04:20:39.835 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:20:39.835 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:20:39.835 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:39.835 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:20:39.835 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:3300", 2024-04-26T04:20:39.835 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:20:39.835 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:39.835 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:39.835 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:20:39.836 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6789", 2024-04-26T04:20:39.836 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:20:39.836 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:20:39.836 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:20:39.836 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:39.836 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6789/0", 2024-04-26T04:20:39.836 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6789/0", 2024-04-26T04:20:39.836 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:20:39.836 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:20:39.836 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:20:39.836 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:39.836 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:39.836 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 1, 2024-04-26T04:20:39.836 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b", 2024-04-26T04:20:39.836 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:20:39.837 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:20:39.837 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:39.837 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:20:39.837 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:3300", 2024-04-26T04:20:39.837 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:20:39.837 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:39.837 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:39.837 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:20:39.837 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:6789", 2024-04-26T04:20:39.837 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:20:39.837 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:20:39.837 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:20:39.837 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:39.837 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:6789/0", 2024-04-26T04:20:39.837 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.186:6789/0", 2024-04-26T04:20:39.838 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:20:39.838 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:20:39.838 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:20:39.838 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:39.838 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:39.838 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 2, 2024-04-26T04:20:39.838 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "c", 2024-04-26T04:20:39.838 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:20:39.838 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:20:39.838 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:39.838 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:20:39.838 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:3301", 2024-04-26T04:20:39.838 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:20:39.838 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:39.839 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:39.839 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:20:39.839 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6790", 2024-04-26T04:20:39.839 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:20:39.839 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:20:39.839 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:20:39.839 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:39.839 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6790/0", 2024-04-26T04:20:39.839 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6790/0", 2024-04-26T04:20:39.839 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:20:39.839 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:20:39.839 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:20:39.839 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:20:39.839 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:20:39.839 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:39.840 INFO:teuthology.orchestra.run.smithi063.stdout: "feature_map": { 2024-04-26T04:20:39.840 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": [ 2024-04-26T04:20:39.840 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:39.840 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:20:39.840 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:20:39.840 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 1 2024-04-26T04:20:39.840 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:20:39.840 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:20:39.840 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": [ 2024-04-26T04:20:39.840 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:39.840 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:20:39.840 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:20:39.840 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 4 2024-04-26T04:20:39.840 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:20:39.841 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:20:39.841 INFO:teuthology.orchestra.run.smithi063.stdout: "client": [ 2024-04-26T04:20:39.841 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:39.841 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:20:39.841 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:20:39.841 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 2 2024-04-26T04:20:39.841 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:20:39.841 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:20:39.841 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:39.841 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": false 2024-04-26T04:20:39.841 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-26T04:20:39.842 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.821+0000 7f90a2682640 1 -- 172.21.15.63:0/3792814066 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f907c035ec0 msgr2=0x7f907c038380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:39.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.821+0000 7f90a2682640 1 --2- 172.21.15.63:0/3792814066 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f907c035ec0 0x7f907c038380 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f909c06b260 tx=0x7f90940a7070 comp rx=0 tx=0).stop 2024-04-26T04:20:39.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.821+0000 7f90a2682640 10 monclient: shutdown 2024-04-26T04:20:39.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.821+0000 7f90a2682640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:20:39.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.821+0000 7f90a2682640 1 -- 172.21.15.63:0/3792814066 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f909c09b3b0 msgr2=0x7f909c1cc9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:39.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.821+0000 7f90a2682640 1 --2- 172.21.15.63:0/3792814066 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f909c09b3b0 0x7f909c1cc9a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f908c00c430 tx=0x7f908c00c8f0 comp rx=0 tx=0).stop 2024-04-26T04:20:39.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.821+0000 7f90a0c1f640 1 -- 172.21.15.63:0/3792814066 reap_dead start 2024-04-26T04:20:39.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.821+0000 7f90a0c1f640 1 -- 172.21.15.63:0/3792814066 reap_dead start 2024-04-26T04:20:39.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.821+0000 7f90a2682640 1 -- 172.21.15.63:0/3792814066 shutdown_connections 2024-04-26T04:20:39.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.821+0000 7f90a2682640 1 -- 172.21.15.63:0/3792814066 >> 172.21.15.63:0/3792814066 conn(0x7f909c0a1e70 msgr2=0x7f909c072d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:20:39.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.821+0000 7f90a2682640 1 -- 172.21.15.63:0/3792814066 shutdown_connections 2024-04-26T04:20:39.843 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.821+0000 7f90a2682640 1 -- 172.21.15.63:0/3792814066 wait complete. 2024-04-26T04:20:39.843 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T04:20:39.844 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-26T04:20:39.844 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T04:20:39.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.957+0000 7f427a70a640 10 monclient: get_monmap_and_config 2024-04-26T04:20:39.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.957+0000 7f427a70a640 10 monclient: build_initial_monmap 2024-04-26T04:20:39.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.957+0000 7f427a70a640 10 monclient: monmap: 2024-04-26T04:20:39.959 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:20:39.959 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:39.959 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:20:39.959064+0000 2024-04-26T04:20:39.959 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:20:39.959064+0000 2024-04-26T04:20:39.959 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:20:39.959 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:39.960 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:20:39.960 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:20:39.960 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:20:39.960 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:20:39.960 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:20:39.960 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:39.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.957+0000 7f427a70a640 1 Processor -- start 2024-04-26T04:20:39.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.957+0000 7f427a70a640 1 -- start start 2024-04-26T04:20:39.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.957+0000 7f427a70a640 10 monclient: init 2024-04-26T04:20:39.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.957+0000 7f427a70a640 10 monclient: _reopen_session rank -1 2024-04-26T04:20:39.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.957+0000 7f427a70a640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-04-26T04:20:39.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.957+0000 7f427a70a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f427409fe60 addr v1:172.21.15.63:6789/0 2024-04-26T04:20:39.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.957+0000 7f427a70a640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f42740a0240 0x7f42740a2a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:39.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.957+0000 7f427a70a640 10 monclient(hunting): picked mon.noname-c con 0x7f42740a0240 addr v2:172.21.15.186:3300/0 2024-04-26T04:20:39.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.957+0000 7f427a70a640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f42740a2f70 0x7f427413e110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:39.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.957+0000 7f427a70a640 10 monclient(hunting): picked mon.noname-a con 0x7f42740a2f70 addr v2:172.21.15.63:3300/0 2024-04-26T04:20:39.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.957+0000 7f427a70a640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42740a0c60 con 0x7f427409fe60 2024-04-26T04:20:39.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.957+0000 7f427a70a640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:39.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.957+0000 7f427a70a640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7f42740a3730 con 0x7f42740a2f70 2024-04-26T04:20:39.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.957+0000 7f427a70a640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:39.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.957+0000 7f427a70a640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f42740a0f40 con 0x7f42740a0240 2024-04-26T04:20:39.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.957+0000 7f427a70a640 10 monclient(hunting): _renew_subs 2024-04-26T04:20:39.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.957+0000 7f427a70a640 10 monclient(hunting): authenticate will time out at 1071.666738s 2024-04-26T04:20:39.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.957+0000 7f4278ca7640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f42740a2f70 0x7f427413e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:39.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.957+0000 7f4278ca7640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f42740a2f70 0x7f427413e110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:36494/0 (socket says 172.21.15.63:36494) 2024-04-26T04:20:39.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.957+0000 7f4278ca7640 1 -- 172.21.15.63:0/2909923790 learned_addr learned my addr 172.21.15.63:0/2909923790 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:20:39.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.957+0000 7f4278ca7640 10 monclient(hunting): get_auth_request con 0x7f42740a2f70 auth_method 0 2024-04-26T04:20:39.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.957+0000 7f4278ca7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:39.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.957+0000 7f4278ca7640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:39.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.957+0000 7f4278ca7640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:20:39.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.957+0000 7f4278ca7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:20:39.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.957+0000 7f4278ca7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:20:39.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.957+0000 7f42737fe640 1 --2- 172.21.15.63:0/2909923790 >> v2:172.21.15.186:3300/0 conn(0x7f42740a0240 0x7f42740a2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:39.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.957+0000 7f4278ca7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:20:39.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.957+0000 7f42737fe640 10 monclient(hunting): get_auth_request con 0x7f42740a0240 auth_method 0 2024-04-26T04:20:39.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.957+0000 7f42737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:39.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.957+0000 7f42737fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:39.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.957+0000 7f42737fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:20:39.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.957+0000 7f42737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:20:39.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.957+0000 7f42737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:20:39.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.957+0000 7f42737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:20:39.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.957+0000 7f42737fe640 10 monclient(hunting): handle_auth_done global_id 4705 payload 274 2024-04-26T04:20:39.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.957+0000 7f42737fe640 1 -- 172.21.15.63:0/2909923790 >> v2:172.21.15.63:3300/0 conn(0x7f42740a2f70 msgr2=0x7f427413e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:20:39.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.957+0000 7f42737fe640 1 --2- 172.21.15.63:0/2909923790 >> v2:172.21.15.63:3300/0 conn(0x7f42740a2f70 0x7f427413e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:39.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.957+0000 7f42737fe640 1 -- 172.21.15.63:0/2909923790 >> v1:172.21.15.63:6789/0 conn(0x7f427409fe60 legacy=0x7f42741372f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:20:39.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.957+0000 7f42737fe640 10 monclient: _finish_hunting 0 2024-04-26T04:20:39.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.957+0000 7f42737fe640 1 monclient: found mon.noname-c 2024-04-26T04:20:39.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.957+0000 7f42737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:20:39.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.957+0000 7f42737fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.186:3300/0 2024-04-26T04:20:39.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.957+0000 7f42737fe640 1 -- 172.21.15.63:0/2909923790 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42740a1220 con 0x7f42740a0240 2024-04-26T04:20:39.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f42737fe640 1 -- 172.21.15.63:0/2909923790 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42740a0240 msgr2=0x7f42740a2a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:20:39.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f42737fe640 1 --2- 172.21.15.63:0/2909923790 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42740a0240 0x7f42740a2a30 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f4274137a00 tx=0x7f425c01c300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de1321cb7bd65b9b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:39.967 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f4272ffd640 1 -- 172.21.15.63:0/2909923790 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f425c0b0070 con 0x7f42740a0240 2024-04-26T04:20:39.967 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f4272ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:20:39.967 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f4272ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:20:39.967 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f4272ffd640 10 monclient: dump: 2024-04-26T04:20:39.967 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:20:39.967 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:39.967 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:39.967 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:39.967 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:20:39.967 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:39.967 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:20:39.967 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:20:39.967 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:20:39.967 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:39.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f4272ffd640 10 monclient: _finish_auth 0 2024-04-26T04:20:39.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f4272ffd640 10 monclient: _check_auth_tickets 2024-04-26T04:20:39.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f4272ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:20:39.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f4272ffd640 1 -- 172.21.15.63:0/2909923790 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f425c0a9710 con 0x7f42740a0240 2024-04-26T04:20:39.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f427a70a640 5 monclient: authenticate success, global_id 4705 2024-04-26T04:20:39.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f427a70a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T04:20:39.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f4272ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T04:20:39.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f4272ffd640 1 -- 172.21.15.63:0/2909923790 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f425c0a9ae0 con 0x7f42740a0240 2024-04-26T04:20:39.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f427a70a640 10 monclient: get_monmap_and_config success 2024-04-26T04:20:39.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f427a70a640 4 set_mon_vals no callback set 2024-04-26T04:20:39.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f4272ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:20:39.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f4272ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T04:20:39.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f4272ffd640 10 monclient: dump: 2024-04-26T04:20:39.968 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:20:39.968 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:39.969 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:39.969 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:39.969 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:20:39.969 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:39.969 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:20:39.969 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:20:39.969 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:20:39.969 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:39.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f427a70a640 10 monclient: shutdown 2024-04-26T04:20:39.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f427a70a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:20:39.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f427a70a640 1 -- 172.21.15.63:0/2909923790 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42740a0240 msgr2=0x7f42740a2a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:39.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f427a70a640 1 --2- 172.21.15.63:0/2909923790 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42740a0240 0x7f42740a2a30 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f4274137a00 tx=0x7f425c01c300 comp rx=0 tx=0).stop 2024-04-26T04:20:39.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f427a70a640 1 -- 172.21.15.63:0/2909923790 shutdown_connections 2024-04-26T04:20:39.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f427a70a640 1 --2- 172.21.15.63:0/2909923790 >> v2:172.21.15.63:3300/0 conn(0x7f42740a2f70 0x7f427413e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:39.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f427a70a640 1 --2- 172.21.15.63:0/2909923790 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42740a0240 0x7f42740a2a30 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:39.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f427a70a640 1 -- 172.21.15.63:0/2909923790 >> 172.21.15.63:0/2909923790 conn(0x7f42740a5fb0 msgr2=0x7f42740a63c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:20:39.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f427a70a640 1 -- 172.21.15.63:0/2909923790 shutdown_connections 2024-04-26T04:20:39.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f427a70a640 1 -- 172.21.15.63:0/2909923790 wait complete. 2024-04-26T04:20:39.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f427a70a640 10 monclient: build_initial_monmap 2024-04-26T04:20:39.976 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f427a70a640 10 monclient: monmap: 2024-04-26T04:20:39.976 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:20:39.976 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:20:39.976 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:20:39.976 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:20:39.976 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:20:39.976 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:39.976 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:20:39.976 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:20:39.977 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:20:39.977 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:39.977 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f427a70a640 1 Processor -- start 2024-04-26T04:20:39.977 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f427a70a640 1 -- start start 2024-04-26T04:20:39.977 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f427a70a640 10 monclient: init 2024-04-26T04:20:39.977 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f427a70a640 10 monclient: _reopen_session rank -1 2024-04-26T04:20:39.977 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f427a70a640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T04:20:39.977 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f427a70a640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f427409fe60 0x7f4274132e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:39.977 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f427a70a640 10 monclient(hunting): picked mon.noname-c con 0x7f427409fe60 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:20:39.977 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f427a70a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f42740a0240 0x7f42741333d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:39.977 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f427a70a640 10 monclient(hunting): picked mon.noname-a con 0x7f42740a0240 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:20:39.977 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f427a70a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42740a2f70 0x7f4274137df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:39.977 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f427a70a640 10 monclient(hunting): picked mon.noname-b con 0x7f42740a2f70 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:20:39.977 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f427a70a640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:39.978 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f427a70a640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f4274141290 con 0x7f42740a0240 2024-04-26T04:20:39.978 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f427a70a640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:39.978 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f427a70a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4274141110 con 0x7f42740a2f70 2024-04-26T04:20:39.978 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f427a70a640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:39.978 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f427a70a640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f4274141410 con 0x7f427409fe60 2024-04-26T04:20:39.978 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f427a70a640 10 monclient(hunting): _renew_subs 2024-04-26T04:20:39.978 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f427a70a640 10 monclient(hunting): authenticate will time out at 1071.674787s 2024-04-26T04:20:39.978 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f42737fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f42740a0240 0x7f42741333d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:39.978 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f42737fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f42740a0240 0x7f42741333d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:36506/0 (socket says 172.21.15.63:36506) 2024-04-26T04:20:39.978 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f4273fff640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f427409fe60 0x7f4274132e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:39.978 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f42737fe640 1 -- 172.21.15.63:0/1324363801 learned_addr learned my addr 172.21.15.63:0/1324363801 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:20:39.978 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f42737fe640 10 monclient(hunting): get_auth_request con 0x7f42740a0240 auth_method 0 2024-04-26T04:20:39.978 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f42737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:39.978 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f42737fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:39.979 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f42737fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:20:39.979 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f4273fff640 10 monclient(hunting): get_auth_request con 0x7f427409fe60 auth_method 0 2024-04-26T04:20:39.979 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f4273fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:39.979 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f4273fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:39.979 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f4273fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:20:39.979 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f42737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:20:39.979 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f42737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:20:39.979 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f42737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:20:39.979 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f4273fff640 1 -- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f427409fe60 msgr2=0x7f4274132e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T04:20:39.979 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f4273fff640 1 -- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f427409fe60 msgr2=0x7f4274132e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T04:20:39.979 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f4273fff640 1 --2- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f427409fe60 0x7f4274132e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:20:39.979 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f4273fff640 1 --2- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f427409fe60 0x7f4274132e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:20:39.979 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f4278ca7640 1 --2- 172.21.15.63:0/1324363801 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42740a2f70 0x7f4274137df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:39.979 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f42737fe640 10 monclient(hunting): handle_auth_done global_id 4773 payload 824 2024-04-26T04:20:39.980 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f42737fe640 1 -- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f427409fe60 msgr2=0x7f4274132e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T04:20:39.980 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f42737fe640 1 --2- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f427409fe60 0x7f4274132e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:39.980 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f42737fe640 1 -- 172.21.15.63:0/1324363801 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42740a2f70 msgr2=0x7f4274137df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:39.980 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f42737fe640 1 --2- 172.21.15.63:0/1324363801 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42740a2f70 0x7f4274137df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:39.980 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f42737fe640 10 monclient: _finish_hunting 0 2024-04-26T04:20:39.980 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f42737fe640 1 monclient: found mon.noname-a 2024-04-26T04:20:39.980 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f42737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:20:39.980 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f42737fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:20:39.980 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f42737fe640 1 -- 172.21.15.63:0/1324363801 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42740a4470 con 0x7f42740a0240 2024-04-26T04:20:39.980 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f4278ca7640 10 monclient: get_auth_request con 0x7f42740a2f70 auth_method 0 2024-04-26T04:20:39.980 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.965+0000 7f4278ca7640 1 --2- 172.21.15.63:0/1324363801 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42740a2f70 0x7f4274137df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T04:20:39.980 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.969+0000 7f42737fe640 1 -- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f42740a0240 msgr2=0x7f42741333d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:20:39.980 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.969+0000 7f42737fe640 1 --2- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f42740a0240 0x7f42741333d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f425c018f10 tx=0x7f425c0190a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:39.980 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.969+0000 7f4270ff9640 1 -- 172.21.15.63:0/1324363801 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f425c0b9070 con 0x7f42740a0240 2024-04-26T04:20:39.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.969+0000 7f4270ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:20:39.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.969+0000 7f4270ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:20:39.981 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.969+0000 7f4270ff9640 10 monclient: dump: 2024-04-26T04:20:39.981 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:20:39.981 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:39.981 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:39.981 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:39.981 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:20:39.981 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:39.981 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:20:39.981 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:20:39.982 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:20:39.982 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:39.982 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.969+0000 7f4270ff9640 10 monclient: _finish_auth 0 2024-04-26T04:20:39.982 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.969+0000 7f4270ff9640 10 monclient: _check_auth_tickets 2024-04-26T04:20:39.982 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.969+0000 7f4270ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:20:39.982 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.969+0000 7f4270ff9640 1 -- 172.21.15.63:0/1324363801 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f425c0b58a0 con 0x7f42740a0240 2024-04-26T04:20:39.982 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.969+0000 7f4270ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T04:20:39.982 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.969+0000 7f4270ff9640 1 -- 172.21.15.63:0/1324363801 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f425c0b0040 con 0x7f42740a0240 2024-04-26T04:20:39.982 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.969+0000 7f4270ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:20:39.982 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.969+0000 7f4270ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:20:39.982 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.969+0000 7f4270ff9640 10 monclient: dump: 2024-04-26T04:20:39.982 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:20:39.982 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:39.982 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:39.983 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:39.983 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:20:39.983 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:39.983 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:20:39.983 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:20:39.983 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:20:39.983 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:39.983 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.969+0000 7f427a70a640 5 monclient: authenticate success, global_id 4773 2024-04-26T04:20:39.983 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.969+0000 7f427a70a640 10 monclient: _renew_subs 2024-04-26T04:20:39.983 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.969+0000 7f427a70a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:20:39.983 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.969+0000 7f4271ffb640 4 set_mon_vals no callback set 2024-04-26T04:20:39.983 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.969+0000 7f427a70a640 1 -- 172.21.15.63:0/1324363801 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42740a38f0 con 0x7f42740a0240 2024-04-26T04:20:39.983 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.969+0000 7f427a70a640 10 monclient: _renew_subs 2024-04-26T04:20:39.983 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.969+0000 7f427a70a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:20:39.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.969+0000 7f427a70a640 1 -- 172.21.15.63:0/1324363801 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f427413b4b0 con 0x7f42740a0240 2024-04-26T04:20:39.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.969+0000 7f427a70a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:20:39.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.969+0000 7f427a70a640 1 --2- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4274138330 0x7f4274138710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:39.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.969+0000 7f427a70a640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:39.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.969+0000 7f427a70a640 1 -- 172.21.15.63:0/1324363801 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f4274139ea0 con 0x7f4274138330 2024-04-26T04:20:39.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.969+0000 7f4270ff9640 1 -- 172.21.15.63:0/1324363801 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f425c0bc070 con 0x7f42740a0240 2024-04-26T04:20:39.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.969+0000 7f4273fff640 1 --2- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4274138330 0x7f4274138710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:39.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.969+0000 7f4273fff640 10 monclient: get_auth_request con 0x7f4274138330 auth_method 0 2024-04-26T04:20:39.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.969+0000 7f4273fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:39.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.969+0000 7f4273fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:39.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.969+0000 7f4273fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:20:39.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.969+0000 7f4273fff640 1 -- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4274138330 msgr2=0x7f4274138710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T04:20:39.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.969+0000 7f4273fff640 1 -- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4274138330 msgr2=0x7f4274138710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T04:20:39.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.969+0000 7f4273fff640 1 --2- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4274138330 0x7f4274138710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:20:39.984 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.969+0000 7f4273fff640 1 --2- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4274138330 0x7f4274138710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:20:39.985 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.969+0000 7f4270ff9640 1 --2- 172.21.15.63:0/1324363801 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f4254035ec0 0x7f4254038380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:39.985 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.969+0000 7f4270ff9640 1 -- 172.21.15.63:0/1324363801 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 4738+0+0 (secure 0 0 0) 0x7f425c0e3360 con 0x7f42740a0240 2024-04-26T04:20:39.985 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.969+0000 7f4278ca7640 1 --2- 172.21.15.63:0/1324363801 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f4254035ec0 0x7f4254038380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:39.985 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.969+0000 7f4278ca7640 10 monclient: get_auth_request con 0x7f4254035ec0 auth_method 0 2024-04-26T04:20:39.985 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:39.973+0000 7f4278ca7640 1 --2- 172.21.15.63:0/1324363801 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f4254035ec0 0x7f4254038380 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f4264009d70 tx=0x7f426400b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:40.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:40.173+0000 7f4273fff640 1 --2- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4274138330 0x7f4274138710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:40.211 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:40.173+0000 7f4273fff640 10 monclient: get_auth_request con 0x7f4274138330 auth_method 0 2024-04-26T04:20:40.211 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:40.173+0000 7f4273fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:40.211 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:40.173+0000 7f4273fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:40.211 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:40.173+0000 7f4273fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:20:40.211 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:40.173+0000 7f4273fff640 1 -- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4274138330 msgr2=0x7f4274138710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-26T04:20:40.211 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:40.173+0000 7f4273fff640 1 -- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4274138330 msgr2=0x7f4274138710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T04:20:40.211 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:40.173+0000 7f4273fff640 1 --2- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4274138330 0x7f4274138710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:20:40.211 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:40.173+0000 7f4273fff640 1 --2- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4274138330 0x7f4274138710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T04:20:40.501 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29022]: starting. 2024-04-26T04:20:40.502 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29022]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-29021 2024-04-26T04:20:40.502 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29022]: created object 0... 2024-04-26T04:20:40.502 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29022]: created object 25... 2024-04-26T04:20:40.502 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29022]: created object 49... 2024-04-26T04:20:40.502 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29022]: finishing. 2024-04-26T04:20:40.502 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29022]: shutting down. 2024-04-26T04:20:40.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:40.577+0000 7f4273fff640 1 --2- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4274138330 0x7f4274138710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:40.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:40.577+0000 7f4273fff640 10 monclient: get_auth_request con 0x7f4274138330 auth_method 0 2024-04-26T04:20:40.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:40.577+0000 7f4273fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:40.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:40.577+0000 7f4273fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:40.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:40.577+0000 7f4273fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:20:40.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:40.577+0000 7f4273fff640 1 -- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4274138330 msgr2=0x7f4274138710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-26T04:20:40.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:40.577+0000 7f4273fff640 1 -- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4274138330 msgr2=0x7f4274138710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T04:20:40.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:40.577+0000 7f4273fff640 1 --2- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4274138330 0x7f4274138710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:20:40.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:40.577+0000 7f4273fff640 1 --2- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4274138330 0x7f4274138710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T04:20:41.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:41.381+0000 7f4273fff640 1 --2- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4274138330 0x7f4274138710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:41.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:41.381+0000 7f4273fff640 10 monclient: get_auth_request con 0x7f4274138330 auth_method 0 2024-04-26T04:20:41.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:41.381+0000 7f4273fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:41.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:41.381+0000 7f4273fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:41.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:41.381+0000 7f4273fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:20:41.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:41.381+0000 7f4273fff640 1 -- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4274138330 msgr2=0x7f4274138710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-26T04:20:41.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:41.381+0000 7f4273fff640 1 -- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4274138330 msgr2=0x7f4274138710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T04:20:41.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:41.381+0000 7f4273fff640 1 --2- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4274138330 0x7f4274138710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:20:41.383 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:41.381+0000 7f4273fff640 1 --2- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4274138330 0x7f4274138710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T04:20:41.610 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[29023]: starting. 2024-04-26T04:20:41.610 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[29023]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-29021 2024-04-26T04:20:41.610 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[29023]: shutting down. 2024-04-26T04:20:42.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:42.965+0000 7f426b7fe640 10 monclient: tick 2024-04-26T04:20:42.969 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:42.965+0000 7f426b7fe640 10 monclient: _check_auth_tickets 2024-04-26T04:20:42.969 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:42.965+0000 7f426b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:20:42.969 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:42.965+0000 7f426b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:20:42.986 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:42.985+0000 7f4273fff640 1 --2- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4274138330 0x7f4274138710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:42.986 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:42.985+0000 7f4273fff640 10 monclient: get_auth_request con 0x7f4274138330 auth_method 0 2024-04-26T04:20:42.987 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:42.985+0000 7f4273fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:42.987 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:42.985+0000 7f4273fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:42.987 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:42.985+0000 7f4273fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:20:42.987 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:42.985+0000 7f4273fff640 1 -- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4274138330 msgr2=0x7f4274138710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-26T04:20:42.987 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:42.985+0000 7f4273fff640 1 -- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4274138330 msgr2=0x7f4274138710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T04:20:42.987 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:42.985+0000 7f4273fff640 1 --2- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4274138330 0x7f4274138710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:20:42.987 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:42.985+0000 7f4273fff640 1 --2- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4274138330 0x7f4274138710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-26T04:20:43.844 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:20:43.845 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29056]: starting. 2024-04-26T04:20:43.845 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29056]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-29021 2024-04-26T04:20:43.845 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29056]: created object 0... 2024-04-26T04:20:43.845 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29056]: created object 25... 2024-04-26T04:20:43.845 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29056]: created object 49... 2024-04-26T04:20:43.845 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29056]: finishing. 2024-04-26T04:20:43.845 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29056]: shutting down. 2024-04-26T04:20:43.903 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:20:43.904 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[29058]: starting. 2024-04-26T04:20:43.904 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[29058]: listing objects. 2024-04-26T04:20:43.904 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[29058]: listed object 0... 2024-04-26T04:20:43.904 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[29058]: listed object 25... 2024-04-26T04:20:43.904 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[29058]: saw 50 objects 2024-04-26T04:20:43.904 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[29058]: shutting down. 2024-04-26T04:20:43.969 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:43.965+0000 7f426b7fe640 10 monclient: tick 2024-04-26T04:20:43.970 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:43.965+0000 7f426b7fe640 10 monclient: _check_auth_tickets 2024-04-26T04:20:43.970 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:43.965+0000 7f426b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:20:43.970 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:43.965+0000 7f426b7fe640 5 monclient: _check_tell_commands timeout tell command 1 2024-04-26T04:20:43.970 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:43.965+0000 7f426b7fe640 1 -- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4274138330 msgr2=0x7f4274138710 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T04:20:43.970 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:43.965+0000 7f426b7fe640 1 --2- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4274138330 0x7f4274138710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:43.970 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:43.965+0000 7f426b7fe640 1 --2- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f424c003550 0x7f424c0059e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:43.970 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:43.965+0000 7f426b7fe640 1 -- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4274138330 msgr2=0x7f4274138710 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:20:43.970 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:43.965+0000 7f426b7fe640 1 --2- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4274138330 0x7f4274138710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:43.970 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:43.965+0000 7f426b7fe640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:43.970 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:43.965+0000 7f426b7fe640 1 -- 172.21.15.63:0/1324363801 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f424c006090 con 0x7f424c003550 2024-04-26T04:20:43.970 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:43.965+0000 7f4273fff640 1 --2- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f424c003550 0x7f424c0059e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:43.970 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:43.965+0000 7f4273fff640 10 monclient: get_auth_request con 0x7f424c003550 auth_method 0 2024-04-26T04:20:43.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:43.965+0000 7f4273fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:43.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:43.965+0000 7f4273fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:43.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:43.965+0000 7f4273fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:20:43.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:43.965+0000 7f4273fff640 1 -- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f424c003550 msgr2=0x7f424c0059e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-26T04:20:43.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:43.965+0000 7f4273fff640 1 -- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f424c003550 msgr2=0x7f424c0059e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T04:20:43.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:43.969+0000 7f4273fff640 1 --2- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f424c003550 0x7f424c0059e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:20:43.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:43.969+0000 7f4273fff640 1 --2- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f424c003550 0x7f424c0059e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:20:44.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.173+0000 7f4273fff640 1 --2- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f424c003550 0x7f424c0059e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:44.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.173+0000 7f4273fff640 10 monclient: get_auth_request con 0x7f424c003550 auth_method 0 2024-04-26T04:20:44.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.173+0000 7f4273fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:44.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.173+0000 7f4273fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:44.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.173+0000 7f4273fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:20:44.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.173+0000 7f4273fff640 1 -- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f424c003550 msgr2=0x7f424c0059e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-26T04:20:44.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.173+0000 7f4273fff640 1 -- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f424c003550 msgr2=0x7f424c0059e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T04:20:44.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.173+0000 7f4273fff640 1 --2- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f424c003550 0x7f424c0059e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:20:44.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.173+0000 7f4273fff640 1 --2- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f424c003550 0x7f424c0059e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T04:20:44.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.577+0000 7f4273fff640 1 --2- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f424c003550 0x7f424c0059e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:44.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.577+0000 7f4273fff640 10 monclient: get_auth_request con 0x7f424c003550 auth_method 0 2024-04-26T04:20:44.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.577+0000 7f4273fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:44.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.577+0000 7f4273fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:44.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.577+0000 7f4273fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:20:44.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.577+0000 7f4273fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:20:44.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.577+0000 7f4273fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:20:44.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.577+0000 7f4273fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:20:44.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.577+0000 7f4273fff640 10 monclient(hunting): handle_auth_done global_id 14099 payload 824 2024-04-26T04:20:44.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.577+0000 7f4273fff640 1 -- 172.21.15.63:0/1324363801 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f424c006240 con 0x7f424c003550 2024-04-26T04:20:44.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.577+0000 7f4273fff640 1 -- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f424c003550 msgr2=0x7f424c0059e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:20:44.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.577+0000 7f4273fff640 1 --2- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f424c003550 0x7f424c0059e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f426c0176d0 tx=0x7f426c017b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:44.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.581+0000 7f4270ff9640 1 -- 172.21.15.63:0/1324363801 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f426c016d60 con 0x7f424c003550 2024-04-26T04:20:44.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.581+0000 7f4270ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:20:44.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.581+0000 7f4270ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:20:44.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.581+0000 7f4270ff9640 10 monclient: dump: 2024-04-26T04:20:44.582 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:20:44.582 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:44.582 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:44.582 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:44.582 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:20:44.583 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:44.583 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:20:44.583 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:20:44.583 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:20:44.583 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:44.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.581+0000 7f4270ff9640 1 -- 172.21.15.63:0/1324363801 <== mon.2 v2:172.21.15.63:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f426c016d60 con 0x7f424c003550 2024-04-26T04:20:44.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.581+0000 7f4270ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:20:44.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.581+0000 7f4270ff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:20:44.584 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.581+0000 7f4270ff9640 1 -- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f424c003550 msgr2=0x7f424c0059e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:44.584 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.581+0000 7f4270ff9640 1 --2- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f424c003550 0x7f424c0059e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f426c0176d0 tx=0x7f426c017b90 comp rx=0 tx=0).stop 2024-04-26T04:20:44.584 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.581+0000 7f4270ff9640 1 -- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f424c003550 msgr2=0x7f424c0059e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:20:44.584 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.581+0000 7f4270ff9640 1 --2- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f424c003550 0x7f424c0059e0 secure :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f426c0176d0 tx=0x7f426c017b90 comp rx=0 tx=0).stop 2024-04-26T04:20:44.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.589+0000 7f427a70a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T04:20:44.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.589+0000 7f427a70a640 1 --2- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4274138c50 0x7f427406e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:44.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.589+0000 7f427a70a640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:44.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.589+0000 7f427a70a640 1 -- 172.21.15.63:0/1324363801 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f427413c170 con 0x7f4274138c50 2024-04-26T04:20:44.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.589+0000 7f4273fff640 1 --2- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4274138c50 0x7f427406e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:44.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.589+0000 7f4273fff640 10 monclient: get_auth_request con 0x7f4274138c50 auth_method 0 2024-04-26T04:20:44.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.589+0000 7f4273fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:44.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.589+0000 7f4273fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:44.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.589+0000 7f4273fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:20:44.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.589+0000 7f4273fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:20:44.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.589+0000 7f4273fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:20:44.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.589+0000 7f4273fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:20:44.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.589+0000 7f4273fff640 10 monclient(hunting): handle_auth_done global_id 14102 payload 824 2024-04-26T04:20:44.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.589+0000 7f4273fff640 1 -- 172.21.15.63:0/1324363801 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f427413afa0 con 0x7f4274138c50 2024-04-26T04:20:44.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.593+0000 7f4273fff640 1 -- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4274138c50 msgr2=0x7f427406e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:20:44.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.593+0000 7f4273fff640 1 --2- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4274138c50 0x7f427406e320 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f426c0176d0 tx=0x7f426c003770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:44.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.593+0000 7f4273fff640 1 -- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4274138c50 msgr2=0x7f427406e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-26T04:20:44.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.593+0000 7f4273fff640 1 -- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4274138c50 msgr2=0x7f427406e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T04:20:44.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.593+0000 7f4273fff640 1 --2- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4274138c50 0x7f427406e320 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f426c0176d0 tx=0x7f426c003770 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:20:44.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.593+0000 7f4273fff640 1 --2- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4274138c50 0x7f427406e320 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f426c0176d0 tx=0x7f426c003770 comp rx=0 tx=0).stop 2024-04-26T04:20:44.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.593+0000 7f4278ca7640 1 -- 172.21.15.63:0/1324363801 reap_dead start 2024-04-26T04:20:44.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.593+0000 7f4270ff9640 1 -- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4274138c50 msgr2=0x7f427406e320 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:20:44.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.593+0000 7f4270ff9640 1 --2- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4274138c50 0x7f427406e320 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:44.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.593+0000 7f4270ff9640 1 --2- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f425403c520 0x7f425403e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:44.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.593+0000 7f4270ff9640 1 -- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4274138c50 msgr2=0x7f427406e320 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:20:44.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.593+0000 7f4270ff9640 1 --2- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4274138c50 0x7f427406e320 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:44.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.593+0000 7f4270ff9640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:44.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.593+0000 7f4270ff9640 1 -- 172.21.15.63:0/1324363801 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f425403ef20 con 0x7f425403c520 2024-04-26T04:20:44.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.593+0000 7f4273fff640 1 --2- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f425403c520 0x7f425403e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:44.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.593+0000 7f4273fff640 10 monclient: get_auth_request con 0x7f425403c520 auth_method 0 2024-04-26T04:20:44.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.593+0000 7f4273fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:44.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.593+0000 7f4273fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:44.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.593+0000 7f4273fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:20:44.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.593+0000 7f4273fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:20:44.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.593+0000 7f4273fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:20:44.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.593+0000 7f4273fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:20:44.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.593+0000 7f4273fff640 10 monclient(hunting): handle_auth_done global_id 14105 payload 824 2024-04-26T04:20:44.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.593+0000 7f4273fff640 1 -- 172.21.15.63:0/1324363801 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f426c016d60 con 0x7f425403c520 2024-04-26T04:20:44.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.597+0000 7f4273fff640 1 -- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f425403c520 msgr2=0x7f425403e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:20:44.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.597+0000 7f4273fff640 1 --2- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f425403c520 0x7f425403e910 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f426c017bd0 tx=0x7f426c003700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:44.599 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.597+0000 7f4270ff9640 1 -- 172.21.15.63:0/1324363801 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f426c03d070 con 0x7f425403c520 2024-04-26T04:20:44.600 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.597+0000 7f4270ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:20:44.600 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.597+0000 7f4270ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:20:44.600 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.597+0000 7f4270ff9640 10 monclient: dump: 2024-04-26T04:20:44.600 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:20:44.600 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:44.600 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:44.600 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:44.600 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:20:44.600 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:44.600 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:20:44.600 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:20:44.600 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:20:44.600 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:44.601 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.597+0000 7f4270ff9640 1 -- 172.21.15.63:0/1324363801 <== mon.2 v2:172.21.15.63:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4549 (secure 0 0 0) 0x7f426c02f920 con 0x7f425403c520 2024-04-26T04:20:44.601 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.597+0000 7f4270ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T04:20:44.601 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.597+0000 7f4270ff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:20:44.601 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.597+0000 7f4270ff9640 1 -- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f425403c520 msgr2=0x7f425403e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:44.601 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.597+0000 7f4270ff9640 1 --2- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f425403c520 0x7f425403e910 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f426c017bd0 tx=0x7f426c003700 comp rx=0 tx=0).stop 2024-04-26T04:20:44.601 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.597+0000 7f4270ff9640 1 -- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f425403c520 msgr2=0x7f425403e910 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:20:44.601 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.597+0000 7f4270ff9640 1 --2- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f425403c520 0x7f425403e910 secure :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f426c017bd0 tx=0x7f426c003700 comp rx=0 tx=0).stop 2024-04-26T04:20:44.602 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-26T04:20:44.602 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "c", 2024-04-26T04:20:44.602 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 2, 2024-04-26T04:20:44.602 INFO:teuthology.orchestra.run.smithi063.stdout: "state": "peon", 2024-04-26T04:20:44.602 INFO:teuthology.orchestra.run.smithi063.stdout: "election_epoch": 14, 2024-04-26T04:20:44.602 INFO:teuthology.orchestra.run.smithi063.stdout: "uptime": 5227, 2024-04-26T04:20:44.602 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum": [ 2024-04-26T04:20:44.602 INFO:teuthology.orchestra.run.smithi063.stdout: 0, 2024-04-26T04:20:44.602 INFO:teuthology.orchestra.run.smithi063.stdout: 1, 2024-04-26T04:20:44.602 INFO:teuthology.orchestra.run.smithi063.stdout: 2 2024-04-26T04:20:44.603 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:20:44.603 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_age": 4, 2024-04-26T04:20:44.603 INFO:teuthology.orchestra.run.smithi063.stdout: "features": { 2024-04-26T04:20:44.603 INFO:teuthology.orchestra.run.smithi063.stdout: "required_con": "2450521708007866372", 2024-04-26T04:20:44.603 INFO:teuthology.orchestra.run.smithi063.stdout: "required_mon": [ 2024-04-26T04:20:44.603 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:20:44.603 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:20:44.603 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:20:44.603 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:20:44.603 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:20:44.603 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:20:44.603 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:20:44.603 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:20:44.603 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:20:44.604 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:20:44.604 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:20:44.604 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:20:44.604 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_con": "4540701547738038271", 2024-04-26T04:20:44.604 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_mon": [ 2024-04-26T04:20:44.604 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:20:44.604 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:20:44.604 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:20:44.604 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:20:44.604 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:20:44.604 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:20:44.604 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:20:44.604 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:20:44.604 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:20:44.605 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:20:44.605 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:20:44.605 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:20:44.605 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:44.605 INFO:teuthology.orchestra.run.smithi063.stdout: "outside_quorum": [], 2024-04-26T04:20:44.605 INFO:teuthology.orchestra.run.smithi063.stdout: "extra_probe_peers": [], 2024-04-26T04:20:44.605 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_provider": [], 2024-04-26T04:20:44.605 INFO:teuthology.orchestra.run.smithi063.stdout: "monmap": { 2024-04-26T04:20:44.605 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 1, 2024-04-26T04:20:44.605 INFO:teuthology.orchestra.run.smithi063.stdout: "fsid": "cd1c6a0c-4e4b-4014-89c0-2e535479ef39", 2024-04-26T04:20:44.605 INFO:teuthology.orchestra.run.smithi063.stdout: "modified": "2024-04-26T04:18:03.702339Z", 2024-04-26T04:20:44.605 INFO:teuthology.orchestra.run.smithi063.stdout: "created": "2024-04-26T04:18:03.702339Z", 2024-04-26T04:20:44.605 INFO:teuthology.orchestra.run.smithi063.stdout: "min_mon_release": 19, 2024-04-26T04:20:44.605 INFO:teuthology.orchestra.run.smithi063.stdout: "min_mon_release_name": "squid", 2024-04-26T04:20:44.606 INFO:teuthology.orchestra.run.smithi063.stdout: "election_strategy": 1, 2024-04-26T04:20:44.606 INFO:teuthology.orchestra.run.smithi063.stdout: "disallowed_leaders: ": "", 2024-04-26T04:20:44.606 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": false, 2024-04-26T04:20:44.606 INFO:teuthology.orchestra.run.smithi063.stdout: "tiebreaker_mon": "", 2024-04-26T04:20:44.606 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_ranks: ": "", 2024-04-26T04:20:44.606 INFO:teuthology.orchestra.run.smithi063.stdout: "features": { 2024-04-26T04:20:44.606 INFO:teuthology.orchestra.run.smithi063.stdout: "persistent": [ 2024-04-26T04:20:44.606 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:20:44.606 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:20:44.606 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:20:44.606 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:20:44.606 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:20:44.606 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:20:44.606 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:20:44.606 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:20:44.607 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:20:44.607 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:20:44.607 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:20:44.607 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:20:44.607 INFO:teuthology.orchestra.run.smithi063.stdout: "optional": [] 2024-04-26T04:20:44.607 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:44.607 INFO:teuthology.orchestra.run.smithi063.stdout: "mons": [ 2024-04-26T04:20:44.607 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:44.607 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 0, 2024-04-26T04:20:44.607 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a", 2024-04-26T04:20:44.607 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:20:44.607 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:20:44.607 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:44.607 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:20:44.607 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:3300", 2024-04-26T04:20:44.608 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:20:44.608 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:44.608 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:44.608 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:20:44.608 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6789", 2024-04-26T04:20:44.608 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:20:44.608 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:20:44.608 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:20:44.608 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:44.608 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6789/0", 2024-04-26T04:20:44.608 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6789/0", 2024-04-26T04:20:44.608 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:20:44.608 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:20:44.608 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:20:44.608 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:44.609 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:44.609 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 1, 2024-04-26T04:20:44.609 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b", 2024-04-26T04:20:44.609 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:20:44.609 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:20:44.609 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:44.609 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:20:44.609 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:3300", 2024-04-26T04:20:44.609 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:20:44.609 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:44.609 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:44.609 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:20:44.609 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:6789", 2024-04-26T04:20:44.609 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:20:44.609 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:20:44.610 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:20:44.610 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:44.610 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:6789/0", 2024-04-26T04:20:44.610 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.186:6789/0", 2024-04-26T04:20:44.610 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:20:44.610 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:20:44.610 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:20:44.610 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:44.610 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:44.610 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 2, 2024-04-26T04:20:44.610 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "c", 2024-04-26T04:20:44.610 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:20:44.610 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:20:44.610 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:44.610 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:20:44.611 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:3301", 2024-04-26T04:20:44.611 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:20:44.611 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:44.611 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:44.611 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:20:44.611 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6790", 2024-04-26T04:20:44.611 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:20:44.611 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:20:44.611 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:20:44.611 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:44.611 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6790/0", 2024-04-26T04:20:44.611 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6790/0", 2024-04-26T04:20:44.611 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:20:44.611 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:20:44.612 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:20:44.612 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:20:44.612 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:20:44.612 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:44.612 INFO:teuthology.orchestra.run.smithi063.stdout: "feature_map": { 2024-04-26T04:20:44.612 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": [ 2024-04-26T04:20:44.612 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:44.612 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:20:44.612 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:20:44.612 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 1 2024-04-26T04:20:44.612 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:20:44.612 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:20:44.612 INFO:teuthology.orchestra.run.smithi063.stdout: "client": [ 2024-04-26T04:20:44.612 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:44.612 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:20:44.613 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:20:44.613 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 1 2024-04-26T04:20:44.613 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:20:44.613 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:20:44.613 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:44.613 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": false 2024-04-26T04:20:44.613 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-26T04:20:44.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.597+0000 7f427a70a640 1 -- 172.21.15.63:0/1324363801 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f4254035ec0 msgr2=0x7f4254038380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:44.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.597+0000 7f427a70a640 1 --2- 172.21.15.63:0/1324363801 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f4254035ec0 0x7f4254038380 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f4264009d70 tx=0x7f426400b040 comp rx=0 tx=0).stop 2024-04-26T04:20:44.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.597+0000 7f427a70a640 10 monclient: shutdown 2024-04-26T04:20:44.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.597+0000 7f427a70a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:20:44.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.597+0000 7f427a70a640 1 -- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f42740a0240 msgr2=0x7f42741333d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:44.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.597+0000 7f427a70a640 1 --2- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f42740a0240 0x7f42741333d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f425c018f10 tx=0x7f425c0190a0 comp rx=0 tx=0).stop 2024-04-26T04:20:44.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.597+0000 7f427a70a640 1 -- 172.21.15.63:0/1324363801 shutdown_connections 2024-04-26T04:20:44.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.597+0000 7f427a70a640 1 --2- 172.21.15.63:0/1324363801 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f4254035ec0 0x7f4254038380 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:44.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.597+0000 7f427a70a640 1 --2- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f42740a0240 0x7f42741333d0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:44.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.597+0000 7f427a70a640 1 --2- 172.21.15.63:0/1324363801 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f425403c520 0x7f425403e910 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:44.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.597+0000 7f427a70a640 1 -- 172.21.15.63:0/1324363801 >> 172.21.15.63:0/1324363801 conn(0x7f42740a5fb0 msgr2=0x7f42740a19b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:20:44.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.597+0000 7f427a70a640 1 -- 172.21.15.63:0/1324363801 shutdown_connections 2024-04-26T04:20:44.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.597+0000 7f427a70a640 1 -- 172.21.15.63:0/1324363801 wait complete. 2024-04-26T04:20:44.634 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:20:44.634 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[29057]: starting. 2024-04-26T04:20:44.635 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[29057]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-29021 2024-04-26T04:20:44.635 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[29057]: shutting down. 2024-04-26T04:20:44.638 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T04:20:44.638 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-26T04:20:44.639 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T04:20:44.639 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:20:44.639 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:20:44.639 INFO:teuthology.orchestra.run.smithi186.stdout:******* SUCCESS ********** 2024-04-26T04:20:44.640 DEBUG:teuthology.orchestra.run.smithi186:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-26T04:20:44.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.765+0000 7f18258e4640 10 monclient: get_monmap_and_config 2024-04-26T04:20:44.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.765+0000 7f18258e4640 10 monclient: build_initial_monmap 2024-04-26T04:20:44.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.765+0000 7f18258e4640 10 monclient: monmap: 2024-04-26T04:20:44.770 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:20:44.770 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:44.770 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:20:44.769538+0000 2024-04-26T04:20:44.770 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:20:44.769538+0000 2024-04-26T04:20:44.770 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:20:44.770 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:44.770 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:20:44.770 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:20:44.770 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:20:44.770 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:20:44.770 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:20:44.770 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:44.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.769+0000 7f18258e4640 1 Processor -- start 2024-04-26T04:20:44.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.769+0000 7f18258e4640 1 -- start start 2024-04-26T04:20:44.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.769+0000 7f18258e4640 10 monclient: init 2024-04-26T04:20:44.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.769+0000 7f18258e4640 10 monclient: _reopen_session rank -1 2024-04-26T04:20:44.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.769+0000 7f18258e4640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-04-26T04:20:44.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.769+0000 7f18258e4640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f18200a2cd0 0x7f18200a30b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:44.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.769+0000 7f18258e4640 10 monclient(hunting): picked mon.noname-c con 0x7f18200a2cd0 addr v2:172.21.15.186:3300/0 2024-04-26T04:20:44.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.769+0000 7f18258e4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f18200a47d0 addr v1:172.21.15.63:6789/0 2024-04-26T04:20:44.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.769+0000 7f18258e4640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f18200a4c00 0x7f18201373f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:44.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.769+0000 7f18258e4640 10 monclient(hunting): picked mon.noname-a con 0x7f18200a4c00 addr v2:172.21.15.63:3300/0 2024-04-26T04:20:44.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.769+0000 7f18258e4640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18200a0fb0 con 0x7f18200a47d0 2024-04-26T04:20:44.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.769+0000 7f18258e4640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:44.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.769+0000 7f18258e4640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7f18200a35f0 con 0x7f18200a4c00 2024-04-26T04:20:44.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.769+0000 7f18258e4640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:44.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.769+0000 7f18258e4640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f18200a5380 con 0x7f18200a2cd0 2024-04-26T04:20:44.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.769+0000 7f18258e4640 10 monclient(hunting): _renew_subs 2024-04-26T04:20:44.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.769+0000 7f18258e4640 10 monclient(hunting): authenticate will time out at 1076.477377s 2024-04-26T04:20:44.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.769+0000 7f181e7fc640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f18200a47d0 0x7f1820136ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:48794/0 (socket says 172.21.15.63:48794) 2024-04-26T04:20:44.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.769+0000 7f181e7fc640 1 -- 172.21.15.63:0/1947633167 learned_addr learned my addr 172.21.15.63:0/1947633167 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-04-26T04:20:44.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.769+0000 7f181f7fe640 1 --2- 172.21.15.63:0/1947633167 >> v2:172.21.15.63:3300/0 conn(0x7f18200a4c00 0x7f18201373f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:44.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.769+0000 7f181f7fe640 10 monclient(hunting): get_auth_request con 0x7f18200a4c00 auth_method 0 2024-04-26T04:20:44.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.769+0000 7f181f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:44.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.769+0000 7f181f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:44.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.769+0000 7f181f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:20:44.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.769+0000 7f181effd640 1 --2- 172.21.15.63:0/1947633167 >> v2:172.21.15.186:3300/0 conn(0x7f18200a2cd0 0x7f18200a30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:44.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.769+0000 7f181effd640 10 monclient(hunting): get_auth_request con 0x7f18200a2cd0 auth_method 0 2024-04-26T04:20:44.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.769+0000 7f181effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:44.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.769+0000 7f181effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:44.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.769+0000 7f181effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:20:44.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.769+0000 7f181effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:20:44.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.769+0000 7f181effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:20:44.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.769+0000 7f181effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:20:44.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.769+0000 7f181effd640 10 monclient(hunting): handle_auth_done global_id 4726 payload 274 2024-04-26T04:20:44.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.769+0000 7f181effd640 1 -- 172.21.15.63:0/1947633167 >> v2:172.21.15.63:3300/0 conn(0x7f18200a4c00 msgr2=0x7f18201373f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:20:44.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.769+0000 7f181effd640 1 --2- 172.21.15.63:0/1947633167 >> v2:172.21.15.63:3300/0 conn(0x7f18200a4c00 0x7f18201373f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:44.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.769+0000 7f181effd640 1 -- 172.21.15.63:0/1947633167 >> v1:172.21.15.63:6789/0 conn(0x7f18200a47d0 legacy=0x7f1820136ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:20:44.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.769+0000 7f181effd640 10 monclient: _finish_hunting 0 2024-04-26T04:20:44.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.769+0000 7f181effd640 1 monclient: found mon.noname-c 2024-04-26T04:20:44.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.769+0000 7f181effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:20:44.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.769+0000 7f181effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.186:3300/0 2024-04-26T04:20:44.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.769+0000 7f181effd640 1 -- 172.21.15.63:0/1947633167 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18200a1290 con 0x7f18200a2cd0 2024-04-26T04:20:44.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.773+0000 7f181effd640 1 -- 172.21.15.63:0/1947633167 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f18200a2cd0 msgr2=0x7f18200a30b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:20:44.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.773+0000 7f181effd640 1 --2- 172.21.15.63:0/1947633167 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f18200a2cd0 0x7f18200a30b0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f182006c760 tx=0x7f181001c420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f797e3a5cb699017 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:44.777 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.773+0000 7f181dffb640 1 -- 172.21.15.63:0/1947633167 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18100b0070 con 0x7f18200a2cd0 2024-04-26T04:20:44.777 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.773+0000 7f181dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:20:44.777 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.773+0000 7f181dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:20:44.777 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.773+0000 7f181dffb640 10 monclient: dump: 2024-04-26T04:20:44.777 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:20:44.777 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:44.777 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:44.777 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:44.777 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:20:44.777 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:44.778 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:20:44.778 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:20:44.778 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:20:44.778 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:44.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.773+0000 7f181dffb640 10 monclient: _finish_auth 0 2024-04-26T04:20:44.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.773+0000 7f181dffb640 10 monclient: _check_auth_tickets 2024-04-26T04:20:44.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.773+0000 7f181dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:20:44.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.773+0000 7f181dffb640 1 -- 172.21.15.63:0/1947633167 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18100ab600 con 0x7f18200a2cd0 2024-04-26T04:20:44.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.773+0000 7f181dffb640 10 monclient: handle_config config(0 keys) 2024-04-26T04:20:44.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.773+0000 7f181dffb640 1 -- 172.21.15.63:0/1947633167 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18100ab9d0 con 0x7f18200a2cd0 2024-04-26T04:20:44.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.773+0000 7f181dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:20:44.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.773+0000 7f181dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T04:20:44.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.773+0000 7f181dffb640 10 monclient: dump: 2024-04-26T04:20:44.778 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:20:44.779 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:44.779 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:44.779 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:44.779 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:20:44.779 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:44.779 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:20:44.779 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:20:44.779 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:20:44.779 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:44.779 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.773+0000 7f18258e4640 5 monclient: authenticate success, global_id 4726 2024-04-26T04:20:44.779 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.773+0000 7f18258e4640 10 monclient: get_monmap_and_config success 2024-04-26T04:20:44.779 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.773+0000 7f18258e4640 4 set_mon_vals no callback set 2024-04-26T04:20:44.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.777+0000 7f18258e4640 10 monclient: shutdown 2024-04-26T04:20:44.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.777+0000 7f18258e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:20:44.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.777+0000 7f18258e4640 1 -- 172.21.15.63:0/1947633167 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f18200a2cd0 msgr2=0x7f18200a30b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:44.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.777+0000 7f18258e4640 1 --2- 172.21.15.63:0/1947633167 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f18200a2cd0 0x7f18200a30b0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f182006c760 tx=0x7f181001c420 comp rx=0 tx=0).stop 2024-04-26T04:20:44.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.777+0000 7f18258e4640 1 -- 172.21.15.63:0/1947633167 shutdown_connections 2024-04-26T04:20:44.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.777+0000 7f18258e4640 1 --2- 172.21.15.63:0/1947633167 >> v2:172.21.15.63:3300/0 conn(0x7f18200a4c00 0x7f18201373f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:44.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.777+0000 7f18258e4640 1 --2- 172.21.15.63:0/1947633167 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f18200a2cd0 0x7f18200a30b0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:44.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.777+0000 7f18258e4640 1 -- 172.21.15.63:0/1947633167 >> 172.21.15.63:0/1947633167 conn(0x7f18200a5f40 msgr2=0x7f18200a6320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:20:44.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.777+0000 7f18258e4640 1 -- 172.21.15.63:0/1947633167 shutdown_connections 2024-04-26T04:20:44.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.777+0000 7f18258e4640 1 -- 172.21.15.63:0/1947633167 wait complete. 2024-04-26T04:20:44.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.777+0000 7f18258e4640 10 monclient: build_initial_monmap 2024-04-26T04:20:44.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.777+0000 7f18258e4640 10 monclient: monmap: 2024-04-26T04:20:44.783 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:20:44.783 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:20:44.783 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:20:44.783 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:20:44.783 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:20:44.783 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:44.783 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:20:44.784 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:20:44.784 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:20:44.784 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:44.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.777+0000 7f18258e4640 1 Processor -- start 2024-04-26T04:20:44.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.777+0000 7f18258e4640 1 -- start start 2024-04-26T04:20:44.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.777+0000 7f18258e4640 10 monclient: init 2024-04-26T04:20:44.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.777+0000 7f18258e4640 10 monclient: _reopen_session rank -1 2024-04-26T04:20:44.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.777+0000 7f18258e4640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T04:20:44.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.777+0000 7f18258e4640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f18200a2cd0 0x7f182013e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:44.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.777+0000 7f18258e4640 10 monclient(hunting): picked mon.noname-c con 0x7f18200a2cd0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:20:44.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.777+0000 7f18258e4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f18200a47d0 0x7f1820135b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:44.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.777+0000 7f18258e4640 10 monclient(hunting): picked mon.noname-b con 0x7f18200a47d0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:20:44.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.777+0000 7f18258e4640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f18200a4c00 0x7f18201ced70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:44.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.777+0000 7f18258e4640 10 monclient(hunting): picked mon.noname-a con 0x7f18200a4c00 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:20:44.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.777+0000 7f18258e4640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:44.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.777+0000 7f18258e4640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f1820141a50 con 0x7f18200a4c00 2024-04-26T04:20:44.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.777+0000 7f18258e4640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:44.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.777+0000 7f18258e4640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1820137b40 con 0x7f18200a47d0 2024-04-26T04:20:44.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.777+0000 7f18258e4640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:44.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.777+0000 7f18258e4640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f1820141bd0 con 0x7f18200a2cd0 2024-04-26T04:20:44.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.777+0000 7f18258e4640 10 monclient(hunting): _renew_subs 2024-04-26T04:20:44.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.777+0000 7f18258e4640 10 monclient(hunting): authenticate will time out at 1076.486330s 2024-04-26T04:20:44.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.777+0000 7f181effd640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f18200a2cd0 0x7f182013e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:44.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.777+0000 7f181effd640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f18200a2cd0 0x7f182013e2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:37204/0 (socket says 172.21.15.63:37204) 2024-04-26T04:20:44.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.777+0000 7f181effd640 1 -- 172.21.15.63:0/863185329 learned_addr learned my addr 172.21.15.63:0/863185329 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:20:44.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.777+0000 7f181effd640 10 monclient(hunting): get_auth_request con 0x7f18200a2cd0 auth_method 0 2024-04-26T04:20:44.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.777+0000 7f181effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:44.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.777+0000 7f181effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:44.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.777+0000 7f181effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:20:44.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.777+0000 7f181f7fe640 1 --2- 172.21.15.63:0/863185329 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f18200a4c00 0x7f18201ced70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:44.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.777+0000 7f181f7fe640 10 monclient(hunting): get_auth_request con 0x7f18200a4c00 auth_method 0 2024-04-26T04:20:44.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.777+0000 7f181f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:44.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.777+0000 7f181f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:44.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.777+0000 7f181f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:20:44.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.777+0000 7f181effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:20:44.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.777+0000 7f181effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:20:44.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.777+0000 7f181effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:20:44.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.777+0000 7f181f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:20:44.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.777+0000 7f181f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:20:44.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.777+0000 7f181f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:20:44.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.777+0000 7f181e7fc640 1 --2- 172.21.15.63:0/863185329 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f18200a47d0 0x7f1820135b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:44.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.777+0000 7f181e7fc640 10 monclient(hunting): get_auth_request con 0x7f18200a47d0 auth_method 0 2024-04-26T04:20:44.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.777+0000 7f181e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:44.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.777+0000 7f181e7fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:44.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.777+0000 7f181e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:20:44.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.777+0000 7f181effd640 10 monclient(hunting): handle_auth_done global_id 14120 payload 824 2024-04-26T04:20:44.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.777+0000 7f181effd640 1 -- 172.21.15.63:0/863185329 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f18200a47d0 msgr2=0x7f1820135b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:44.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.777+0000 7f181effd640 1 --2- 172.21.15.63:0/863185329 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f18200a47d0 0x7f1820135b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:44.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.777+0000 7f181effd640 1 -- 172.21.15.63:0/863185329 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f18200a4c00 msgr2=0x7f18201ced70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:44.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.777+0000 7f181effd640 1 --2- 172.21.15.63:0/863185329 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f18200a4c00 0x7f18201ced70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:44.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.777+0000 7f181effd640 10 monclient: _finish_hunting 0 2024-04-26T04:20:44.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.777+0000 7f181effd640 1 monclient: found mon.noname-c 2024-04-26T04:20:44.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.777+0000 7f181effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:20:44.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.777+0000 7f181effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.63:3301/0 2024-04-26T04:20:44.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.777+0000 7f181effd640 1 -- 172.21.15.63:0/863185329 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f182011fc50 con 0x7f18200a2cd0 2024-04-26T04:20:44.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.781+0000 7f181effd640 1 -- 172.21.15.63:0/863185329 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f18200a2cd0 msgr2=0x7f182013e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:20:44.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.781+0000 7f181effd640 1 --2- 172.21.15.63:0/863185329 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f18200a2cd0 0x7f182013e2f0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f18100ae040 tx=0x7f18100191c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:44.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.781+0000 7f1807fff640 1 -- 172.21.15.63:0/863185329 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18100b0040 con 0x7f18200a2cd0 2024-04-26T04:20:44.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.781+0000 7f1807fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:20:44.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.781+0000 7f1807fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:20:44.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.781+0000 7f1807fff640 10 monclient: dump: 2024-04-26T04:20:44.791 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:20:44.791 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:44.791 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:44.791 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:44.791 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:20:44.791 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:44.791 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:20:44.791 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:20:44.791 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:20:44.791 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:44.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.781+0000 7f1807fff640 10 monclient: _finish_auth 0 2024-04-26T04:20:44.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.781+0000 7f1807fff640 10 monclient: _check_auth_tickets 2024-04-26T04:20:44.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.781+0000 7f1807fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:20:44.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.781+0000 7f1807fff640 1 -- 172.21.15.63:0/863185329 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18100b9070 con 0x7f18200a2cd0 2024-04-26T04:20:44.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.781+0000 7f1807fff640 10 monclient: handle_config config(0 keys) 2024-04-26T04:20:44.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.781+0000 7f1807fff640 1 -- 172.21.15.63:0/863185329 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18100aaa10 con 0x7f18200a2cd0 2024-04-26T04:20:44.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.781+0000 7f18258e4640 5 monclient: authenticate success, global_id 14120 2024-04-26T04:20:44.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.781+0000 7f18258e4640 10 monclient: _renew_subs 2024-04-26T04:20:44.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.781+0000 7f18258e4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:20:44.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.781+0000 7f18258e4640 1 -- 172.21.15.63:0/863185329 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1820072d40 con 0x7f18200a2cd0 2024-04-26T04:20:44.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.781+0000 7f1807fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:20:44.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.781+0000 7f181cff9640 4 set_mon_vals no callback set 2024-04-26T04:20:44.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.781+0000 7f1807fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:20:44.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.781+0000 7f1807fff640 10 monclient: dump: 2024-04-26T04:20:44.792 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:20:44.792 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:44.792 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:44.792 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:44.793 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:20:44.793 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:44.793 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:20:44.793 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:20:44.793 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:20:44.793 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:44.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.781+0000 7f18258e4640 10 monclient: _renew_subs 2024-04-26T04:20:44.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.781+0000 7f18258e4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:20:44.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.781+0000 7f18258e4640 1 -- 172.21.15.63:0/863185329 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f182013d790 con 0x7f18200a2cd0 2024-04-26T04:20:44.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.781+0000 7f18258e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:20:44.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.781+0000 7f18258e4640 1 --2- 172.21.15.63:0/863185329 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f182013c0a0 0x7f18201cf2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:44.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.781+0000 7f18258e4640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:44.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.781+0000 7f18258e4640 1 -- 172.21.15.63:0/863185329 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1820062e10 con 0x7f182013c0a0 2024-04-26T04:20:44.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.781+0000 7f1807fff640 1 -- 172.21.15.63:0/863185329 <== mon.2 v2:172.21.15.63:3301/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f18100ab4c0 con 0x7f18200a2cd0 2024-04-26T04:20:44.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.781+0000 7f181e7fc640 1 --2- 172.21.15.63:0/863185329 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f182013c0a0 0x7f18201cf2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:44.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.781+0000 7f1807fff640 1 --2- 172.21.15.63:0/863185329 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f180c035f10 0x7f180c0383d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:44.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.781+0000 7f181e7fc640 10 monclient: get_auth_request con 0x7f182013c0a0 auth_method 0 2024-04-26T04:20:44.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.781+0000 7f181e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:44.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.781+0000 7f181e7fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:44.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.781+0000 7f181e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:20:44.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.781+0000 7f1807fff640 1 -- 172.21.15.63:0/863185329 <== mon.2 v2:172.21.15.63:3301/0 5 ==== osd_map(112..112 src has 1..112) ==== 4346+0+0 (secure 0 0 0) 0x7f18100e29a0 con 0x7f18200a2cd0 2024-04-26T04:20:44.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.781+0000 7f181e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:20:44.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.781+0000 7f181e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:20:44.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.781+0000 7f181e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:20:44.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.781+0000 7f181f7fe640 1 --2- 172.21.15.63:0/863185329 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f180c035f10 0x7f180c0383d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:44.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.781+0000 7f181f7fe640 10 monclient: get_auth_request con 0x7f180c035f10 auth_method 0 2024-04-26T04:20:44.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.781+0000 7f181e7fc640 10 monclient(hunting): handle_auth_done global_id 4732 payload 824 2024-04-26T04:20:44.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.781+0000 7f181e7fc640 1 -- 172.21.15.63:0/863185329 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f18200008e0 con 0x7f182013c0a0 2024-04-26T04:20:44.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.785+0000 7f181e7fc640 1 -- 172.21.15.63:0/863185329 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f182013c0a0 msgr2=0x7f18201cf2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:20:44.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.785+0000 7f181e7fc640 1 --2- 172.21.15.63:0/863185329 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f182013c0a0 0x7f18201cf2b0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f182013d390 tx=0x7f181400e960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:44.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.785+0000 7f181f7fe640 1 --2- 172.21.15.63:0/863185329 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f180c035f10 0x7f180c0383d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1818003ab0 tx=0x7f1818009820 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:44.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.785+0000 7f1807fff640 1 -- 172.21.15.63:0/863185329 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1814004e40 con 0x7f182013c0a0 2024-04-26T04:20:44.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.785+0000 7f1807fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:20:44.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.785+0000 7f1807fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T04:20:44.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.785+0000 7f1807fff640 10 monclient: dump: 2024-04-26T04:20:44.795 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:20:44.795 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:44.795 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:44.795 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:44.795 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:20:44.795 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:44.795 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:20:44.795 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:20:44.795 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:20:44.795 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:44.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.785+0000 7f1807fff640 1 -- 172.21.15.63:0/863185329 <== mon.1 v2:172.21.15.186:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1814018070 con 0x7f182013c0a0 2024-04-26T04:20:44.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.785+0000 7f1807fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:20:44.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.785+0000 7f1807fff640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:20:44.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.785+0000 7f1807fff640 1 -- 172.21.15.63:0/863185329 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f182013c0a0 msgr2=0x7f18201cf2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:44.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.785+0000 7f1807fff640 1 --2- 172.21.15.63:0/863185329 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f182013c0a0 0x7f18201cf2b0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f182013d390 tx=0x7f181400e960 comp rx=0 tx=0).stop 2024-04-26T04:20:44.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.785+0000 7f1807fff640 1 -- 172.21.15.63:0/863185329 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f182013c0a0 msgr2=0x7f18201cf2b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:20:44.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.785+0000 7f1807fff640 1 --2- 172.21.15.63:0/863185329 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f182013c0a0 0x7f18201cf2b0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:44.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.793+0000 7f18258e4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T04:20:44.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.793+0000 7f18258e4640 1 --2- 172.21.15.63:0/863185329 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f18201cf7f0 0x7f1820137dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:44.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.793+0000 7f18258e4640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:44.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.793+0000 7f18258e4640 1 -- 172.21.15.63:0/863185329 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f18201cfc00 con 0x7f18201cf7f0 2024-04-26T04:20:44.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.793+0000 7f181e7fc640 1 --2- 172.21.15.63:0/863185329 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f18201cf7f0 0x7f1820137dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:44.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.793+0000 7f181e7fc640 10 monclient: get_auth_request con 0x7f18201cf7f0 auth_method 0 2024-04-26T04:20:44.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.793+0000 7f181e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:44.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.793+0000 7f181e7fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:44.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.793+0000 7f181e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:20:44.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.793+0000 7f181e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:20:44.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.793+0000 7f181e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:20:44.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.793+0000 7f181e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:20:44.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.793+0000 7f181e7fc640 10 monclient(hunting): handle_auth_done global_id 4735 payload 824 2024-04-26T04:20:44.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.793+0000 7f181e7fc640 1 -- 172.21.15.63:0/863185329 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f18200683d0 con 0x7f18201cf7f0 2024-04-26T04:20:44.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.797+0000 7f181e7fc640 1 -- 172.21.15.63:0/863185329 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f18201cf7f0 msgr2=0x7f1820137dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:20:44.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.797+0000 7f181e7fc640 1 --2- 172.21.15.63:0/863185329 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f18201cf7f0 0x7f1820137dd0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f182011fb40 tx=0x7f1814004950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:44.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.797+0000 7f1807fff640 1 -- 172.21.15.63:0/863185329 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18140067b0 con 0x7f18201cf7f0 2024-04-26T04:20:44.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.797+0000 7f1807fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:20:44.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.797+0000 7f1807fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T04:20:44.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.797+0000 7f1807fff640 10 monclient: dump: 2024-04-26T04:20:44.799 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:20:44.799 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:44.799 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:44.799 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:44.799 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:20:44.800 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:44.800 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:20:44.800 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:20:44.800 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:20:44.800 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:44.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.797+0000 7f1807fff640 1 -- 172.21.15.63:0/863185329 <== mon.1 v2:172.21.15.186:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4891 (secure 0 0 0) 0x7f18140067b0 con 0x7f18201cf7f0 2024-04-26T04:20:44.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.797+0000 7f1807fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T04:20:44.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.797+0000 7f1807fff640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:20:44.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.797+0000 7f1807fff640 1 -- 172.21.15.63:0/863185329 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f18201cf7f0 msgr2=0x7f1820137dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:44.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.797+0000 7f1807fff640 1 --2- 172.21.15.63:0/863185329 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f18201cf7f0 0x7f1820137dd0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f182011fb40 tx=0x7f1814004950 comp rx=0 tx=0).stop 2024-04-26T04:20:44.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.797+0000 7f1807fff640 1 -- 172.21.15.63:0/863185329 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f18201cf7f0 msgr2=0x7f1820137dd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:20:44.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.797+0000 7f1807fff640 1 --2- 172.21.15.63:0/863185329 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f18201cf7f0 0x7f1820137dd0 secure :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f182011fb40 tx=0x7f1814004950 comp rx=0 tx=0).stop 2024-04-26T04:20:44.801 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-26T04:20:44.801 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b", 2024-04-26T04:20:44.801 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 1, 2024-04-26T04:20:44.801 INFO:teuthology.orchestra.run.smithi063.stdout: "state": "peon", 2024-04-26T04:20:44.801 INFO:teuthology.orchestra.run.smithi063.stdout: "election_epoch": 14, 2024-04-26T04:20:44.801 INFO:teuthology.orchestra.run.smithi063.stdout: "uptime": 135904, 2024-04-26T04:20:44.802 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum": [ 2024-04-26T04:20:44.802 INFO:teuthology.orchestra.run.smithi063.stdout: 0, 2024-04-26T04:20:44.802 INFO:teuthology.orchestra.run.smithi063.stdout: 1, 2024-04-26T04:20:44.802 INFO:teuthology.orchestra.run.smithi063.stdout: 2 2024-04-26T04:20:44.802 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:20:44.802 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_age": 5, 2024-04-26T04:20:44.802 INFO:teuthology.orchestra.run.smithi063.stdout: "features": { 2024-04-26T04:20:44.802 INFO:teuthology.orchestra.run.smithi063.stdout: "required_con": "2450521708007866372", 2024-04-26T04:20:44.802 INFO:teuthology.orchestra.run.smithi063.stdout: "required_mon": [ 2024-04-26T04:20:44.802 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:20:44.802 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:20:44.802 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:20:44.802 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:20:44.803 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:20:44.803 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:20:44.803 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:20:44.803 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:20:44.803 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:20:44.803 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:20:44.803 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:20:44.803 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:20:44.803 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_con": "4540701547738038271", 2024-04-26T04:20:44.803 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_mon": [ 2024-04-26T04:20:44.803 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:20:44.803 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:20:44.803 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:20:44.803 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:20:44.803 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:20:44.804 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:20:44.804 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:20:44.804 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:20:44.804 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:20:44.804 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:20:44.804 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:20:44.804 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:20:44.804 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:44.804 INFO:teuthology.orchestra.run.smithi063.stdout: "outside_quorum": [], 2024-04-26T04:20:44.804 INFO:teuthology.orchestra.run.smithi063.stdout: "extra_probe_peers": [], 2024-04-26T04:20:44.804 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_provider": [], 2024-04-26T04:20:44.804 INFO:teuthology.orchestra.run.smithi063.stdout: "monmap": { 2024-04-26T04:20:44.804 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 1, 2024-04-26T04:20:44.804 INFO:teuthology.orchestra.run.smithi063.stdout: "fsid": "cd1c6a0c-4e4b-4014-89c0-2e535479ef39", 2024-04-26T04:20:44.805 INFO:teuthology.orchestra.run.smithi063.stdout: "modified": "2024-04-26T04:18:03.702339Z", 2024-04-26T04:20:44.805 INFO:teuthology.orchestra.run.smithi063.stdout: "created": "2024-04-26T04:18:03.702339Z", 2024-04-26T04:20:44.805 INFO:teuthology.orchestra.run.smithi063.stdout: "min_mon_release": 19, 2024-04-26T04:20:44.805 INFO:teuthology.orchestra.run.smithi063.stdout: "min_mon_release_name": "squid", 2024-04-26T04:20:44.805 INFO:teuthology.orchestra.run.smithi063.stdout: "election_strategy": 1, 2024-04-26T04:20:44.805 INFO:teuthology.orchestra.run.smithi063.stdout: "disallowed_leaders: ": "", 2024-04-26T04:20:44.805 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": false, 2024-04-26T04:20:44.805 INFO:teuthology.orchestra.run.smithi063.stdout: "tiebreaker_mon": "", 2024-04-26T04:20:44.805 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_ranks: ": "", 2024-04-26T04:20:44.805 INFO:teuthology.orchestra.run.smithi063.stdout: "features": { 2024-04-26T04:20:44.805 INFO:teuthology.orchestra.run.smithi063.stdout: "persistent": [ 2024-04-26T04:20:44.805 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:20:44.805 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:20:44.805 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:20:44.805 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:20:44.806 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:20:44.806 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:20:44.806 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:20:44.806 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:20:44.806 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:20:44.806 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:20:44.806 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:20:44.806 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:20:44.806 INFO:teuthology.orchestra.run.smithi063.stdout: "optional": [] 2024-04-26T04:20:44.806 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:44.806 INFO:teuthology.orchestra.run.smithi063.stdout: "mons": [ 2024-04-26T04:20:44.806 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:44.806 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 0, 2024-04-26T04:20:44.806 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a", 2024-04-26T04:20:44.806 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:20:44.807 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:20:44.807 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:44.807 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:20:44.807 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:3300", 2024-04-26T04:20:44.807 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:20:44.807 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:44.807 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:44.807 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:20:44.807 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6789", 2024-04-26T04:20:44.807 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:20:44.807 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:20:44.807 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:20:44.807 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:44.807 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6789/0", 2024-04-26T04:20:44.807 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6789/0", 2024-04-26T04:20:44.808 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:20:44.808 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:20:44.808 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:20:44.808 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:44.808 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:44.808 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 1, 2024-04-26T04:20:44.808 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b", 2024-04-26T04:20:44.808 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:20:44.808 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:20:44.808 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:44.808 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:20:44.808 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:3300", 2024-04-26T04:20:44.809 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:20:44.809 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:44.809 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:44.809 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:20:44.809 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:6789", 2024-04-26T04:20:44.809 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:20:44.809 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:20:44.809 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:20:44.809 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:44.809 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:6789/0", 2024-04-26T04:20:44.809 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.186:6789/0", 2024-04-26T04:20:44.809 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:20:44.809 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:20:44.809 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:20:44.809 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:44.810 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:44.810 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 2, 2024-04-26T04:20:44.810 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "c", 2024-04-26T04:20:44.810 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:20:44.810 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:20:44.810 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:44.810 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:20:44.810 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:3301", 2024-04-26T04:20:44.810 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:20:44.810 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:44.810 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:44.810 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:20:44.810 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6790", 2024-04-26T04:20:44.810 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:20:44.811 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:20:44.811 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:20:44.811 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:44.811 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6790/0", 2024-04-26T04:20:44.811 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6790/0", 2024-04-26T04:20:44.811 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:20:44.811 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:20:44.811 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:20:44.811 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:20:44.811 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:20:44.811 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:44.811 INFO:teuthology.orchestra.run.smithi063.stdout: "feature_map": { 2024-04-26T04:20:44.811 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": [ 2024-04-26T04:20:44.811 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:44.811 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:20:44.812 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:20:44.812 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 1 2024-04-26T04:20:44.812 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:20:44.812 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:20:44.812 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": [ 2024-04-26T04:20:44.812 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:44.812 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:20:44.812 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:20:44.812 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 2 2024-04-26T04:20:44.812 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:20:44.812 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:20:44.812 INFO:teuthology.orchestra.run.smithi063.stdout: "client": [ 2024-04-26T04:20:44.812 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:44.812 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:20:44.812 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:20:44.813 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 7 2024-04-26T04:20:44.813 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:20:44.813 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:20:44.813 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": [ 2024-04-26T04:20:44.813 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:44.813 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:20:44.813 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:20:44.813 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 1 2024-04-26T04:20:44.813 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:20:44.813 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:20:44.813 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:44.813 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": false 2024-04-26T04:20:44.813 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-26T04:20:44.814 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.797+0000 7f18258e4640 1 -- 172.21.15.63:0/863185329 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f180c035f10 msgr2=0x7f180c0383d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:44.814 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.797+0000 7f18258e4640 1 --2- 172.21.15.63:0/863185329 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f180c035f10 0x7f180c0383d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1818003ab0 tx=0x7f1818009820 comp rx=0 tx=0).stop 2024-04-26T04:20:44.814 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.797+0000 7f18258e4640 10 monclient: shutdown 2024-04-26T04:20:44.814 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.797+0000 7f18258e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:20:44.814 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.797+0000 7f18258e4640 1 -- 172.21.15.63:0/863185329 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f18200a2cd0 msgr2=0x7f182013e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:44.814 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.797+0000 7f18258e4640 1 --2- 172.21.15.63:0/863185329 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f18200a2cd0 0x7f182013e2f0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f18100ae040 tx=0x7f18100191c0 comp rx=0 tx=0).stop 2024-04-26T04:20:44.814 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.797+0000 7f181f7fe640 1 -- 172.21.15.63:0/863185329 reap_dead start 2024-04-26T04:20:44.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.797+0000 7f181f7fe640 1 -- 172.21.15.63:0/863185329 reap_dead start 2024-04-26T04:20:44.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.797+0000 7f18258e4640 1 -- 172.21.15.63:0/863185329 shutdown_connections 2024-04-26T04:20:44.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.797+0000 7f18258e4640 1 -- 172.21.15.63:0/863185329 >> 172.21.15.63:0/863185329 conn(0x7f18200a5f40 msgr2=0x7f18200a1a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:20:44.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.797+0000 7f18258e4640 1 -- 172.21.15.63:0/863185329 shutdown_connections 2024-04-26T04:20:44.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.797+0000 7f18258e4640 1 -- 172.21.15.63:0/863185329 wait complete. 2024-04-26T04:20:44.817 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T04:20:44.817 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-04-26T04:20:44.818 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-04-26T04:20:44.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.929+0000 7f153dacb640 10 monclient: get_monmap_and_config 2024-04-26T04:20:44.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.929+0000 7f153dacb640 10 monclient: build_initial_monmap 2024-04-26T04:20:44.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.929+0000 7f153dacb640 10 monclient: monmap: 2024-04-26T04:20:44.933 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:20:44.933 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:44.933 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:20:44.932974+0000 2024-04-26T04:20:44.933 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:20:44.932974+0000 2024-04-26T04:20:44.933 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:20:44.933 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:44.934 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:20:44.934 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:20:44.934 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:20:44.934 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:20:44.934 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:20:44.934 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:44.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.933+0000 7f153dacb640 1 Processor -- start 2024-04-26T04:20:44.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.933+0000 7f153dacb640 1 -- start start 2024-04-26T04:20:44.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.933+0000 7f153dacb640 10 monclient: init 2024-04-26T04:20:44.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.933+0000 7f153dacb640 10 monclient: _reopen_session rank -1 2024-04-26T04:20:44.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.933+0000 7f153dacb640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-04-26T04:20:44.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.933+0000 7f153dacb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f153809faf0 addr v1:172.21.15.63:6789/0 2024-04-26T04:20:44.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.933+0000 7f153dacb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f15381350c0 addr v1:172.21.15.186:6789/0 2024-04-26T04:20:44.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.933+0000 7f153dacb640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f153809fed0 0x7f153813df50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:44.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.933+0000 7f153dacb640 10 monclient(hunting): picked mon.noname-b con 0x7f153809fed0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:20:44.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.933+0000 7f153dacb640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15380a0380 con 0x7f153809faf0 2024-04-26T04:20:44.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.933+0000 7f153dacb640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15380a28e0 con 0x7f15381350c0 2024-04-26T04:20:44.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.933+0000 7f153dacb640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:44.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.933+0000 7f153dacb640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f1538135550 con 0x7f153809fed0 2024-04-26T04:20:44.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.933+0000 7f153dacb640 10 monclient(hunting): _renew_subs 2024-04-26T04:20:44.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.933+0000 7f153dacb640 10 monclient(hunting): authenticate will time out at 1076.641285s 2024-04-26T04:20:44.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.933+0000 7f15377fe640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f153809faf0 0x7f15380a1ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:48796/0 (socket says 172.21.15.63:48796) 2024-04-26T04:20:44.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.933+0000 7f15377fe640 1 -- 172.21.15.63:0/881688074 learned_addr learned my addr 172.21.15.63:0/881688074 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-04-26T04:20:44.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.933+0000 7f1537fff640 1 --2- 172.21.15.63:0/881688074 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f153809fed0 0x7f153813df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:44.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.933+0000 7f1537fff640 10 monclient(hunting): get_auth_request con 0x7f153809fed0 auth_method 0 2024-04-26T04:20:44.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.933+0000 7f1537fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:44.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.933+0000 7f1537fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:44.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.933+0000 7f1537fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:20:44.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.933+0000 7f1537fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:20:44.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.933+0000 7f1537fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:20:44.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.933+0000 7f1537fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:20:44.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.933+0000 7f1537fff640 10 monclient(hunting): handle_auth_done global_id 14123 payload 274 2024-04-26T04:20:44.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.933+0000 7f1537fff640 1 -- 172.21.15.63:0/881688074 >> v1:172.21.15.186:6789/0 conn(0x7f15381350c0 legacy=0x7f15380a21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:20:44.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.933+0000 7f1537fff640 1 -- 172.21.15.63:0/881688074 >> v1:172.21.15.63:6789/0 conn(0x7f153809faf0 legacy=0x7f15380a1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:20:44.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.933+0000 7f1537fff640 10 monclient: _finish_hunting 0 2024-04-26T04:20:44.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.933+0000 7f1537fff640 1 monclient: found mon.noname-b 2024-04-26T04:20:44.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.933+0000 7f1537fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:20:44.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.933+0000 7f1537fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.63:3301/0 2024-04-26T04:20:44.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.933+0000 7f1537fff640 1 -- 172.21.15.63:0/881688074 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f153809db10 con 0x7f153809fed0 2024-04-26T04:20:44.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.937+0000 7f1537fff640 1 -- 172.21.15.63:0/881688074 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f153809fed0 msgr2=0x7f153813df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:20:44.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.937+0000 7f1537fff640 1 --2- 172.21.15.63:0/881688074 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f153809fed0 0x7f153813df50 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f1530009dc0 tx=0x7f153001c5f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d676b642bda3ad6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:44.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.937+0000 7f15367fc640 1 -- 172.21.15.63:0/881688074 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15300a86d0 con 0x7f153809fed0 2024-04-26T04:20:44.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.937+0000 7f15367fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:20:44.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.937+0000 7f15367fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T04:20:44.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.937+0000 7f15367fc640 10 monclient: dump: 2024-04-26T04:20:44.940 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:20:44.940 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:44.940 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:44.940 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:44.940 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:20:44.941 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:44.941 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:20:44.941 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:20:44.941 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:20:44.941 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:44.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.937+0000 7f15367fc640 10 monclient: _finish_auth 0 2024-04-26T04:20:44.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.937+0000 7f15367fc640 10 monclient: _check_auth_tickets 2024-04-26T04:20:44.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.937+0000 7f15367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:20:44.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.937+0000 7f15367fc640 1 -- 172.21.15.63:0/881688074 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15300a8870 con 0x7f153809fed0 2024-04-26T04:20:44.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.937+0000 7f15367fc640 10 monclient: handle_config config(0 keys) 2024-04-26T04:20:44.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.937+0000 7f15367fc640 1 -- 172.21.15.63:0/881688074 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15300a8c40 con 0x7f153809fed0 2024-04-26T04:20:44.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.937+0000 7f153dacb640 5 monclient: authenticate success, global_id 14123 2024-04-26T04:20:44.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.937+0000 7f153dacb640 10 monclient: get_monmap_and_config success 2024-04-26T04:20:44.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.937+0000 7f153dacb640 4 set_mon_vals no callback set 2024-04-26T04:20:44.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.937+0000 7f15367fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:20:44.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.937+0000 7f15367fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:20:44.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.937+0000 7f15367fc640 10 monclient: dump: 2024-04-26T04:20:44.942 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:20:44.942 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:44.942 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:44.942 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:44.942 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:20:44.942 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:44.942 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:20:44.943 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:20:44.943 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:20:44.943 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:44.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.937+0000 7f153dacb640 10 monclient: shutdown 2024-04-26T04:20:44.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.937+0000 7f153dacb640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:20:44.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.937+0000 7f153dacb640 1 -- 172.21.15.63:0/881688074 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f153809fed0 msgr2=0x7f153813df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:44.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.937+0000 7f153dacb640 1 --2- 172.21.15.63:0/881688074 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f153809fed0 0x7f153813df50 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f1530009dc0 tx=0x7f153001c5f0 comp rx=0 tx=0).stop 2024-04-26T04:20:44.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.937+0000 7f153dacb640 1 -- 172.21.15.63:0/881688074 shutdown_connections 2024-04-26T04:20:44.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.937+0000 7f153dacb640 1 --2- 172.21.15.63:0/881688074 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f153809fed0 0x7f153813df50 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:44.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.937+0000 7f153dacb640 1 -- 172.21.15.63:0/881688074 >> 172.21.15.63:0/881688074 conn(0x7f15380a2bb0 msgr2=0x7f15380a2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:20:44.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.937+0000 7f153dacb640 1 -- 172.21.15.63:0/881688074 shutdown_connections 2024-04-26T04:20:44.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.937+0000 7f153dacb640 1 -- 172.21.15.63:0/881688074 wait complete. 2024-04-26T04:20:44.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.937+0000 7f153dacb640 10 monclient: build_initial_monmap 2024-04-26T04:20:44.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.937+0000 7f153dacb640 10 monclient: monmap: 2024-04-26T04:20:44.948 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:20:44.948 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:20:44.948 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:20:44.948 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:20:44.948 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:20:44.949 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:44.949 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:20:44.949 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:20:44.949 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:20:44.949 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:44.949 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.941+0000 7f153dacb640 1 Processor -- start 2024-04-26T04:20:44.949 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.941+0000 7f153dacb640 1 -- start start 2024-04-26T04:20:44.949 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.941+0000 7f153dacb640 10 monclient: init 2024-04-26T04:20:44.949 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.941+0000 7f153dacb640 10 monclient: _reopen_session rank -1 2024-04-26T04:20:44.949 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.941+0000 7f153dacb640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T04:20:44.949 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.941+0000 7f153dacb640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f153809faf0 0x7f153809aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:44.949 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.941+0000 7f153dacb640 10 monclient(hunting): picked mon.noname-c con 0x7f153809faf0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:20:44.949 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.941+0000 7f153dacb640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f153809fed0 0x7f153809af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:44.949 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.941+0000 7f153dacb640 10 monclient(hunting): picked mon.noname-b con 0x7f153809fed0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:20:44.949 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.941+0000 7f153dacb640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f15381350c0 0x7f153809b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:44.950 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.941+0000 7f153dacb640 10 monclient(hunting): picked mon.noname-a con 0x7f15381350c0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:20:44.950 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.941+0000 7f153dacb640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:44.950 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.941+0000 7f153dacb640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f1538135b30 con 0x7f15381350c0 2024-04-26T04:20:44.950 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.941+0000 7f153dacb640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:44.950 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.941+0000 7f153dacb640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f153809e0d0 con 0x7f153809fed0 2024-04-26T04:20:44.950 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.941+0000 7f153dacb640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:44.950 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.941+0000 7f153dacb640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f1538140cb0 con 0x7f153809faf0 2024-04-26T04:20:44.950 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.941+0000 7f153dacb640 10 monclient(hunting): _renew_subs 2024-04-26T04:20:44.950 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.941+0000 7f153dacb640 10 monclient(hunting): authenticate will time out at 1076.649037s 2024-04-26T04:20:44.950 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.941+0000 7f1537fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f15381350c0 0x7f153809b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:44.950 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.941+0000 7f15377fe640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f153809faf0 0x7f153809aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:44.950 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.941+0000 7f15377fe640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f153809faf0 0x7f153809aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:37214/0 (socket says 172.21.15.63:37214) 2024-04-26T04:20:44.950 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.941+0000 7f15377fe640 1 -- 172.21.15.63:0/1544207514 learned_addr learned my addr 172.21.15.63:0/1544207514 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:20:44.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.941+0000 7f15377fe640 10 monclient(hunting): get_auth_request con 0x7f153809faf0 auth_method 0 2024-04-26T04:20:44.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.941+0000 7f15377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:44.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.941+0000 7f15377fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:44.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.941+0000 7f15377fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:20:44.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.941+0000 7f1537fff640 10 monclient(hunting): get_auth_request con 0x7f15381350c0 auth_method 0 2024-04-26T04:20:44.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.941+0000 7f1537fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:44.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.941+0000 7f1537fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:44.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.941+0000 7f1537fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:20:44.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.941+0000 7f15377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:20:44.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.941+0000 7f15377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:20:44.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.941+0000 7f15377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:20:44.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.941+0000 7f1537fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:20:44.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.941+0000 7f1537fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:20:44.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.941+0000 7f1537fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:20:44.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.941+0000 7f1536ffd640 1 --2- 172.21.15.63:0/1544207514 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f153809fed0 0x7f153809af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:44.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.941+0000 7f1536ffd640 10 monclient(hunting): get_auth_request con 0x7f153809fed0 auth_method 0 2024-04-26T04:20:44.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.941+0000 7f1536ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:44.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.941+0000 7f1536ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:44.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.941+0000 7f1536ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:20:44.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.941+0000 7f15377fe640 10 monclient(hunting): handle_auth_done global_id 14126 payload 824 2024-04-26T04:20:44.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.941+0000 7f15377fe640 1 -- 172.21.15.63:0/1544207514 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f153809fed0 msgr2=0x7f153809af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:44.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.941+0000 7f15377fe640 1 --2- 172.21.15.63:0/1544207514 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f153809fed0 0x7f153809af80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:44.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.941+0000 7f15377fe640 1 -- 172.21.15.63:0/1544207514 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f15381350c0 msgr2=0x7f153809b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:44.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.941+0000 7f15377fe640 1 --2- 172.21.15.63:0/1544207514 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f15381350c0 0x7f153809b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:44.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.941+0000 7f15377fe640 10 monclient: _finish_hunting 0 2024-04-26T04:20:44.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.941+0000 7f15377fe640 1 monclient: found mon.noname-c 2024-04-26T04:20:44.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.941+0000 7f15377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:20:44.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.941+0000 7f15377fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.63:3301/0 2024-04-26T04:20:44.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.941+0000 7f15377fe640 1 -- 172.21.15.63:0/1544207514 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f153811cd40 con 0x7f153809faf0 2024-04-26T04:20:44.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.941+0000 7f1536ffd640 1 --2- 172.21.15.63:0/1544207514 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f153809fed0 0x7f153809af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T04:20:44.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.945+0000 7f15377fe640 1 -- 172.21.15.63:0/1544207514 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f153809faf0 msgr2=0x7f153809aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:20:44.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.945+0000 7f15377fe640 1 --2- 172.21.15.63:0/1544207514 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f153809faf0 0x7f153809aa40 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f152800b430 tx=0x7f152800b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:44.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.945+0000 7f151ffff640 1 -- 172.21.15.63:0/1544207514 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1528015020 con 0x7f153809faf0 2024-04-26T04:20:44.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.945+0000 7f151ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:20:44.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.945+0000 7f151ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:20:44.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.945+0000 7f151ffff640 10 monclient: dump: 2024-04-26T04:20:44.953 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:20:44.953 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:44.953 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:44.953 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:44.953 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:20:44.953 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:44.953 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:20:44.954 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:20:44.954 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:20:44.954 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:44.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.945+0000 7f151ffff640 10 monclient: _finish_auth 0 2024-04-26T04:20:44.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.945+0000 7f151ffff640 10 monclient: _check_auth_tickets 2024-04-26T04:20:44.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.945+0000 7f151ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:20:44.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.945+0000 7f151ffff640 1 -- 172.21.15.63:0/1544207514 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1528011a60 con 0x7f153809faf0 2024-04-26T04:20:44.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.945+0000 7f151ffff640 10 monclient: handle_config config(0 keys) 2024-04-26T04:20:44.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.945+0000 7f151ffff640 1 -- 172.21.15.63:0/1544207514 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1528011e50 con 0x7f153809faf0 2024-04-26T04:20:44.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.945+0000 7f151ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:20:44.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.945+0000 7f151ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:20:44.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.945+0000 7f151ffff640 10 monclient: dump: 2024-04-26T04:20:44.954 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:20:44.954 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:44.954 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:44.955 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:44.955 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:20:44.955 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:44.955 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:20:44.955 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:20:44.955 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:20:44.955 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:44.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.945+0000 7f153dacb640 5 monclient: authenticate success, global_id 14126 2024-04-26T04:20:44.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.945+0000 7f153dacb640 10 monclient: _renew_subs 2024-04-26T04:20:44.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.945+0000 7f153dacb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:20:44.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.945+0000 7f153dacb640 1 -- 172.21.15.63:0/1544207514 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1538073590 con 0x7f153809faf0 2024-04-26T04:20:44.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.945+0000 7f1535ffb640 4 set_mon_vals no callback set 2024-04-26T04:20:44.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.945+0000 7f153dacb640 10 monclient: _renew_subs 2024-04-26T04:20:44.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.945+0000 7f153dacb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:20:44.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.945+0000 7f153dacb640 1 -- 172.21.15.63:0/1544207514 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f153813dd70 con 0x7f153809faf0 2024-04-26T04:20:44.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.945+0000 7f153dacb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:20:44.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.945+0000 7f153dacb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:20:44.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.945+0000 7f153dacb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:20:44.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.945+0000 7f151ffff640 1 -- 172.21.15.63:0/1544207514 <== mon.2 v2:172.21.15.63:3301/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f1528018020 con 0x7f153809faf0 2024-04-26T04:20:44.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.945+0000 7f153dacb640 1 -- 172.21.15.63:0/1544207514 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1538062e10 con 0x7f153809faf0 2024-04-26T04:20:44.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.945+0000 7f151ffff640 1 --2- 172.21.15.63:0/1544207514 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f1524036090 0x7f1524038550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:44.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.945+0000 7f151ffff640 1 -- 172.21.15.63:0/1544207514 <== mon.2 v2:172.21.15.63:3301/0 5 ==== osd_map(112..112 src has 1..112) ==== 4346+0+0 (secure 0 0 0) 0x7f15280486a0 con 0x7f153809faf0 2024-04-26T04:20:44.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.945+0000 7f1536ffd640 1 --2- 172.21.15.63:0/1544207514 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f1524036090 0x7f1524038550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:44.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.945+0000 7f1536ffd640 10 monclient: get_auth_request con 0x7f1524036090 auth_method 0 2024-04-26T04:20:44.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.949+0000 7f1536ffd640 1 --2- 172.21.15.63:0/1544207514 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f1524036090 0x7f1524038550 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f1538133d30 tx=0x7f152c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:44.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.949+0000 7f151ffff640 1 -- 172.21.15.63:0/1544207514 <== mon.2 v2:172.21.15.63:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f152801f070 con 0x7f153809faf0 2024-04-26T04:20:44.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.949+0000 7f151ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:20:44.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:44.949+0000 7f151ffff640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:20:45.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:45.173+0000 7f153dacb640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-04-26T04:20:45.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:45.173+0000 7f153dacb640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-04-26T04:20:45.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:45.173+0000 7f153dacb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:20:45.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:45.173+0000 7f153dacb640 1 -- 172.21.15.63:0/1544207514 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f15380a0580 con 0x7f153809faf0 2024-04-26T04:20:45.178 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:20:45.177+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:20:45.179 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:20:45.177+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=7,config=2,health=12,kv=11,logm=68} crc {auth=2163617280,config=2586702516,health=320688512,kv=2798470503,logm=1263825531}) 2024-04-26T04:20:45.179 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:20:45.177+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=8,config=2,health=12,kv=12,logm=66} crc {auth=1101469633,config=2586702516,health=320688512,kv=3642618011,logm=3611767896}) 2024-04-26T04:20:45.179 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:20:45.177+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:20:45.179 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:20:45.177+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=7,config=2,health=12,kv=11,logm=68} crc {auth=2163617280,config=2586702516,health=320688512,kv=2798470503,logm=1263825531}) 2024-04-26T04:20:45.179 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:20:45.177+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=8,config=2,health=12,kv=12,logm=66} crc {auth=1101469633,config=2586702516,health=320688512,kv=3642618011,logm=135284231}) 2024-04-26T04:20:45.179 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:45.177+0000 7f151ffff640 1 -- 172.21.15.63:0/1544207514 <== mon.2 v2:172.21.15.63:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f153811cd40 con 0x7f153809faf0 2024-04-26T04:20:45.179 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:45.177+0000 7f151ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-04-26T04:20:45.179 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:45.177+0000 7f151ffff640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:20:45.180 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:20:45.177+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:20:45.181 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:20:45.177+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=220436271}) 2024-04-26T04:20:45.181 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:20:45.177+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2359316319}) 2024-04-26T04:20:45.181 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:20:45.177+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:20:45.181 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:20:45.177+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=220436271}) 2024-04-26T04:20:45.181 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:20:45.177+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2016515368}) 2024-04-26T04:20:45.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:45.177+0000 7f153dacb640 1 -- 172.21.15.63:0/1544207514 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f1524036090 msgr2=0x7f1524038550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:45.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:45.177+0000 7f153dacb640 1 --2- 172.21.15.63:0/1544207514 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f1524036090 0x7f1524038550 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f1538133d30 tx=0x7f152c009040 comp rx=0 tx=0).stop 2024-04-26T04:20:45.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:45.177+0000 7f153dacb640 10 monclient: shutdown 2024-04-26T04:20:45.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:45.177+0000 7f153dacb640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:20:45.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:45.177+0000 7f153dacb640 1 -- 172.21.15.63:0/1544207514 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f153809faf0 msgr2=0x7f153809aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:45.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:45.177+0000 7f153dacb640 1 --2- 172.21.15.63:0/1544207514 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f153809faf0 0x7f153809aa40 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f152800b430 tx=0x7f152800b8f0 comp rx=0 tx=0).stop 2024-04-26T04:20:45.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:45.177+0000 7f153dacb640 1 -- 172.21.15.63:0/1544207514 shutdown_connections 2024-04-26T04:20:45.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:45.177+0000 7f153dacb640 1 --2- 172.21.15.63:0/1544207514 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f1524036090 0x7f1524038550 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:45.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:45.177+0000 7f153dacb640 1 --2- 172.21.15.63:0/1544207514 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f15381350c0 0x7f153809b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:45.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:45.177+0000 7f153dacb640 1 --2- 172.21.15.63:0/1544207514 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f153809fed0 0x7f153809af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:45.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:45.177+0000 7f153dacb640 1 --2- 172.21.15.63:0/1544207514 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f153809faf0 0x7f153809aa40 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:45.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:45.177+0000 7f153dacb640 1 -- 172.21.15.63:0/1544207514 >> 172.21.15.63:0/1544207514 conn(0x7f15380a2bb0 msgr2=0x7f1538132d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:20:45.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:45.177+0000 7f153dacb640 1 -- 172.21.15.63:0/1544207514 shutdown_connections 2024-04-26T04:20:45.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:45.177+0000 7f153dacb640 1 -- 172.21.15.63:0/1544207514 wait complete. 2024-04-26T04:20:45.184 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:20:45.181+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:20:45.184 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:20:45.181+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3624471660}) 2024-04-26T04:20:45.184 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:20:45.181+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=994771138}) 2024-04-26T04:20:45.184 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:20:45.181+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:20:45.184 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:20:45.181+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3624471660}) 2024-04-26T04:20:45.184 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:20:45.181+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2047777990}) 2024-04-26T04:20:45.185 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:20:45.181+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:20:45.185 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:20:45.181+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2696504249}) 2024-04-26T04:20:45.185 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:20:45.181+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1741859430}) 2024-04-26T04:20:45.185 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:20:45.181+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:20:45.185 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:20:45.181+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2696504249}) 2024-04-26T04:20:45.185 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:20:45.181+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3154182995}) 2024-04-26T04:20:45.187 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:20:45.185+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:20:45.187 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:20:45.185+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3698366260}) 2024-04-26T04:20:45.187 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:20:45.185+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3663803235}) 2024-04-26T04:20:45.187 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:20:45.185+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:20:45.187 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:20:45.185+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3698366260}) 2024-04-26T04:20:45.187 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:20:45.185+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1483291476}) 2024-04-26T04:20:45.189 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:20:45.185+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:20:45.189 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:20:45.185+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=91,mdsmap=3,mgr=6} crc {logm=1846936891,mdsmap=2840952586,mgr=1450518642}) 2024-04-26T04:20:45.190 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:20:45.185+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=88,mdsmap=3,mgr=8,mgr_command_descs=1} crc {logm=2959248254,mdsmap=2840952586,mgr=4110609431,mgr_command_descs=3057339438}) 2024-04-26T04:20:45.190 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:20:45.185+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:20:45.190 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:20:45.185+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=91,mdsmap=3,mgr=6} crc {logm=1846936891,mdsmap=2840952586,mgr=1450518642}) 2024-04-26T04:20:45.190 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:20:45.185+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=88,mdsmap=3,mgr=8,mgr_command_descs=1} crc {logm=991079527,mdsmap=2840952586,mgr=4110609431,mgr_command_descs=3057339438}) 2024-04-26T04:20:45.201 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-04-26T04:20:45.482 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:20:45.481+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:20:45.482 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:20:45.481+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=1,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=15,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=63} crc {mgr=1747058506,mgr_command_descs=3057339438,mgr_metadata=3709105003,mgrstat=706451192,mon_config_key=723353660,monmap=1192322635,osd_metadata=1397308133,osd_pg_creating=988059225,osdmap=1173461773}) 2024-04-26T04:20:45.482 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:20:45.481+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgr=2,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=16,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=61} crc {mgr=2586702516,mgr_command_descs=3057339438,mgr_metadata=3709105003,mgrstat=964235276,mon_config_key=3877226886,monmap=1192322635,osd_metadata=1397308133,osd_pg_creating=988059225,osdmap=924601704}) 2024-04-26T04:20:45.483 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:20:45.481+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:20:45.483 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:20:45.481+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=1,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=15,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=63} crc {mgr=1747058506,mgr_command_descs=3057339438,mgr_metadata=3709105003,mgrstat=706451192,mon_config_key=723353660,monmap=1192322635,osd_metadata=1397308133,osd_pg_creating=988059225,osdmap=1173461773}) 2024-04-26T04:20:45.483 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:20:45.481+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgr=2,mgr_metadata=1,mgrstat=10,mon_config_key=17,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=60} crc {mgr=2586702516,mgr_metadata=3709105003,mgrstat=1750652664,mon_config_key=4032562882,monmap=1192322635,osd_metadata=1397308133,osd_pg_creating=988059225,osdmap=3370938360}) 2024-04-26T04:20:45.544 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:20:45.541+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:20:45.544 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:20:45.541+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2986220528}) 2024-04-26T04:20:45.544 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:20:45.541+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2723247438}) 2024-04-26T04:20:45.544 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:20:45.541+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:20:45.544 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:20:45.541+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2986220528}) 2024-04-26T04:20:45.544 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:20:45.541+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=933761359}) 2024-04-26T04:20:45.545 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:20:45.541+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:20:45.545 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:20:45.541+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=14} crc {osdmap=865974321}) 2024-04-26T04:20:45.545 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:20:45.541+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=14} crc {osdmap=2512860313}) 2024-04-26T04:20:45.545 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:20:45.541+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:20:45.545 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:20:45.541+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=14} crc {osdmap=865974321}) 2024-04-26T04:20:45.545 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:20:45.541+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=14} crc {osdmap=97287636}) 2024-04-26T04:20:46.201 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-26T04:20:46.202 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T04:20:46.335 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.333+0000 7f453d2e4640 10 monclient: get_monmap_and_config 2024-04-26T04:20:46.335 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.333+0000 7f453d2e4640 10 monclient: build_initial_monmap 2024-04-26T04:20:46.335 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.333+0000 7f453d2e4640 10 monclient: monmap: 2024-04-26T04:20:46.335 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:20:46.335 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:46.336 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:20:46.335380+0000 2024-04-26T04:20:46.336 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:20:46.335380+0000 2024-04-26T04:20:46.336 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:20:46.336 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:46.336 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:20:46.336 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:20:46.336 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:20:46.336 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:20:46.336 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:20:46.336 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:46.339 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.333+0000 7f453d2e4640 1 Processor -- start 2024-04-26T04:20:46.339 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.333+0000 7f453d2e4640 1 -- start start 2024-04-26T04:20:46.339 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.333+0000 7f453d2e4640 10 monclient: init 2024-04-26T04:20:46.339 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.333+0000 7f453d2e4640 10 monclient: _reopen_session rank -1 2024-04-26T04:20:46.339 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.333+0000 7f453d2e4640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-04-26T04:20:46.339 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.333+0000 7f453d2e4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f453809fe60 addr v1:172.21.15.63:6789/0 2024-04-26T04:20:46.339 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.333+0000 7f453d2e4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f45380a0240 addr v1:172.21.15.186:6789/0 2024-04-26T04:20:46.339 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.333+0000 7f453d2e4640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f453813d110 0x7f453813f500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:46.339 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.333+0000 7f453d2e4640 10 monclient(hunting): picked mon.noname-b con 0x7f453813d110 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:20:46.339 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.333+0000 7f453d2e4640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45380a0c60 con 0x7f453809fe60 2024-04-26T04:20:46.339 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.333+0000 7f453d2e4640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45380a0f40 con 0x7f45380a0240 2024-04-26T04:20:46.339 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.333+0000 7f453d2e4640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:46.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.333+0000 7f453d2e4640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f45380a3730 con 0x7f453813d110 2024-04-26T04:20:46.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.333+0000 7f453d2e4640 10 monclient(hunting): _renew_subs 2024-04-26T04:20:46.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.333+0000 7f453d2e4640 10 monclient(hunting): authenticate will time out at 1078.043166s 2024-04-26T04:20:46.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.333+0000 7f4536ffd640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f453809fe60 0x7f45381372f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:48800/0 (socket says 172.21.15.63:48800) 2024-04-26T04:20:46.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.333+0000 7f4536ffd640 1 -- 172.21.15.63:0/3028536054 learned_addr learned my addr 172.21.15.63:0/3028536054 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-04-26T04:20:46.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.333+0000 7f45377fe640 1 --2- 172.21.15.63:0/3028536054 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f453813d110 0x7f453813f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:46.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.333+0000 7f45377fe640 10 monclient(hunting): get_auth_request con 0x7f453813d110 auth_method 0 2024-04-26T04:20:46.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.333+0000 7f45377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:46.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.333+0000 7f45377fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:46.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.333+0000 7f45377fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:20:46.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.333+0000 7f45377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:20:46.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.333+0000 7f45377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:20:46.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.333+0000 7f45377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:20:46.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.333+0000 7f45377fe640 10 monclient(hunting): handle_auth_done global_id 14129 payload 274 2024-04-26T04:20:46.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.333+0000 7f45377fe640 1 -- 172.21.15.63:0/3028536054 >> v1:172.21.15.186:6789/0 conn(0x7f45380a0240 legacy=0x7f45380a2a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:20:46.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.333+0000 7f45377fe640 1 -- 172.21.15.63:0/3028536054 >> v1:172.21.15.63:6789/0 conn(0x7f453809fe60 legacy=0x7f45381372f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:20:46.341 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.333+0000 7f45377fe640 10 monclient: _finish_hunting 0 2024-04-26T04:20:46.341 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.333+0000 7f45377fe640 1 monclient: found mon.noname-b 2024-04-26T04:20:46.341 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.333+0000 7f45377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:20:46.341 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.333+0000 7f45377fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.63:3301/0 2024-04-26T04:20:46.341 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.333+0000 7f45377fe640 1 -- 172.21.15.63:0/3028536054 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45380a1220 con 0x7f453813d110 2024-04-26T04:20:46.342 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.337+0000 7f45377fe640 1 -- 172.21.15.63:0/3028536054 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f453813d110 msgr2=0x7f453813f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:20:46.342 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.337+0000 7f45377fe640 1 --2- 172.21.15.63:0/3028536054 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f453813d110 0x7f453813f500 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f45300097f0 tx=0x7f453001c320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5682222319172673 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:46.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f4535ffb640 1 -- 172.21.15.63:0/3028536054 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45300b0070 con 0x7f453813d110 2024-04-26T04:20:46.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f4535ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:20:46.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f4535ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T04:20:46.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f4535ffb640 10 monclient: dump: 2024-04-26T04:20:46.343 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:20:46.343 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:46.343 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:46.343 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:46.343 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:20:46.343 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:46.343 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:20:46.343 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:20:46.343 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:20:46.343 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:46.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f4535ffb640 10 monclient: _finish_auth 0 2024-04-26T04:20:46.344 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f4535ffb640 10 monclient: _check_auth_tickets 2024-04-26T04:20:46.344 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f4535ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:20:46.344 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f4535ffb640 1 -- 172.21.15.63:0/3028536054 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45300ab600 con 0x7f453813d110 2024-04-26T04:20:46.344 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f453d2e4640 5 monclient: authenticate success, global_id 14129 2024-04-26T04:20:46.344 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f453d2e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T04:20:46.344 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f4535ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T04:20:46.344 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f453d2e4640 10 monclient: get_monmap_and_config success 2024-04-26T04:20:46.344 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f4535ffb640 1 -- 172.21.15.63:0/3028536054 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45300ab9d0 con 0x7f453813d110 2024-04-26T04:20:46.344 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f4535ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:20:46.344 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f453d2e4640 4 set_mon_vals no callback set 2024-04-26T04:20:46.344 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f4535ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:20:46.344 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f4535ffb640 10 monclient: dump: 2024-04-26T04:20:46.344 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:20:46.344 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:46.344 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:46.345 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:46.345 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:20:46.345 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:46.345 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:20:46.345 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:20:46.345 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:20:46.345 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:46.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f453d2e4640 10 monclient: shutdown 2024-04-26T04:20:46.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f453d2e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:20:46.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f453d2e4640 1 -- 172.21.15.63:0/3028536054 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f453813d110 msgr2=0x7f453813f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:46.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f453d2e4640 1 --2- 172.21.15.63:0/3028536054 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f453813d110 0x7f453813f500 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f45300097f0 tx=0x7f453001c320 comp rx=0 tx=0).stop 2024-04-26T04:20:46.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f453d2e4640 1 -- 172.21.15.63:0/3028536054 shutdown_connections 2024-04-26T04:20:46.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f453d2e4640 1 --2- 172.21.15.63:0/3028536054 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f453813d110 0x7f453813f500 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:46.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f453d2e4640 1 -- 172.21.15.63:0/3028536054 >> 172.21.15.63:0/3028536054 conn(0x7f45380a5fb0 msgr2=0x7f45380a63c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:20:46.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f453d2e4640 1 -- 172.21.15.63:0/3028536054 shutdown_connections 2024-04-26T04:20:46.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f453d2e4640 1 -- 172.21.15.63:0/3028536054 wait complete. 2024-04-26T04:20:46.349 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f453d2e4640 10 monclient: build_initial_monmap 2024-04-26T04:20:46.349 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f453d2e4640 10 monclient: monmap: 2024-04-26T04:20:46.350 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:20:46.350 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:20:46.350 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:20:46.350 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:20:46.350 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:20:46.350 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:46.350 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:20:46.350 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:20:46.350 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:20:46.350 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:46.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f453d2e4640 1 Processor -- start 2024-04-26T04:20:46.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f453d2e4640 1 -- start start 2024-04-26T04:20:46.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f453d2e4640 10 monclient: init 2024-04-26T04:20:46.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f453d2e4640 10 monclient: _reopen_session rank -1 2024-04-26T04:20:46.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f453d2e4640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T04:20:46.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f453d2e4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f453809fe60 0x7f453806ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:46.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f453d2e4640 10 monclient(hunting): picked mon.noname-b con 0x7f453809fe60 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:20:46.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f453d2e4640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f45380a0240 0x7f4538070500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:46.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f453d2e4640 10 monclient(hunting): picked mon.noname-a con 0x7f45380a0240 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:20:46.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f453d2e4640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f453813d110 0x7f4538070a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:46.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f453d2e4640 10 monclient(hunting): picked mon.noname-c con 0x7f453813d110 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:20:46.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f453d2e4640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:46.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f453d2e4640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f4538142b40 con 0x7f45380a0240 2024-04-26T04:20:46.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f453d2e4640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:46.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f453d2e4640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f45381429c0 con 0x7f453809fe60 2024-04-26T04:20:46.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f453d2e4640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:46.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f453d2e4640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f4538142cc0 con 0x7f453813d110 2024-04-26T04:20:46.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f453d2e4640 10 monclient(hunting): _renew_subs 2024-04-26T04:20:46.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f453d2e4640 10 monclient(hunting): authenticate will time out at 1078.050455s 2024-04-26T04:20:46.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f45367fc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f45380a0240 0x7f4538070500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:46.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f45367fc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f45380a0240 0x7f4538070500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:36544/0 (socket says 172.21.15.63:36544) 2024-04-26T04:20:46.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f45367fc640 1 -- 172.21.15.63:0/1278028899 learned_addr learned my addr 172.21.15.63:0/1278028899 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:20:46.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f45367fc640 10 monclient(hunting): get_auth_request con 0x7f45380a0240 auth_method 0 2024-04-26T04:20:46.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f45367fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:46.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f45367fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:46.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f45367fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:20:46.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f45377fe640 1 --2- 172.21.15.63:0/1278028899 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f453813d110 0x7f4538070a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:46.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f45367fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:20:46.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f45367fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:20:46.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f45367fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:20:46.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f45377fe640 10 monclient(hunting): get_auth_request con 0x7f453813d110 auth_method 0 2024-04-26T04:20:46.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f45377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:46.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f45377fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:46.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f45377fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:20:46.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f4536ffd640 1 --2- 172.21.15.63:0/1278028899 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f453809fe60 0x7f453806ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:46.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f4536ffd640 10 monclient(hunting): get_auth_request con 0x7f453809fe60 auth_method 0 2024-04-26T04:20:46.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f4536ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:46.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f4536ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:46.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f4536ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:20:46.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f45377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:20:46.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f45377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:20:46.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f45377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:20:46.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f45367fc640 10 monclient(hunting): handle_auth_done global_id 4806 payload 824 2024-04-26T04:20:46.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f45367fc640 1 -- 172.21.15.63:0/1278028899 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f453813d110 msgr2=0x7f4538070a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:46.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f45367fc640 1 --2- 172.21.15.63:0/1278028899 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f453813d110 0x7f4538070a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:46.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f45367fc640 1 -- 172.21.15.63:0/1278028899 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f453809fe60 msgr2=0x7f453806ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:46.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f45367fc640 1 --2- 172.21.15.63:0/1278028899 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f453809fe60 0x7f453806ffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:46.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f45367fc640 10 monclient: _finish_hunting 0 2024-04-26T04:20:46.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f45367fc640 1 monclient: found mon.noname-a 2024-04-26T04:20:46.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f45367fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:20:46.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f45367fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:20:46.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.341+0000 7f45367fc640 1 -- 172.21.15.63:0/1278028899 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4538138650 con 0x7f45380a0240 2024-04-26T04:20:46.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.345+0000 7f45367fc640 1 -- 172.21.15.63:0/1278028899 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f45380a0240 msgr2=0x7f4538070500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:20:46.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.345+0000 7f45367fc640 1 --2- 172.21.15.63:0/1278028899 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f45380a0240 0x7f4538070500 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f452000c7a0 tx=0x7f452000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:46.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.345+0000 7f452f7fe640 1 -- 172.21.15.63:0/1278028899 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4520016020 con 0x7f45380a0240 2024-04-26T04:20:46.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.345+0000 7f452f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:20:46.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.345+0000 7f452f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:20:46.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.345+0000 7f452f7fe640 10 monclient: dump: 2024-04-26T04:20:46.354 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:20:46.354 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:46.354 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:46.354 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:46.354 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:20:46.354 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:46.354 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:20:46.355 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:20:46.355 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:20:46.355 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:46.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.345+0000 7f452f7fe640 10 monclient: _finish_auth 0 2024-04-26T04:20:46.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.345+0000 7f452f7fe640 10 monclient: _check_auth_tickets 2024-04-26T04:20:46.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.345+0000 7f452f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:20:46.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.345+0000 7f453d2e4640 5 monclient: authenticate success, global_id 4806 2024-04-26T04:20:46.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.345+0000 7f452f7fe640 1 -- 172.21.15.63:0/1278028899 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f452000ac60 con 0x7f45380a0240 2024-04-26T04:20:46.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.345+0000 7f453d2e4640 10 monclient: _renew_subs 2024-04-26T04:20:46.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.345+0000 7f453d2e4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:20:46.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.345+0000 7f453d2e4640 1 -- 172.21.15.63:0/1278028899 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4538137df0 con 0x7f45380a0240 2024-04-26T04:20:46.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.345+0000 7f452f7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T04:20:46.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.345+0000 7f452f7fe640 1 -- 172.21.15.63:0/1278028899 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4520012660 con 0x7f45380a0240 2024-04-26T04:20:46.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.345+0000 7f452f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:20:46.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.345+0000 7f452f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:20:46.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.345+0000 7f452f7fe640 10 monclient: dump: 2024-04-26T04:20:46.356 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:20:46.356 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:46.356 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:46.356 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:46.356 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:20:46.356 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:46.356 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:20:46.356 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:20:46.356 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:20:46.356 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:46.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.345+0000 7f453d2e4640 10 monclient: _renew_subs 2024-04-26T04:20:46.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.345+0000 7f453d2e4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:20:46.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.345+0000 7f453d2e4640 1 -- 172.21.15.63:0/1278028899 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45380a7360 con 0x7f45380a0240 2024-04-26T04:20:46.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.345+0000 7f4534ff9640 4 set_mon_vals no callback set 2024-04-26T04:20:46.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.345+0000 7f453d2e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:20:46.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.345+0000 7f453d2e4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:20:46.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.345+0000 7f453d2e4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:20:46.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.345+0000 7f453d2e4640 1 -- 172.21.15.63:0/1278028899 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4538062e10 con 0x7f45380a0240 2024-04-26T04:20:46.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.345+0000 7f452f7fe640 1 -- 172.21.15.63:0/1278028899 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f4520019020 con 0x7f45380a0240 2024-04-26T04:20:46.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.345+0000 7f452f7fe640 1 --2- 172.21.15.63:0/1278028899 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f451c035f10 0x7f451c0383d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:46.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.345+0000 7f452f7fe640 1 -- 172.21.15.63:0/1278028899 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(113..113 src has 1..113) ==== 4738+0+0 (secure 0 0 0) 0x7f4520048de0 con 0x7f45380a0240 2024-04-26T04:20:46.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.345+0000 7f4536ffd640 1 --2- 172.21.15.63:0/1278028899 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f451c035f10 0x7f451c0383d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:46.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.345+0000 7f4536ffd640 10 monclient: get_auth_request con 0x7f451c035f10 auth_method 0 2024-04-26T04:20:46.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.349+0000 7f4536ffd640 1 --2- 172.21.15.63:0/1278028899 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f451c035f10 0x7f451c0383d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f4528007fd0 tx=0x7f4528009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:46.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.349+0000 7f452f7fe640 1 -- 172.21.15.63:0/1278028899 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f4520020030 con 0x7f45380a0240 2024-04-26T04:20:46.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.349+0000 7f452f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:20:46.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.349+0000 7f452f7fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:20:46.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.577+0000 7f453d2e4640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T04:20:46.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.577+0000 7f453d2e4640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T04:20:46.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.577+0000 7f453d2e4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:20:46.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.577+0000 7f453d2e4640 1 -- 172.21.15.63:0/1278028899 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4538068430 con 0x7f45380a0240 2024-04-26T04:20:46.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.577+0000 7f452f7fe640 1 -- 172.21.15.63:0/1278028899 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f452000ae10 con 0x7f45380a0240 2024-04-26T04:20:46.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.577+0000 7f452f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T04:20:46.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.577+0000 7f452f7fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:20:46.580 INFO:teuthology.orchestra.run.smithi063.stdout:{"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cd1c6a0c-4e4b-4014-89c0-2e535479ef39","modified":"2024-04-26T04:18:03.702339Z","created":"2024-04-26T04:18:03.702339Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3300","nonce":0},{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:3300","nonce":0},{"type":"v1","addr":"172.21.15.186:6789","nonce":0}]},"addr":"172.21.15.186:6789/0","public_addr":"172.21.15.186:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3301","nonce":0},{"type":"v1","addr":"172.21.15.63:6790","nonce":0}]},"addr":"172.21.15.63:6790/0","public_addr":"172.21.15.63:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T04:20:46.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.577+0000 7f453d2e4640 1 -- 172.21.15.63:0/1278028899 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f451c035f10 msgr2=0x7f451c0383d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:46.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.577+0000 7f453d2e4640 1 --2- 172.21.15.63:0/1278028899 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f451c035f10 0x7f451c0383d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f4528007fd0 tx=0x7f4528009040 comp rx=0 tx=0).stop 2024-04-26T04:20:46.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.577+0000 7f453d2e4640 10 monclient: shutdown 2024-04-26T04:20:46.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.577+0000 7f453d2e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:20:46.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.577+0000 7f453d2e4640 1 -- 172.21.15.63:0/1278028899 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f45380a0240 msgr2=0x7f4538070500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:46.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.577+0000 7f453d2e4640 1 --2- 172.21.15.63:0/1278028899 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f45380a0240 0x7f4538070500 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f452000c7a0 tx=0x7f452000cc60 comp rx=0 tx=0).stop 2024-04-26T04:20:46.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.577+0000 7f453d2e4640 1 -- 172.21.15.63:0/1278028899 shutdown_connections 2024-04-26T04:20:46.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.577+0000 7f453d2e4640 1 --2- 172.21.15.63:0/1278028899 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f451c035f10 0x7f451c0383d0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:46.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.577+0000 7f453d2e4640 1 --2- 172.21.15.63:0/1278028899 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f453813d110 0x7f4538070a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:46.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.577+0000 7f453d2e4640 1 --2- 172.21.15.63:0/1278028899 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f45380a0240 0x7f4538070500 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:46.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.577+0000 7f453d2e4640 1 --2- 172.21.15.63:0/1278028899 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f453809fe60 0x7f453806ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:46.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.577+0000 7f453d2e4640 1 -- 172.21.15.63:0/1278028899 >> 172.21.15.63:0/1278028899 conn(0x7f45380a5fb0 msgr2=0x7f45380a86a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:20:46.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.581+0000 7f453d2e4640 1 -- 172.21.15.63:0/1278028899 shutdown_connections 2024-04-26T04:20:46.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.581+0000 7f453d2e4640 1 -- 172.21.15.63:0/1278028899 wait complete. 2024-04-26T04:20:46.598 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-26T04:20:46.598 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-04-26T04:20:46.598 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-26T04:20:46.598 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T04:20:46.722 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.721+0000 7f64e226a640 10 monclient: get_monmap_and_config 2024-04-26T04:20:46.722 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.721+0000 7f64e226a640 10 monclient: build_initial_monmap 2024-04-26T04:20:46.722 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.721+0000 7f64e226a640 10 monclient: monmap: 2024-04-26T04:20:46.723 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:20:46.723 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:46.723 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:20:46.722382+0000 2024-04-26T04:20:46.723 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:20:46.722382+0000 2024-04-26T04:20:46.723 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:20:46.723 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:46.723 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:20:46.723 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:20:46.723 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:20:46.723 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:20:46.723 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:20:46.723 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:46.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.721+0000 7f64e226a640 1 Processor -- start 2024-04-26T04:20:46.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.721+0000 7f64e226a640 1 -- start start 2024-04-26T04:20:46.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.721+0000 7f64e226a640 10 monclient: init 2024-04-26T04:20:46.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.721+0000 7f64e226a640 10 monclient: _reopen_session rank -1 2024-04-26T04:20:46.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.721+0000 7f64e226a640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-04-26T04:20:46.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.721+0000 7f64e226a640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f64dc0a2cd0 0x7f64dc0a30b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:46.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.721+0000 7f64e226a640 10 monclient(hunting): picked mon.noname-b con 0x7f64dc0a2cd0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:20:46.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.721+0000 7f64e226a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f64dc0a47d0 addr v1:172.21.15.186:6789/0 2024-04-26T04:20:46.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.721+0000 7f64e226a640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f64dc0a4bd0 0x7f64dc1373f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:46.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.721+0000 7f64e226a640 10 monclient(hunting): picked mon.noname-c con 0x7f64dc0a4bd0 addr v2:172.21.15.186:3300/0 2024-04-26T04:20:46.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.721+0000 7f64e226a640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64dc0a0fb0 con 0x7f64dc0a47d0 2024-04-26T04:20:46.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.721+0000 7f64e226a640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:46.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.721+0000 7f64e226a640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f64dc0a35f0 con 0x7f64dc0a4bd0 2024-04-26T04:20:46.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.721+0000 7f64e226a640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:46.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.721+0000 7f64e226a640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f64dc0a5380 con 0x7f64dc0a2cd0 2024-04-26T04:20:46.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.721+0000 7f64e226a640 10 monclient(hunting): _renew_subs 2024-04-26T04:20:46.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.721+0000 7f64e226a640 10 monclient(hunting): authenticate will time out at 1078.430021s 2024-04-26T04:20:46.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.721+0000 7f64dbfff640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f64dc0a2cd0 0x7f64dc0a30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:46.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.721+0000 7f64dbfff640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f64dc0a2cd0 0x7f64dc0a30b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:37248/0 (socket says 172.21.15.63:37248) 2024-04-26T04:20:46.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.721+0000 7f64dbfff640 1 -- 172.21.15.63:0/2378233327 learned_addr learned my addr 172.21.15.63:0/2378233327 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:20:46.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.721+0000 7f64dbfff640 10 monclient(hunting): get_auth_request con 0x7f64dc0a2cd0 auth_method 0 2024-04-26T04:20:46.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.721+0000 7f64dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:46.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.721+0000 7f64dbfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:46.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.721+0000 7f64dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:20:46.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.721+0000 7f64dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:20:46.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.721+0000 7f64dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:20:46.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.721+0000 7f64dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:20:46.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.721+0000 7f64e0807640 1 --2- 172.21.15.63:0/2378233327 >> v2:172.21.15.186:3300/0 conn(0x7f64dc0a4bd0 0x7f64dc1373f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:46.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.721+0000 7f64e0807640 10 monclient(hunting): get_auth_request con 0x7f64dc0a4bd0 auth_method 0 2024-04-26T04:20:46.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.721+0000 7f64e0807640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:46.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.721+0000 7f64e0807640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:46.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.721+0000 7f64e0807640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:20:46.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.721+0000 7f64dbfff640 10 monclient(hunting): handle_auth_done global_id 14135 payload 274 2024-04-26T04:20:46.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.721+0000 7f64dbfff640 1 -- 172.21.15.63:0/2378233327 >> v2:172.21.15.186:3300/0 conn(0x7f64dc0a4bd0 msgr2=0x7f64dc1373f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:20:46.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.721+0000 7f64dbfff640 1 --2- 172.21.15.63:0/2378233327 >> v2:172.21.15.186:3300/0 conn(0x7f64dc0a4bd0 0x7f64dc1373f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:46.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.721+0000 7f64dbfff640 1 -- 172.21.15.63:0/2378233327 >> v1:172.21.15.186:6789/0 conn(0x7f64dc0a47d0 legacy=0x7f64dc136ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:20:46.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.721+0000 7f64dbfff640 10 monclient: _finish_hunting 0 2024-04-26T04:20:46.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.721+0000 7f64dbfff640 1 monclient: found mon.noname-b 2024-04-26T04:20:46.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.721+0000 7f64dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:20:46.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.721+0000 7f64dbfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.63:3301/0 2024-04-26T04:20:46.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.721+0000 7f64dbfff640 1 -- 172.21.15.63:0/2378233327 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64dc0a1290 con 0x7f64dc0a2cd0 2024-04-26T04:20:46.736 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.725+0000 7f64dbfff640 1 -- 172.21.15.63:0/2378233327 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f64dc0a2cd0 msgr2=0x7f64dc0a30b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:20:46.736 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.725+0000 7f64dbfff640 1 --2- 172.21.15.63:0/2378233327 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f64dc0a2cd0 0x7f64dc0a30b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f64cc009e50 tx=0x7f64cc01c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2946a3da144498a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:46.736 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.725+0000 7f64daffd640 1 -- 172.21.15.63:0/2378233327 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64cc0a55c0 con 0x7f64dc0a2cd0 2024-04-26T04:20:46.736 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.725+0000 7f64daffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:20:46.736 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.725+0000 7f64daffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T04:20:46.736 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.725+0000 7f64daffd640 10 monclient: dump: 2024-04-26T04:20:46.736 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:20:46.736 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:46.736 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:46.736 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:46.737 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:20:46.737 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:46.737 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:20:46.737 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:20:46.737 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:20:46.737 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:46.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.725+0000 7f64daffd640 10 monclient: _finish_auth 0 2024-04-26T04:20:46.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.725+0000 7f64daffd640 10 monclient: _check_auth_tickets 2024-04-26T04:20:46.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.725+0000 7f64daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:20:46.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.725+0000 7f64e226a640 5 monclient: authenticate success, global_id 14135 2024-04-26T04:20:46.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.725+0000 7f64e226a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T04:20:46.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.725+0000 7f64daffd640 1 -- 172.21.15.63:0/2378233327 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64cc0a5760 con 0x7f64dc0a2cd0 2024-04-26T04:20:46.738 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.725+0000 7f64daffd640 10 monclient: handle_config config(0 keys) 2024-04-26T04:20:46.738 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.725+0000 7f64e226a640 10 monclient: get_monmap_and_config success 2024-04-26T04:20:46.738 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.725+0000 7f64daffd640 1 -- 172.21.15.63:0/2378233327 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64cc0a5b30 con 0x7f64dc0a2cd0 2024-04-26T04:20:46.738 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.725+0000 7f64e226a640 4 set_mon_vals no callback set 2024-04-26T04:20:46.738 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.725+0000 7f64daffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:20:46.738 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.725+0000 7f64daffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:20:46.738 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.725+0000 7f64daffd640 10 monclient: dump: 2024-04-26T04:20:46.738 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:20:46.738 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:46.738 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:46.738 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:46.738 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:20:46.738 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:46.739 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:20:46.739 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:20:46.739 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:20:46.739 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:46.739 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.725+0000 7f64e226a640 10 monclient: shutdown 2024-04-26T04:20:46.739 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.725+0000 7f64e226a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:20:46.739 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.725+0000 7f64e226a640 1 -- 172.21.15.63:0/2378233327 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f64dc0a2cd0 msgr2=0x7f64dc0a30b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:46.739 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.725+0000 7f64e226a640 1 --2- 172.21.15.63:0/2378233327 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f64dc0a2cd0 0x7f64dc0a30b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f64cc009e50 tx=0x7f64cc01c940 comp rx=0 tx=0).stop 2024-04-26T04:20:46.739 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.725+0000 7f64e226a640 1 -- 172.21.15.63:0/2378233327 shutdown_connections 2024-04-26T04:20:46.739 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.725+0000 7f64e226a640 1 --2- 172.21.15.63:0/2378233327 >> v2:172.21.15.186:3300/0 conn(0x7f64dc0a4bd0 0x7f64dc1373f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:46.739 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.725+0000 7f64e226a640 1 --2- 172.21.15.63:0/2378233327 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f64dc0a2cd0 0x7f64dc0a30b0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:46.739 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.725+0000 7f64e226a640 1 -- 172.21.15.63:0/2378233327 >> 172.21.15.63:0/2378233327 conn(0x7f64dc0a5f40 msgr2=0x7f64dc0a6320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:20:46.739 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.725+0000 7f64e226a640 1 -- 172.21.15.63:0/2378233327 shutdown_connections 2024-04-26T04:20:46.739 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.725+0000 7f64e226a640 1 -- 172.21.15.63:0/2378233327 wait complete. 2024-04-26T04:20:46.739 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.725+0000 7f64e226a640 10 monclient: build_initial_monmap 2024-04-26T04:20:46.740 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.725+0000 7f64e226a640 10 monclient: monmap: 2024-04-26T04:20:46.740 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:20:46.740 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:20:46.740 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:20:46.740 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:20:46.740 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:20:46.740 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:46.740 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:20:46.740 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:20:46.740 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:20:46.740 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:46.740 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.729+0000 7f64e226a640 1 Processor -- start 2024-04-26T04:20:46.740 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.729+0000 7f64e226a640 1 -- start start 2024-04-26T04:20:46.740 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.729+0000 7f64e226a640 10 monclient: init 2024-04-26T04:20:46.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.729+0000 7f64e226a640 10 monclient: _reopen_session rank -1 2024-04-26T04:20:46.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.729+0000 7f64e226a640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T04:20:46.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.729+0000 7f64e226a640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f64dc0a2cd0 0x7f64dc1ced70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:46.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.729+0000 7f64e226a640 10 monclient(hunting): picked mon.noname-c con 0x7f64dc0a2cd0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:20:46.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.729+0000 7f64e226a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f64dc0a47d0 0x7f64dc1cf540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:46.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.729+0000 7f64e226a640 10 monclient(hunting): picked mon.noname-b con 0x7f64dc0a47d0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:20:46.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.729+0000 7f64e226a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f64dc0a4bd0 0x7f64dc13bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:46.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.729+0000 7f64e226a640 10 monclient(hunting): picked mon.noname-a con 0x7f64dc0a4bd0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:20:46.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.729+0000 7f64e226a640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:46.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.729+0000 7f64e226a640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f64dc141a50 con 0x7f64dc0a4bd0 2024-04-26T04:20:46.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.729+0000 7f64e226a640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:46.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.729+0000 7f64e226a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f64dc137b40 con 0x7f64dc0a47d0 2024-04-26T04:20:46.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.729+0000 7f64e226a640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:46.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.729+0000 7f64e226a640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f64dc141bd0 con 0x7f64dc0a2cd0 2024-04-26T04:20:46.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.729+0000 7f64e226a640 10 monclient(hunting): _renew_subs 2024-04-26T04:20:46.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.729+0000 7f64e226a640 10 monclient(hunting): authenticate will time out at 1078.436783s 2024-04-26T04:20:46.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.729+0000 7f64dbfff640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f64dc0a2cd0 0x7f64dc1ced70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:46.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.729+0000 7f64dbfff640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f64dc0a2cd0 0x7f64dc1ced70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:37264/0 (socket says 172.21.15.63:37264) 2024-04-26T04:20:46.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.729+0000 7f64e0807640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f64dc0a4bd0 0x7f64dc13bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:46.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.729+0000 7f64dbfff640 1 -- 172.21.15.63:0/3847410604 learned_addr learned my addr 172.21.15.63:0/3847410604 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:20:46.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.729+0000 7f64dbfff640 10 monclient(hunting): get_auth_request con 0x7f64dc0a2cd0 auth_method 0 2024-04-26T04:20:46.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.729+0000 7f64dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:46.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.729+0000 7f64dbfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:46.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.729+0000 7f64dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:20:46.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.729+0000 7f64e0807640 10 monclient(hunting): get_auth_request con 0x7f64dc0a4bd0 auth_method 0 2024-04-26T04:20:46.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.729+0000 7f64e0807640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:46.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.729+0000 7f64e0807640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:46.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.729+0000 7f64e0807640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:20:46.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.729+0000 7f64dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:20:46.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.729+0000 7f64dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:20:46.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.729+0000 7f64dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:20:46.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.729+0000 7f64e0807640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:20:46.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.729+0000 7f64e0807640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:20:46.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.729+0000 7f64e0807640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:20:46.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.729+0000 7f64db7fe640 1 --2- 172.21.15.63:0/3847410604 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f64dc0a47d0 0x7f64dc1cf540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:46.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.729+0000 7f64db7fe640 10 monclient(hunting): get_auth_request con 0x7f64dc0a47d0 auth_method 0 2024-04-26T04:20:46.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.729+0000 7f64db7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:46.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.729+0000 7f64db7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:46.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.729+0000 7f64db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:20:46.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.729+0000 7f64dbfff640 10 monclient(hunting): handle_auth_done global_id 14138 payload 824 2024-04-26T04:20:46.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.729+0000 7f64dbfff640 1 -- 172.21.15.63:0/3847410604 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f64dc0a47d0 msgr2=0x7f64dc1cf540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:46.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.729+0000 7f64dbfff640 1 --2- 172.21.15.63:0/3847410604 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f64dc0a47d0 0x7f64dc1cf540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:46.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.729+0000 7f64dbfff640 1 -- 172.21.15.63:0/3847410604 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f64dc0a4bd0 msgr2=0x7f64dc13bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:46.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.729+0000 7f64dbfff640 1 --2- 172.21.15.63:0/3847410604 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f64dc0a4bd0 0x7f64dc13bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:46.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.729+0000 7f64dbfff640 10 monclient: _finish_hunting 0 2024-04-26T04:20:46.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.729+0000 7f64dbfff640 1 monclient: found mon.noname-c 2024-04-26T04:20:46.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.729+0000 7f64dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:20:46.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.729+0000 7f64dbfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.63:3301/0 2024-04-26T04:20:46.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.729+0000 7f64dbfff640 1 -- 172.21.15.63:0/3847410604 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64dc120230 con 0x7f64dc0a2cd0 2024-04-26T04:20:46.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.729+0000 7f64dbfff640 1 -- 172.21.15.63:0/3847410604 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f64dc0a2cd0 msgr2=0x7f64dc1ced70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:20:46.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.729+0000 7f64dbfff640 1 --2- 172.21.15.63:0/3847410604 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f64dc0a2cd0 0x7f64dc1ced70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f64cc01aff0 tx=0x7f64cc0196e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:46.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.729+0000 7f64d8ff9640 1 -- 172.21.15.63:0/3847410604 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64cc0bc020 con 0x7f64dc0a2cd0 2024-04-26T04:20:46.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.729+0000 7f64d8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:20:46.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.729+0000 7f64d8ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:20:46.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.729+0000 7f64d8ff9640 10 monclient: dump: 2024-04-26T04:20:46.744 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:20:46.744 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:46.744 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:46.744 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:46.745 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:20:46.745 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:46.745 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:20:46.745 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:20:46.745 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:20:46.745 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:46.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.729+0000 7f64d8ff9640 10 monclient: _finish_auth 0 2024-04-26T04:20:46.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.729+0000 7f64d8ff9640 10 monclient: _check_auth_tickets 2024-04-26T04:20:46.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.729+0000 7f64d8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:20:46.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.729+0000 7f64d8ff9640 1 -- 172.21.15.63:0/3847410604 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64cc0b8880 con 0x7f64dc0a2cd0 2024-04-26T04:20:46.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.729+0000 7f64d8ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T04:20:46.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.729+0000 7f64d8ff9640 1 -- 172.21.15.63:0/3847410604 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64cc0a55c0 con 0x7f64dc0a2cd0 2024-04-26T04:20:46.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.729+0000 7f64d8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:20:46.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.729+0000 7f64d8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:20:46.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.729+0000 7f64d8ff9640 10 monclient: dump: 2024-04-26T04:20:46.746 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:20:46.746 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:46.746 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:46.746 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:46.746 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:20:46.746 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:46.746 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:20:46.746 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:20:46.746 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:20:46.746 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:46.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.729+0000 7f64da7fc640 4 set_mon_vals no callback set 2024-04-26T04:20:46.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.733+0000 7f64e226a640 5 monclient: authenticate success, global_id 14138 2024-04-26T04:20:46.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.733+0000 7f64e226a640 10 monclient: _renew_subs 2024-04-26T04:20:46.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.733+0000 7f64e226a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:20:46.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.733+0000 7f64e226a640 1 -- 172.21.15.63:0/3847410604 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64dc0a37d0 con 0x7f64dc0a2cd0 2024-04-26T04:20:46.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.733+0000 7f64e226a640 10 monclient: _renew_subs 2024-04-26T04:20:46.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.733+0000 7f64e226a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:20:46.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.733+0000 7f64e226a640 1 -- 172.21.15.63:0/3847410604 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f64dc11fc70 con 0x7f64dc0a2cd0 2024-04-26T04:20:46.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.733+0000 7f64e226a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:20:46.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.733+0000 7f64e226a640 1 --2- 172.21.15.63:0/3847410604 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f64dc1cfa80 0x7f64dc13c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:46.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.733+0000 7f64e226a640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:46.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.733+0000 7f64e226a640 1 -- 172.21.15.63:0/3847410604 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f64dc072d30 con 0x7f64dc1cfa80 2024-04-26T04:20:46.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.733+0000 7f64db7fe640 1 --2- 172.21.15.63:0/3847410604 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f64dc1cfa80 0x7f64dc13c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:46.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.733+0000 7f64db7fe640 10 monclient: get_auth_request con 0x7f64dc1cfa80 auth_method 0 2024-04-26T04:20:46.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.733+0000 7f64db7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:46.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.733+0000 7f64db7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:46.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.733+0000 7f64db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:20:46.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.733+0000 7f64d8ff9640 1 -- 172.21.15.63:0/3847410604 <== mon.2 v2:172.21.15.63:3301/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f64cc0bf020 con 0x7f64dc0a2cd0 2024-04-26T04:20:46.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.733+0000 7f64db7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:20:46.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.733+0000 7f64db7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:20:46.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.733+0000 7f64db7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:20:46.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.733+0000 7f64d8ff9640 1 --2- 172.21.15.63:0/3847410604 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f64bc035ef0 0x7f64bc0383b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:46.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.733+0000 7f64db7fe640 10 monclient(hunting): handle_auth_done global_id 4812 payload 824 2024-04-26T04:20:46.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.733+0000 7f64db7fe640 1 -- 172.21.15.63:0/3847410604 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f64dc062e10 con 0x7f64dc1cfa80 2024-04-26T04:20:46.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.733+0000 7f64d8ff9640 1 -- 172.21.15.63:0/3847410604 <== mon.2 v2:172.21.15.63:3301/0 5 ==== osd_map(114..114 src has 1..114) ==== 4738+0+0 (secure 0 0 0) 0x7f64cc0e3a00 con 0x7f64dc0a2cd0 2024-04-26T04:20:46.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.733+0000 7f64e0807640 1 --2- 172.21.15.63:0/3847410604 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f64bc035ef0 0x7f64bc0383b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:46.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.733+0000 7f64e0807640 10 monclient: get_auth_request con 0x7f64bc035ef0 auth_method 0 2024-04-26T04:20:46.751 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.733+0000 7f64db7fe640 1 -- 172.21.15.63:0/3847410604 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f64dc1cfa80 msgr2=0x7f64dc13c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:20:46.751 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.733+0000 7f64db7fe640 1 --2- 172.21.15.63:0/3847410604 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f64dc1cfa80 0x7f64dc13c0d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f64c400e420 tx=0x7f64c400e8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:46.751 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.733+0000 7f64d8ff9640 1 -- 172.21.15.63:0/3847410604 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64c4007450 con 0x7f64dc1cfa80 2024-04-26T04:20:46.751 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.733+0000 7f64d8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:20:46.751 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.733+0000 7f64d8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:20:46.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.733+0000 7f64d8ff9640 10 monclient: dump: 2024-04-26T04:20:46.752 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:20:46.752 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:46.752 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:46.752 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:46.752 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:20:46.752 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:46.752 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:20:46.752 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:20:46.752 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:20:46.752 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:46.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.733+0000 7f64d8ff9640 1 -- 172.21.15.63:0/3847410604 <== mon.0 v2:172.21.15.63:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f64c4007450 con 0x7f64dc1cfa80 2024-04-26T04:20:46.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.733+0000 7f64d8ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:20:46.753 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.733+0000 7f64d8ff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:20:46.753 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.733+0000 7f64d8ff9640 1 -- 172.21.15.63:0/3847410604 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f64dc1cfa80 msgr2=0x7f64dc13c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:46.753 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.733+0000 7f64d8ff9640 1 --2- 172.21.15.63:0/3847410604 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f64dc1cfa80 0x7f64dc13c0d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f64c400e420 tx=0x7f64c400e8e0 comp rx=0 tx=0).stop 2024-04-26T04:20:46.753 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.733+0000 7f64d8ff9640 1 -- 172.21.15.63:0/3847410604 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f64dc1cfa80 msgr2=0x7f64dc13c0d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:20:46.753 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.733+0000 7f64d8ff9640 1 --2- 172.21.15.63:0/3847410604 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f64dc1cfa80 0x7f64dc13c0d0 secure :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f64c400e420 tx=0x7f64c400e8e0 comp rx=0 tx=0).stop 2024-04-26T04:20:46.753 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.737+0000 7f64e0807640 1 --2- 172.21.15.63:0/3847410604 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f64bc035ef0 0x7f64bc0383b0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f64d4006aa0 tx=0x7f64d4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:46.753 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.741+0000 7f64e226a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T04:20:46.753 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.741+0000 7f64e226a640 1 --2- 172.21.15.63:0/3847410604 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f64dc13e530 0x7f64dc13c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:46.753 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.741+0000 7f64e226a640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:46.753 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.741+0000 7f64e226a640 1 -- 172.21.15.63:0/3847410604 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f64dc11feb0 con 0x7f64dc13e530 2024-04-26T04:20:46.753 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.741+0000 7f64db7fe640 1 --2- 172.21.15.63:0/3847410604 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f64dc13e530 0x7f64dc13c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:46.753 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.741+0000 7f64db7fe640 10 monclient: get_auth_request con 0x7f64dc13e530 auth_method 0 2024-04-26T04:20:46.753 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.741+0000 7f64db7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:46.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.741+0000 7f64db7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:46.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.741+0000 7f64db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:20:46.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.741+0000 7f64db7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:20:46.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.741+0000 7f64db7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:20:46.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.741+0000 7f64db7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:20:46.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.741+0000 7f64db7fe640 10 monclient(hunting): handle_auth_done global_id 4815 payload 824 2024-04-26T04:20:46.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.741+0000 7f64db7fe640 1 -- 172.21.15.63:0/3847410604 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f64dc0683d0 con 0x7f64dc13e530 2024-04-26T04:20:46.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.745+0000 7f64db7fe640 1 -- 172.21.15.63:0/3847410604 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f64dc13e530 msgr2=0x7f64dc13c610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:20:46.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.745+0000 7f64db7fe640 1 --2- 172.21.15.63:0/3847410604 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f64dc13e530 0x7f64dc13c610 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f64c400e420 tx=0x7f64c4003770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:46.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.745+0000 7f64d8ff9640 1 -- 172.21.15.63:0/3847410604 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64c401e960 con 0x7f64dc13e530 2024-04-26T04:20:46.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.745+0000 7f64d8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:20:46.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.745+0000 7f64d8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:20:46.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.745+0000 7f64d8ff9640 10 monclient: dump: 2024-04-26T04:20:46.755 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:20:46.755 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:46.755 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:46.755 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:46.755 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:20:46.755 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:46.755 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:20:46.755 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:20:46.755 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:20:46.755 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:46.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.745+0000 7f64d8ff9640 1 -- 172.21.15.63:0/3847410604 <== mon.0 v2:172.21.15.63:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4723 (secure 0 0 0) 0x7f64c401e960 con 0x7f64dc13e530 2024-04-26T04:20:46.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.745+0000 7f64d8ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T04:20:46.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.745+0000 7f64d8ff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:20:46.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.745+0000 7f64d8ff9640 1 -- 172.21.15.63:0/3847410604 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f64dc13e530 msgr2=0x7f64dc13c610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:46.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.745+0000 7f64d8ff9640 1 --2- 172.21.15.63:0/3847410604 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f64dc13e530 0x7f64dc13c610 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f64c400e420 tx=0x7f64c4003770 comp rx=0 tx=0).stop 2024-04-26T04:20:46.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.745+0000 7f64d8ff9640 1 -- 172.21.15.63:0/3847410604 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f64dc13e530 msgr2=0x7f64dc13c610 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:20:46.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.745+0000 7f64d8ff9640 1 --2- 172.21.15.63:0/3847410604 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f64dc13e530 0x7f64dc13c610 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:46.756 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-26T04:20:46.757 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a", 2024-04-26T04:20:46.757 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 0, 2024-04-26T04:20:46.757 INFO:teuthology.orchestra.run.smithi063.stdout: "state": "leader", 2024-04-26T04:20:46.757 INFO:teuthology.orchestra.run.smithi063.stdout: "election_epoch": 14, 2024-04-26T04:20:46.757 INFO:teuthology.orchestra.run.smithi063.stdout: "uptime": 137898, 2024-04-26T04:20:46.757 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum": [ 2024-04-26T04:20:46.757 INFO:teuthology.orchestra.run.smithi063.stdout: 0, 2024-04-26T04:20:46.757 INFO:teuthology.orchestra.run.smithi063.stdout: 1, 2024-04-26T04:20:46.757 INFO:teuthology.orchestra.run.smithi063.stdout: 2 2024-04-26T04:20:46.757 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:20:46.757 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_age": 7, 2024-04-26T04:20:46.757 INFO:teuthology.orchestra.run.smithi063.stdout: "features": { 2024-04-26T04:20:46.758 INFO:teuthology.orchestra.run.smithi063.stdout: "required_con": "2450521708007866372", 2024-04-26T04:20:46.758 INFO:teuthology.orchestra.run.smithi063.stdout: "required_mon": [ 2024-04-26T04:20:46.758 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:20:46.758 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:20:46.758 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:20:46.758 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:20:46.758 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:20:46.758 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:20:46.758 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:20:46.758 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:20:46.758 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:20:46.759 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:20:46.759 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:20:46.759 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:20:46.759 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_con": "4540701547738038271", 2024-04-26T04:20:46.759 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_mon": [ 2024-04-26T04:20:46.759 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:20:46.759 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:20:46.759 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:20:46.759 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:20:46.759 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:20:46.760 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:20:46.760 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:20:46.760 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:20:46.760 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:20:46.760 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:20:46.760 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:20:46.760 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:20:46.760 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:46.760 INFO:teuthology.orchestra.run.smithi063.stdout: "outside_quorum": [], 2024-04-26T04:20:46.760 INFO:teuthology.orchestra.run.smithi063.stdout: "extra_probe_peers": [], 2024-04-26T04:20:46.760 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_provider": [], 2024-04-26T04:20:46.760 INFO:teuthology.orchestra.run.smithi063.stdout: "monmap": { 2024-04-26T04:20:46.761 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 1, 2024-04-26T04:20:46.761 INFO:teuthology.orchestra.run.smithi063.stdout: "fsid": "cd1c6a0c-4e4b-4014-89c0-2e535479ef39", 2024-04-26T04:20:46.761 INFO:teuthology.orchestra.run.smithi063.stdout: "modified": "2024-04-26T04:18:03.702339Z", 2024-04-26T04:20:46.761 INFO:teuthology.orchestra.run.smithi063.stdout: "created": "2024-04-26T04:18:03.702339Z", 2024-04-26T04:20:46.761 INFO:teuthology.orchestra.run.smithi063.stdout: "min_mon_release": 19, 2024-04-26T04:20:46.761 INFO:teuthology.orchestra.run.smithi063.stdout: "min_mon_release_name": "squid", 2024-04-26T04:20:46.761 INFO:teuthology.orchestra.run.smithi063.stdout: "election_strategy": 1, 2024-04-26T04:20:46.761 INFO:teuthology.orchestra.run.smithi063.stdout: "disallowed_leaders: ": "", 2024-04-26T04:20:46.761 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": false, 2024-04-26T04:20:46.762 INFO:teuthology.orchestra.run.smithi063.stdout: "tiebreaker_mon": "", 2024-04-26T04:20:46.762 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_ranks: ": "", 2024-04-26T04:20:46.762 INFO:teuthology.orchestra.run.smithi063.stdout: "features": { 2024-04-26T04:20:46.762 INFO:teuthology.orchestra.run.smithi063.stdout: "persistent": [ 2024-04-26T04:20:46.762 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:20:46.762 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:20:46.763 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:20:46.763 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:20:46.763 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:20:46.763 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:20:46.763 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:20:46.763 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:20:46.763 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:20:46.763 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:20:46.763 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:20:46.763 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:20:46.763 INFO:teuthology.orchestra.run.smithi063.stdout: "optional": [] 2024-04-26T04:20:46.764 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:46.764 INFO:teuthology.orchestra.run.smithi063.stdout: "mons": [ 2024-04-26T04:20:46.764 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:46.764 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 0, 2024-04-26T04:20:46.764 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a", 2024-04-26T04:20:46.764 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:20:46.764 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:20:46.764 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:46.764 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:20:46.764 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:3300", 2024-04-26T04:20:46.764 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:20:46.764 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:46.764 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:46.765 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:20:46.765 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6789", 2024-04-26T04:20:46.765 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:20:46.765 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:20:46.765 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:20:46.765 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:46.765 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6789/0", 2024-04-26T04:20:46.765 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6789/0", 2024-04-26T04:20:46.765 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:20:46.765 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:20:46.765 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:20:46.765 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:46.765 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:46.766 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 1, 2024-04-26T04:20:46.766 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b", 2024-04-26T04:20:46.766 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:20:46.766 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:20:46.766 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:46.766 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:20:46.766 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:3300", 2024-04-26T04:20:46.766 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:20:46.766 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:46.766 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:46.766 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:20:46.766 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:6789", 2024-04-26T04:20:46.766 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:20:46.766 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:20:46.767 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:20:46.767 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:46.767 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:6789/0", 2024-04-26T04:20:46.767 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.186:6789/0", 2024-04-26T04:20:46.767 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:20:46.767 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:20:46.767 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:20:46.767 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:46.767 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:46.767 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 2, 2024-04-26T04:20:46.767 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "c", 2024-04-26T04:20:46.767 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:20:46.767 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:20:46.767 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:46.767 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:20:46.768 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:3301", 2024-04-26T04:20:46.768 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:20:46.768 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:46.768 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:46.768 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:20:46.768 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6790", 2024-04-26T04:20:46.768 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:20:46.768 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:20:46.768 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:20:46.768 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:46.768 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6790/0", 2024-04-26T04:20:46.768 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6790/0", 2024-04-26T04:20:46.768 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:20:46.769 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:20:46.769 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:20:46.769 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:20:46.769 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:20:46.769 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:46.769 INFO:teuthology.orchestra.run.smithi063.stdout: "feature_map": { 2024-04-26T04:20:46.769 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": [ 2024-04-26T04:20:46.769 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:46.769 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:20:46.769 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:20:46.769 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 1 2024-04-26T04:20:46.769 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:20:46.769 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:20:46.769 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": [ 2024-04-26T04:20:46.770 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:46.770 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:20:46.770 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:20:46.770 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 4 2024-04-26T04:20:46.770 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:20:46.770 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:20:46.770 INFO:teuthology.orchestra.run.smithi063.stdout: "client": [ 2024-04-26T04:20:46.770 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:46.770 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:20:46.770 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:20:46.770 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 1 2024-04-26T04:20:46.770 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:20:46.770 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:20:46.770 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:46.770 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": false 2024-04-26T04:20:46.771 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-26T04:20:46.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.745+0000 7f64e226a640 1 -- 172.21.15.63:0/3847410604 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f64bc035ef0 msgr2=0x7f64bc0383b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:46.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.745+0000 7f64e226a640 1 --2- 172.21.15.63:0/3847410604 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f64bc035ef0 0x7f64bc0383b0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f64d4006aa0 tx=0x7f64d4008040 comp rx=0 tx=0).stop 2024-04-26T04:20:46.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.745+0000 7f64e226a640 10 monclient: shutdown 2024-04-26T04:20:46.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.745+0000 7f64e226a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:20:46.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.745+0000 7f64e226a640 1 -- 172.21.15.63:0/3847410604 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f64dc0a2cd0 msgr2=0x7f64dc1ced70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:46.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.745+0000 7f64e226a640 1 --2- 172.21.15.63:0/3847410604 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f64dc0a2cd0 0x7f64dc1ced70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f64cc01aff0 tx=0x7f64cc0196e0 comp rx=0 tx=0).stop 2024-04-26T04:20:46.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.745+0000 7f64e0807640 1 -- 172.21.15.63:0/3847410604 reap_dead start 2024-04-26T04:20:46.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.745+0000 7f64e0807640 1 -- 172.21.15.63:0/3847410604 reap_dead start 2024-04-26T04:20:46.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.745+0000 7f64e226a640 1 -- 172.21.15.63:0/3847410604 shutdown_connections 2024-04-26T04:20:46.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.745+0000 7f64e226a640 1 -- 172.21.15.63:0/3847410604 >> 172.21.15.63:0/3847410604 conn(0x7f64dc0a5f40 msgr2=0x7f64dc0a1a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:20:46.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.745+0000 7f64e226a640 1 -- 172.21.15.63:0/3847410604 shutdown_connections 2024-04-26T04:20:46.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.745+0000 7f64e226a640 1 -- 172.21.15.63:0/3847410604 wait complete. 2024-04-26T04:20:46.773 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T04:20:46.773 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-26T04:20:46.773 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T04:20:46.814 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29109]: starting. 2024-04-26T04:20:46.814 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29109]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-29108 2024-04-26T04:20:46.814 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29109]: created object 0... 2024-04-26T04:20:46.814 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29109]: created object 25... 2024-04-26T04:20:46.814 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29109]: created object 49... 2024-04-26T04:20:46.815 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29109]: finishing. 2024-04-26T04:20:46.815 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29109]: shutting down. 2024-04-26T04:20:46.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.881+0000 7f9d0e0a2640 10 monclient: get_monmap_and_config 2024-04-26T04:20:46.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.881+0000 7f9d0e0a2640 10 monclient: build_initial_monmap 2024-04-26T04:20:46.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.881+0000 7f9d0e0a2640 10 monclient: monmap: 2024-04-26T04:20:46.882 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:20:46.882 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:46.882 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:20:46.881898+0000 2024-04-26T04:20:46.882 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:20:46.881898+0000 2024-04-26T04:20:46.882 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:20:46.882 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:46.882 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:20:46.882 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:20:46.883 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:20:46.883 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:20:46.883 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:20:46.883 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:46.885 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.881+0000 7f9d0e0a2640 1 Processor -- start 2024-04-26T04:20:46.885 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.881+0000 7f9d0e0a2640 1 -- start start 2024-04-26T04:20:46.885 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.881+0000 7f9d0e0a2640 10 monclient: init 2024-04-26T04:20:46.885 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.881+0000 7f9d0e0a2640 10 monclient: _reopen_session rank -1 2024-04-26T04:20:46.885 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.881+0000 7f9d0e0a2640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-04-26T04:20:46.885 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.881+0000 7f9d0e0a2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9d0809b3b0 addr v1:172.21.15.63:6789/0 2024-04-26T04:20:46.885 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.881+0000 7f9d0e0a2640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f9d08137320 0x7f9d08139710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:46.885 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.881+0000 7f9d0e0a2640 10 monclient(hunting): picked mon.noname-c con 0x7f9d08137320 addr v2:172.21.15.186:3300/0 2024-04-26T04:20:46.885 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.881+0000 7f9d0e0a2640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9d08139c50 0x7f9d0813c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:46.885 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.881+0000 7f9d0e0a2640 10 monclient(hunting): picked mon.noname-b con 0x7f9d08139c50 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:20:46.885 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.881+0000 7f9d0e0a2640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d0809dde0 con 0x7f9d0809b3b0 2024-04-26T04:20:46.885 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.881+0000 7f9d0e0a2640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:46.886 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.881+0000 7f9d0e0a2640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f9d0809c100 con 0x7f9d08137320 2024-04-26T04:20:46.886 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.881+0000 7f9d0e0a2640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:46.886 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.881+0000 7f9d0e0a2640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f9d0809b790 con 0x7f9d08139c50 2024-04-26T04:20:46.886 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.881+0000 7f9d0e0a2640 10 monclient(hunting): _renew_subs 2024-04-26T04:20:46.886 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.881+0000 7f9d0e0a2640 10 monclient(hunting): authenticate will time out at 1078.589464s 2024-04-26T04:20:46.886 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.881+0000 7f9d077fe640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f9d0809b3b0 0x7f9d0809e4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:48806/0 (socket says 172.21.15.63:48806) 2024-04-26T04:20:46.886 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.881+0000 7f9d077fe640 1 -- 172.21.15.63:0/3277588004 learned_addr learned my addr 172.21.15.63:0/3277588004 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-04-26T04:20:46.886 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.881+0000 7f9d07fff640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9d08139c50 0x7f9d0813c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:46.886 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.881+0000 7f9d07fff640 10 monclient(hunting): get_auth_request con 0x7f9d08139c50 auth_method 0 2024-04-26T04:20:46.886 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.881+0000 7f9d07fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:46.886 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.881+0000 7f9d07fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:46.886 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.881+0000 7f9d07fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:20:46.886 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.881+0000 7f9d07fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:20:46.886 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.881+0000 7f9d07fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:20:46.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.881+0000 7f9d07fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:20:46.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.881+0000 7f9d06ffd640 1 --2- 172.21.15.63:0/3277588004 >> v2:172.21.15.186:3300/0 conn(0x7f9d08137320 0x7f9d08139710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:46.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.881+0000 7f9d07fff640 10 monclient(hunting): handle_auth_done global_id 14141 payload 274 2024-04-26T04:20:46.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.881+0000 7f9d07fff640 1 -- 172.21.15.63:0/3277588004 >> v2:172.21.15.186:3300/0 conn(0x7f9d08137320 msgr2=0x7f9d08139710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:20:46.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.881+0000 7f9d07fff640 1 --2- 172.21.15.63:0/3277588004 >> v2:172.21.15.186:3300/0 conn(0x7f9d08137320 0x7f9d08139710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:46.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.881+0000 7f9d07fff640 1 -- 172.21.15.63:0/3277588004 >> v1:172.21.15.63:6789/0 conn(0x7f9d0809b3b0 legacy=0x7f9d0809e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:20:46.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.881+0000 7f9d07fff640 10 monclient: _finish_hunting 0 2024-04-26T04:20:46.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.881+0000 7f9d07fff640 1 monclient: found mon.noname-b 2024-04-26T04:20:46.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.881+0000 7f9d07fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:20:46.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.881+0000 7f9d07fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.63:3301/0 2024-04-26T04:20:46.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.881+0000 7f9d07fff640 1 -- 172.21.15.63:0/3277588004 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d0809c9b0 con 0x7f9d08139c50 2024-04-26T04:20:46.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.885+0000 7f9d07fff640 1 -- 172.21.15.63:0/3277588004 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9d08139c50 msgr2=0x7f9d0813c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:20:46.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.885+0000 7f9d07fff640 1 --2- 172.21.15.63:0/3277588004 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9d08139c50 0x7f9d0813c040 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f9d00009dc0 tx=0x7f9d0001c5f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c02b4a7eb3a599ac server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:46.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.885+0000 7f9d067fc640 1 -- 172.21.15.63:0/3277588004 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d000a86d0 con 0x7f9d08139c50 2024-04-26T04:20:46.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.885+0000 7f9d067fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:20:46.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.885+0000 7f9d067fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T04:20:46.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.885+0000 7f9d067fc640 10 monclient: dump: 2024-04-26T04:20:46.892 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:20:46.892 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:46.892 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:46.892 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:46.892 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:20:46.892 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:46.892 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:20:46.892 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:20:46.893 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:20:46.893 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:46.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.885+0000 7f9d067fc640 10 monclient: _finish_auth 0 2024-04-26T04:20:46.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.885+0000 7f9d067fc640 10 monclient: _check_auth_tickets 2024-04-26T04:20:46.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.885+0000 7f9d067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:20:46.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.885+0000 7f9d067fc640 1 -- 172.21.15.63:0/3277588004 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d000a8870 con 0x7f9d08139c50 2024-04-26T04:20:46.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.885+0000 7f9d0e0a2640 5 monclient: authenticate success, global_id 14141 2024-04-26T04:20:46.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.885+0000 7f9d0e0a2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T04:20:46.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.885+0000 7f9d067fc640 10 monclient: handle_config config(0 keys) 2024-04-26T04:20:46.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.885+0000 7f9d0e0a2640 10 monclient: get_monmap_and_config success 2024-04-26T04:20:46.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.885+0000 7f9d067fc640 1 -- 172.21.15.63:0/3277588004 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d000a8c40 con 0x7f9d08139c50 2024-04-26T04:20:46.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.885+0000 7f9d067fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:20:46.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.885+0000 7f9d0e0a2640 4 set_mon_vals no callback set 2024-04-26T04:20:46.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.885+0000 7f9d067fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:20:46.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.885+0000 7f9d067fc640 10 monclient: dump: 2024-04-26T04:20:46.894 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:20:46.894 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:46.894 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:46.894 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:46.894 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:20:46.894 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:46.894 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:20:46.895 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:20:46.895 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:20:46.895 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:46.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.885+0000 7f9d0e0a2640 10 monclient: shutdown 2024-04-26T04:20:46.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.885+0000 7f9d0e0a2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:20:46.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.885+0000 7f9d0e0a2640 1 -- 172.21.15.63:0/3277588004 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9d08139c50 msgr2=0x7f9d0813c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:46.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.885+0000 7f9d0e0a2640 1 --2- 172.21.15.63:0/3277588004 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9d08139c50 0x7f9d0813c040 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f9d00009dc0 tx=0x7f9d0001c5f0 comp rx=0 tx=0).stop 2024-04-26T04:20:46.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.885+0000 7f9d0e0a2640 1 -- 172.21.15.63:0/3277588004 shutdown_connections 2024-04-26T04:20:46.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.885+0000 7f9d0e0a2640 1 --2- 172.21.15.63:0/3277588004 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9d08139c50 0x7f9d0813c040 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:46.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.885+0000 7f9d0e0a2640 1 --2- 172.21.15.63:0/3277588004 >> v2:172.21.15.186:3300/0 conn(0x7f9d08137320 0x7f9d08139710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:46.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.885+0000 7f9d0e0a2640 1 -- 172.21.15.63:0/3277588004 >> 172.21.15.63:0/3277588004 conn(0x7f9d080a1e70 msgr2=0x7f9d080a2280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:20:46.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.885+0000 7f9d0e0a2640 1 -- 172.21.15.63:0/3277588004 shutdown_connections 2024-04-26T04:20:46.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.885+0000 7f9d0e0a2640 1 -- 172.21.15.63:0/3277588004 wait complete. 2024-04-26T04:20:46.896 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.885+0000 7f9d0e0a2640 10 monclient: build_initial_monmap 2024-04-26T04:20:46.896 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.885+0000 7f9d0e0a2640 10 monclient: monmap: 2024-04-26T04:20:46.896 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:20:46.896 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:20:46.896 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:20:46.896 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:20:46.896 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:20:46.896 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:46.896 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:20:46.896 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:20:46.897 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:20:46.897 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:46.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.885+0000 7f9d0e0a2640 1 Processor -- start 2024-04-26T04:20:46.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.885+0000 7f9d0e0a2640 1 -- start start 2024-04-26T04:20:46.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.885+0000 7f9d0e0a2640 10 monclient: init 2024-04-26T04:20:46.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9d0e0a2640 10 monclient: _reopen_session rank -1 2024-04-26T04:20:46.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9d0e0a2640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T04:20:46.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9d0e0a2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9d0809b3b0 0x7f9d081cc920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:46.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9d0e0a2640 10 monclient(hunting): picked mon.noname-b con 0x7f9d0809b3b0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:20:46.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9d0e0a2640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9d08137320 0x7f9d081364f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:46.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9d0e0a2640 10 monclient(hunting): picked mon.noname-a con 0x7f9d08137320 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:20:46.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9d0e0a2640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9d08139c50 0x7f9d08136a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:46.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9d0e0a2640 10 monclient(hunting): picked mon.noname-c con 0x7f9d08139c50 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:20:46.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9d0e0a2640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:46.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9d0e0a2640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f9d0813f170 con 0x7f9d08137320 2024-04-26T04:20:46.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9d0e0a2640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:46.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9d0e0a2640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9d0813eff0 con 0x7f9d0809b3b0 2024-04-26T04:20:46.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9d0e0a2640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:46.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9d0e0a2640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f9d0813f2f0 con 0x7f9d08139c50 2024-04-26T04:20:46.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9d0e0a2640 10 monclient(hunting): _renew_subs 2024-04-26T04:20:46.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9d0e0a2640 10 monclient(hunting): authenticate will time out at 1078.596271s 2024-04-26T04:20:46.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9d06ffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9d08137320 0x7f9d081364f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:46.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9d07fff640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9d08139c50 0x7f9d08136a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:46.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9d06ffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9d08137320 0x7f9d081364f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:36580/0 (socket says 172.21.15.63:36580) 2024-04-26T04:20:46.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9d07fff640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9d08139c50 0x7f9d08136a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:37286/0 (socket says 172.21.15.63:37286) 2024-04-26T04:20:46.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9d06ffd640 1 -- 172.21.15.63:0/1112782848 learned_addr learned my addr 172.21.15.63:0/1112782848 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:20:46.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9d06ffd640 10 monclient(hunting): get_auth_request con 0x7f9d08137320 auth_method 0 2024-04-26T04:20:46.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9d06ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:46.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9d06ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:46.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9d06ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:20:46.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9d07fff640 10 monclient(hunting): get_auth_request con 0x7f9d08139c50 auth_method 0 2024-04-26T04:20:46.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9d07fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:46.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9d07fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:46.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9d07fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:20:46.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9d07fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:20:46.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9d07fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:20:46.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9d07fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:20:46.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9d06ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:20:46.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9d06ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:20:46.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9d06ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:20:46.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9d077fe640 1 --2- 172.21.15.63:0/1112782848 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9d0809b3b0 0x7f9d081cc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:46.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9d077fe640 10 monclient(hunting): get_auth_request con 0x7f9d0809b3b0 auth_method 0 2024-04-26T04:20:46.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9d077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:46.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9d077fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:46.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9d077fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:20:46.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9d07fff640 10 monclient(hunting): handle_auth_done global_id 14144 payload 824 2024-04-26T04:20:46.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9d07fff640 1 -- 172.21.15.63:0/1112782848 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9d0809b3b0 msgr2=0x7f9d081cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:46.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9d07fff640 1 --2- 172.21.15.63:0/1112782848 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9d0809b3b0 0x7f9d081cc920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:46.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9d07fff640 1 -- 172.21.15.63:0/1112782848 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9d08137320 msgr2=0x7f9d081364f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:46.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9d07fff640 1 --2- 172.21.15.63:0/1112782848 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9d08137320 0x7f9d081364f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:46.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9d07fff640 10 monclient: _finish_hunting 0 2024-04-26T04:20:46.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9d07fff640 1 monclient: found mon.noname-c 2024-04-26T04:20:46.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9d07fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:20:46.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9d07fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.63:3301/0 2024-04-26T04:20:46.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9d07fff640 1 -- 172.21.15.63:0/1112782848 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d0812df70 con 0x7f9d08139c50 2024-04-26T04:20:46.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9d06ffd640 1 --2- 172.21.15.63:0/1112782848 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9d08137320 0x7f9d081364f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T04:20:46.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9d07fff640 1 -- 172.21.15.63:0/1112782848 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9d08139c50 msgr2=0x7f9d08136a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:20:46.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9d07fff640 1 --2- 172.21.15.63:0/1112782848 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9d08139c50 0x7f9d08136a30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9d000af000 tx=0x7f9d00019390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:46.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9ceffff640 1 -- 172.21.15.63:0/1112782848 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d000b0a20 con 0x7f9d08139c50 2024-04-26T04:20:46.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9ceffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:20:46.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9ceffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:20:46.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9ceffff640 10 monclient: dump: 2024-04-26T04:20:46.907 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:20:46.907 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:46.907 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:46.907 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:46.907 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:20:46.907 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:46.908 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:20:46.908 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:20:46.908 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:20:46.908 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:46.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9ceffff640 10 monclient: _finish_auth 0 2024-04-26T04:20:46.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9ceffff640 10 monclient: _check_auth_tickets 2024-04-26T04:20:46.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9ceffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:20:46.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9ceffff640 1 -- 172.21.15.63:0/1112782848 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d000bf070 con 0x7f9d08139c50 2024-04-26T04:20:46.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9ceffff640 10 monclient: handle_config config(0 keys) 2024-04-26T04:20:46.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9ceffff640 1 -- 172.21.15.63:0/1112782848 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d000b0de0 con 0x7f9d08139c50 2024-04-26T04:20:46.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9d0e0a2640 5 monclient: authenticate success, global_id 14144 2024-04-26T04:20:46.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9ceffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:20:46.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9ceffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:20:46.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9ceffff640 10 monclient: dump: 2024-04-26T04:20:46.909 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:20:46.909 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:46.909 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:46.909 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:46.909 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:20:46.909 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:46.909 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:20:46.909 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:20:46.909 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:20:46.910 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:46.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9d0e0a2640 10 monclient: _renew_subs 2024-04-26T04:20:46.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9d05ffb640 4 set_mon_vals no callback set 2024-04-26T04:20:46.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9d0e0a2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:20:46.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9d0e0a2640 1 -- 172.21.15.63:0/1112782848 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d0809cc50 con 0x7f9d08139c50 2024-04-26T04:20:46.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9d0e0a2640 10 monclient: _renew_subs 2024-04-26T04:20:46.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9d0e0a2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:20:46.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.889+0000 7f9d0e0a2640 1 -- 172.21.15.63:0/1112782848 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d0812d9b0 con 0x7f9d08139c50 2024-04-26T04:20:46.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.893+0000 7f9d0e0a2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:20:46.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.893+0000 7f9d0e0a2640 1 --2- 172.21.15.63:0/1112782848 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9d0812a260 0x7f9d0812a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:46.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.893+0000 7f9d0e0a2640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:46.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.893+0000 7f9d0e0a2640 1 -- 172.21.15.63:0/1112782848 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f9d080a47d0 con 0x7f9d0812a260 2024-04-26T04:20:46.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.893+0000 7f9d077fe640 1 --2- 172.21.15.63:0/1112782848 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9d0812a260 0x7f9d0812a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:46.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.893+0000 7f9d077fe640 10 monclient: get_auth_request con 0x7f9d0812a260 auth_method 0 2024-04-26T04:20:46.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.893+0000 7f9d077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:46.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.893+0000 7f9d077fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:46.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.893+0000 7f9d077fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:20:46.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.893+0000 7f9ceffff640 1 -- 172.21.15.63:0/1112782848 <== mon.2 v2:172.21.15.63:3301/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f9d000c4460 con 0x7f9d08139c50 2024-04-26T04:20:46.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.893+0000 7f9d077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:20:46.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.893+0000 7f9d077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:20:46.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.893+0000 7f9d077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:20:46.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.893+0000 7f9ceffff640 1 --2- 172.21.15.63:0/1112782848 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f9cf4035f10 0x7f9cf40383d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:46.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.893+0000 7f9d077fe640 10 monclient(hunting): handle_auth_done global_id 14147 payload 824 2024-04-26T04:20:46.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.893+0000 7f9d077fe640 1 -- 172.21.15.63:0/1112782848 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9d08062e10 con 0x7f9d0812a260 2024-04-26T04:20:46.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.893+0000 7f9ceffff640 1 -- 172.21.15.63:0/1112782848 <== mon.2 v2:172.21.15.63:3301/0 5 ==== osd_map(114..114 src has 1..114) ==== 4738+0+0 (secure 0 0 0) 0x7f9d000e42a0 con 0x7f9d08139c50 2024-04-26T04:20:46.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.893+0000 7f9d06ffd640 1 --2- 172.21.15.63:0/1112782848 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f9cf4035f10 0x7f9cf40383d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:46.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.893+0000 7f9d06ffd640 10 monclient: get_auth_request con 0x7f9cf4035f10 auth_method 0 2024-04-26T04:20:46.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.893+0000 7f9d077fe640 1 -- 172.21.15.63:0/1112782848 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9d0812a260 msgr2=0x7f9d0812a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:20:46.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.893+0000 7f9d077fe640 1 --2- 172.21.15.63:0/1112782848 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9d0812a260 0x7f9d0812a640 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc00e500 tx=0x7f9cfc00e9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:46.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.893+0000 7f9ceffff640 1 -- 172.21.15.63:0/1112782848 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cfc004e40 con 0x7f9d0812a260 2024-04-26T04:20:46.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.893+0000 7f9ceffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:20:46.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.893+0000 7f9ceffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:20:46.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.893+0000 7f9ceffff640 10 monclient: dump: 2024-04-26T04:20:46.912 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:20:46.912 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:46.912 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:46.912 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:46.912 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:20:46.912 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:46.912 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:20:46.913 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:20:46.913 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:20:46.913 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:46.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.893+0000 7f9ceffff640 1 -- 172.21.15.63:0/1112782848 <== mon.2 v2:172.21.15.63:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9cfc018070 con 0x7f9d0812a260 2024-04-26T04:20:46.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.893+0000 7f9ceffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:20:46.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.893+0000 7f9ceffff640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:20:46.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.893+0000 7f9ceffff640 1 -- 172.21.15.63:0/1112782848 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9d0812a260 msgr2=0x7f9d0812a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:46.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.893+0000 7f9ceffff640 1 --2- 172.21.15.63:0/1112782848 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9d0812a260 0x7f9d0812a640 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc00e500 tx=0x7f9cfc00e9c0 comp rx=0 tx=0).stop 2024-04-26T04:20:46.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.893+0000 7f9ceffff640 1 -- 172.21.15.63:0/1112782848 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9d0812a260 msgr2=0x7f9d0812a640 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:20:46.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.893+0000 7f9ceffff640 1 --2- 172.21.15.63:0/1112782848 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9d0812a260 0x7f9d0812a640 secure :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc00e500 tx=0x7f9cfc00e9c0 comp rx=0 tx=0).stop 2024-04-26T04:20:46.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.893+0000 7f9d06ffd640 1 --2- 172.21.15.63:0/1112782848 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f9cf4035f10 0x7f9cf40383d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f9d081cd730 tx=0x7f9cf8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:46.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.901+0000 7f9d0e0a2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T04:20:46.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.901+0000 7f9d0e0a2640 1 --2- 172.21.15.63:0/1112782848 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9d0812ab80 0x7f9d0806ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:46.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.901+0000 7f9d0e0a2640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:46.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.901+0000 7f9d0e0a2640 1 -- 172.21.15.63:0/1112782848 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f9d080a0bb0 con 0x7f9d0812ab80 2024-04-26T04:20:46.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.901+0000 7f9d077fe640 1 --2- 172.21.15.63:0/1112782848 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9d0812ab80 0x7f9d0806ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:46.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.901+0000 7f9d077fe640 10 monclient: get_auth_request con 0x7f9d0812ab80 auth_method 0 2024-04-26T04:20:46.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.901+0000 7f9d077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:46.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.901+0000 7f9d077fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:46.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.901+0000 7f9d077fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:20:46.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.901+0000 7f9d077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:20:46.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.901+0000 7f9d077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:20:46.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.901+0000 7f9d077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:20:46.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.901+0000 7f9d077fe640 10 monclient(hunting): handle_auth_done global_id 14150 payload 824 2024-04-26T04:20:46.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.901+0000 7f9d077fe640 1 -- 172.21.15.63:0/1112782848 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9d0812db80 con 0x7f9d0812ab80 2024-04-26T04:20:46.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.901+0000 7f9d077fe640 1 -- 172.21.15.63:0/1112782848 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9d0812ab80 msgr2=0x7f9d0806ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:20:46.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.901+0000 7f9d077fe640 1 --2- 172.21.15.63:0/1112782848 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9d0812ab80 0x7f9d0806ea30 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc00e4d0 tx=0x7f9cfc004950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:46.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.901+0000 7f9ceffff640 1 -- 172.21.15.63:0/1112782848 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cfc0066f0 con 0x7f9d0812ab80 2024-04-26T04:20:46.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.901+0000 7f9ceffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:20:46.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.901+0000 7f9ceffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:20:46.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.901+0000 7f9ceffff640 10 monclient: dump: 2024-04-26T04:20:46.915 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:20:46.915 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:46.915 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:46.915 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:46.915 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:20:46.915 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:46.915 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:20:46.915 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:20:46.915 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:20:46.916 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:46.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.901+0000 7f9ceffff640 1 -- 172.21.15.63:0/1112782848 <== mon.2 v2:172.21.15.63:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4549 (secure 0 0 0) 0x7f9cfc0066f0 con 0x7f9d0812ab80 2024-04-26T04:20:46.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.901+0000 7f9ceffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T04:20:46.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.901+0000 7f9ceffff640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:20:46.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.901+0000 7f9ceffff640 1 -- 172.21.15.63:0/1112782848 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9d0812ab80 msgr2=0x7f9d0806ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:46.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.901+0000 7f9ceffff640 1 --2- 172.21.15.63:0/1112782848 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9d0812ab80 0x7f9d0806ea30 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc00e4d0 tx=0x7f9cfc004950 comp rx=0 tx=0).stop 2024-04-26T04:20:46.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.901+0000 7f9ceffff640 1 -- 172.21.15.63:0/1112782848 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9d0812ab80 msgr2=0x7f9d0806ea30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:20:46.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.901+0000 7f9ceffff640 1 --2- 172.21.15.63:0/1112782848 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9d0812ab80 0x7f9d0806ea30 secure :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc00e4d0 tx=0x7f9cfc004950 comp rx=0 tx=0).stop 2024-04-26T04:20:46.916 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-26T04:20:46.916 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "c", 2024-04-26T04:20:46.917 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 2, 2024-04-26T04:20:46.917 INFO:teuthology.orchestra.run.smithi063.stdout: "state": "peon", 2024-04-26T04:20:46.917 INFO:teuthology.orchestra.run.smithi063.stdout: "election_epoch": 14, 2024-04-26T04:20:46.917 INFO:teuthology.orchestra.run.smithi063.stdout: "uptime": 7531, 2024-04-26T04:20:46.917 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum": [ 2024-04-26T04:20:46.917 INFO:teuthology.orchestra.run.smithi063.stdout: 0, 2024-04-26T04:20:46.917 INFO:teuthology.orchestra.run.smithi063.stdout: 1, 2024-04-26T04:20:46.917 INFO:teuthology.orchestra.run.smithi063.stdout: 2 2024-04-26T04:20:46.917 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:20:46.917 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_age": 7, 2024-04-26T04:20:46.917 INFO:teuthology.orchestra.run.smithi063.stdout: "features": { 2024-04-26T04:20:46.917 INFO:teuthology.orchestra.run.smithi063.stdout: "required_con": "2450521708007866372", 2024-04-26T04:20:46.917 INFO:teuthology.orchestra.run.smithi063.stdout: "required_mon": [ 2024-04-26T04:20:46.917 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:20:46.917 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:20:46.918 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:20:46.918 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:20:46.918 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:20:46.918 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:20:46.918 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:20:46.918 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:20:46.918 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:20:46.918 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:20:46.918 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:20:46.918 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:20:46.918 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_con": "4540701547738038271", 2024-04-26T04:20:46.918 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_mon": [ 2024-04-26T04:20:46.918 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:20:46.918 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:20:46.919 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:20:46.919 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:20:46.919 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:20:46.919 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:20:46.919 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:20:46.919 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:20:46.919 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:20:46.919 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:20:46.919 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:20:46.919 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:20:46.919 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:46.919 INFO:teuthology.orchestra.run.smithi063.stdout: "outside_quorum": [], 2024-04-26T04:20:46.919 INFO:teuthology.orchestra.run.smithi063.stdout: "extra_probe_peers": [], 2024-04-26T04:20:46.920 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_provider": [], 2024-04-26T04:20:46.920 INFO:teuthology.orchestra.run.smithi063.stdout: "monmap": { 2024-04-26T04:20:46.920 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 1, 2024-04-26T04:20:46.920 INFO:teuthology.orchestra.run.smithi063.stdout: "fsid": "cd1c6a0c-4e4b-4014-89c0-2e535479ef39", 2024-04-26T04:20:46.920 INFO:teuthology.orchestra.run.smithi063.stdout: "modified": "2024-04-26T04:18:03.702339Z", 2024-04-26T04:20:46.920 INFO:teuthology.orchestra.run.smithi063.stdout: "created": "2024-04-26T04:18:03.702339Z", 2024-04-26T04:20:46.920 INFO:teuthology.orchestra.run.smithi063.stdout: "min_mon_release": 19, 2024-04-26T04:20:46.920 INFO:teuthology.orchestra.run.smithi063.stdout: "min_mon_release_name": "squid", 2024-04-26T04:20:46.920 INFO:teuthology.orchestra.run.smithi063.stdout: "election_strategy": 1, 2024-04-26T04:20:46.920 INFO:teuthology.orchestra.run.smithi063.stdout: "disallowed_leaders: ": "", 2024-04-26T04:20:46.920 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": false, 2024-04-26T04:20:46.920 INFO:teuthology.orchestra.run.smithi063.stdout: "tiebreaker_mon": "", 2024-04-26T04:20:46.920 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_ranks: ": "", 2024-04-26T04:20:46.920 INFO:teuthology.orchestra.run.smithi063.stdout: "features": { 2024-04-26T04:20:46.920 INFO:teuthology.orchestra.run.smithi063.stdout: "persistent": [ 2024-04-26T04:20:46.921 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:20:46.921 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:20:46.921 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:20:46.921 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:20:46.921 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:20:46.921 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:20:46.921 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:20:46.921 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:20:46.921 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:20:46.921 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:20:46.921 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:20:46.921 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:20:46.921 INFO:teuthology.orchestra.run.smithi063.stdout: "optional": [] 2024-04-26T04:20:46.921 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:46.922 INFO:teuthology.orchestra.run.smithi063.stdout: "mons": [ 2024-04-26T04:20:46.922 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:46.922 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 0, 2024-04-26T04:20:46.922 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a", 2024-04-26T04:20:46.922 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:20:46.922 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:20:46.922 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:46.922 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:20:46.922 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:3300", 2024-04-26T04:20:46.922 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:20:46.922 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:46.922 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:46.922 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:20:46.922 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6789", 2024-04-26T04:20:46.923 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:20:46.923 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:20:46.923 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:20:46.923 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:46.923 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6789/0", 2024-04-26T04:20:46.923 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6789/0", 2024-04-26T04:20:46.923 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:20:46.923 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:20:46.923 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:20:46.923 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:46.923 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:46.923 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 1, 2024-04-26T04:20:46.923 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b", 2024-04-26T04:20:46.923 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:20:46.923 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:20:46.924 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:46.924 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:20:46.924 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:3300", 2024-04-26T04:20:46.924 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:20:46.924 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:46.924 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:46.924 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:20:46.924 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:6789", 2024-04-26T04:20:46.924 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:20:46.924 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:20:46.924 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:20:46.924 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:46.924 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:6789/0", 2024-04-26T04:20:46.924 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.186:6789/0", 2024-04-26T04:20:46.925 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:20:46.925 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:20:46.925 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:20:46.925 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:46.925 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:46.925 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 2, 2024-04-26T04:20:46.925 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "c", 2024-04-26T04:20:46.925 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:20:46.925 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:20:46.925 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:46.925 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:20:46.925 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:3301", 2024-04-26T04:20:46.925 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:20:46.925 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:46.925 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:46.926 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:20:46.926 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6790", 2024-04-26T04:20:46.926 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:20:46.926 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:20:46.926 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:20:46.926 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:46.926 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6790/0", 2024-04-26T04:20:46.926 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6790/0", 2024-04-26T04:20:46.926 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:20:46.926 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:20:46.926 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:20:46.926 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:20:46.926 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:20:46.927 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:46.927 INFO:teuthology.orchestra.run.smithi063.stdout: "feature_map": { 2024-04-26T04:20:46.927 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": [ 2024-04-26T04:20:46.927 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:46.927 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:20:46.927 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:20:46.927 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 1 2024-04-26T04:20:46.927 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:20:46.927 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:20:46.927 INFO:teuthology.orchestra.run.smithi063.stdout: "client": [ 2024-04-26T04:20:46.927 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:46.927 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:20:46.927 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:20:46.927 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 2 2024-04-26T04:20:46.927 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:20:46.928 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:20:46.928 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:46.928 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": false 2024-04-26T04:20:46.928 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-26T04:20:46.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.905+0000 7f9d0e0a2640 1 -- 172.21.15.63:0/1112782848 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f9cf4035f10 msgr2=0x7f9cf40383d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:46.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.905+0000 7f9d0e0a2640 1 --2- 172.21.15.63:0/1112782848 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f9cf4035f10 0x7f9cf40383d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f9d081cd730 tx=0x7f9cf8008040 comp rx=0 tx=0).stop 2024-04-26T04:20:46.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.905+0000 7f9d0e0a2640 10 monclient: shutdown 2024-04-26T04:20:46.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.905+0000 7f9d0e0a2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:20:46.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.905+0000 7f9d0e0a2640 1 -- 172.21.15.63:0/1112782848 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9d08139c50 msgr2=0x7f9d08136a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:46.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.905+0000 7f9d0e0a2640 1 --2- 172.21.15.63:0/1112782848 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9d08139c50 0x7f9d08136a30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9d000af000 tx=0x7f9d00019390 comp rx=0 tx=0).stop 2024-04-26T04:20:46.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.905+0000 7f9d07fff640 1 -- 172.21.15.63:0/1112782848 reap_dead start 2024-04-26T04:20:46.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.905+0000 7f9d07fff640 1 -- 172.21.15.63:0/1112782848 reap_dead start 2024-04-26T04:20:46.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.905+0000 7f9d0e0a2640 1 -- 172.21.15.63:0/1112782848 shutdown_connections 2024-04-26T04:20:46.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.905+0000 7f9d0e0a2640 1 -- 172.21.15.63:0/1112782848 >> 172.21.15.63:0/1112782848 conn(0x7f9d080a1e70 msgr2=0x7f9d0809faa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:20:46.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.905+0000 7f9d0e0a2640 1 -- 172.21.15.63:0/1112782848 shutdown_connections 2024-04-26T04:20:46.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:46.905+0000 7f9d0e0a2640 1 -- 172.21.15.63:0/1112782848 wait complete. 2024-04-26T04:20:46.930 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T04:20:46.930 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-26T04:20:46.931 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T04:20:47.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.041+0000 7ff03d702640 10 monclient: get_monmap_and_config 2024-04-26T04:20:47.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.041+0000 7ff03d702640 10 monclient: build_initial_monmap 2024-04-26T04:20:47.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.041+0000 7ff03d702640 10 monclient: monmap: 2024-04-26T04:20:47.042 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:20:47.042 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:47.042 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:20:47.042062+0000 2024-04-26T04:20:47.042 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:20:47.042062+0000 2024-04-26T04:20:47.042 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:20:47.042 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:47.043 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:20:47.043 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:20:47.043 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:20:47.043 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:20:47.043 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:20:47.043 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:47.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.041+0000 7ff03d702640 1 Processor -- start 2024-04-26T04:20:47.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.041+0000 7ff03d702640 1 -- start start 2024-04-26T04:20:47.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.041+0000 7ff03d702640 10 monclient: init 2024-04-26T04:20:47.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.041+0000 7ff03d702640 10 monclient: _reopen_session rank -1 2024-04-26T04:20:47.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.041+0000 7ff03d702640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-04-26T04:20:47.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.041+0000 7ff03d702640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff0380a2cd0 addr v1:172.21.15.63:6789/0 2024-04-26T04:20:47.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.041+0000 7ff03d702640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff0380a30b0 addr v1:172.21.15.186:6789/0 2024-04-26T04:20:47.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.041+0000 7ff03d702640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff03813d720 0x7ff03813fb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:47.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.041+0000 7ff03d702640 10 monclient(hunting): picked mon.noname-c con 0x7ff03813d720 addr v2:172.21.15.186:3300/0 2024-04-26T04:20:47.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.041+0000 7ff03d702640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0380a4ee0 con 0x7ff0380a2cd0 2024-04-26T04:20:47.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.041+0000 7ff03d702640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0380a0fb0 con 0x7ff0380a30b0 2024-04-26T04:20:47.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.041+0000 7ff03d702640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:47.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.041+0000 7ff03d702640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7ff0380a5380 con 0x7ff03813d720 2024-04-26T04:20:47.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.041+0000 7ff03d702640 10 monclient(hunting): _renew_subs 2024-04-26T04:20:47.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.041+0000 7ff03d702640 10 monclient(hunting): authenticate will time out at 1078.749742s 2024-04-26T04:20:47.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.041+0000 7ff036ffd640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7ff0380a2cd0 0x7ff0380a47d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:48810/0 (socket says 172.21.15.63:48810) 2024-04-26T04:20:47.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.041+0000 7ff036ffd640 1 -- 172.21.15.63:0/776619040 learned_addr learned my addr 172.21.15.63:0/776619040 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-04-26T04:20:47.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.041+0000 7ff0377fe640 1 --2- 172.21.15.63:0/776619040 >> v2:172.21.15.186:3300/0 conn(0x7ff03813d720 0x7ff03813fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:47.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.041+0000 7ff0377fe640 10 monclient(hunting): get_auth_request con 0x7ff03813d720 auth_method 0 2024-04-26T04:20:47.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.041+0000 7ff0377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:47.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.041+0000 7ff0377fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:47.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.041+0000 7ff0377fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:20:47.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.041+0000 7ff0377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:20:47.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.041+0000 7ff0377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:20:47.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.041+0000 7ff0377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:20:47.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.041+0000 7ff0377fe640 10 monclient(hunting): handle_auth_done global_id 4753 payload 274 2024-04-26T04:20:47.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.041+0000 7ff0377fe640 1 -- 172.21.15.63:0/776619040 >> v1:172.21.15.186:6789/0 conn(0x7ff0380a30b0 legacy=0x7ff03813c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:20:47.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.041+0000 7ff0377fe640 1 -- 172.21.15.63:0/776619040 >> v1:172.21.15.63:6789/0 conn(0x7ff0380a2cd0 legacy=0x7ff0380a47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:20:47.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.041+0000 7ff0377fe640 10 monclient: _finish_hunting 0 2024-04-26T04:20:47.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.041+0000 7ff0377fe640 1 monclient: found mon.noname-c 2024-04-26T04:20:47.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.041+0000 7ff0377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:20:47.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.041+0000 7ff0377fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.186:3300/0 2024-04-26T04:20:47.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.041+0000 7ff0377fe640 1 -- 172.21.15.63:0/776619040 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0380a1290 con 0x7ff03813d720 2024-04-26T04:20:47.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.045+0000 7ff0377fe640 1 -- 172.21.15.63:0/776619040 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff03813d720 msgr2=0x7ff03813fb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:20:47.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.045+0000 7ff0377fe640 1 --2- 172.21.15.63:0/776619040 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff03813d720 0x7ff03813fb10 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7ff030009d90 tx=0x7ff03001c840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c8afb703ed95a14 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:47.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.045+0000 7ff035ffb640 1 -- 172.21.15.63:0/776619040 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0300b2070 con 0x7ff03813d720 2024-04-26T04:20:47.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.045+0000 7ff035ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:20:47.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.045+0000 7ff035ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:20:47.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.045+0000 7ff035ffb640 10 monclient: dump: 2024-04-26T04:20:47.050 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:20:47.050 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:47.050 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:47.050 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:47.051 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:20:47.051 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:47.051 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:20:47.051 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:20:47.051 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:20:47.051 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:47.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.045+0000 7ff035ffb640 10 monclient: _finish_auth 0 2024-04-26T04:20:47.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.045+0000 7ff035ffb640 10 monclient: _check_auth_tickets 2024-04-26T04:20:47.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.045+0000 7ff035ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:20:47.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.045+0000 7ff03d702640 5 monclient: authenticate success, global_id 4753 2024-04-26T04:20:47.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.045+0000 7ff03d702640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T04:20:47.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.045+0000 7ff035ffb640 1 -- 172.21.15.63:0/776619040 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0300a58f0 con 0x7ff03813d720 2024-04-26T04:20:47.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.045+0000 7ff035ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T04:20:47.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.045+0000 7ff03d702640 10 monclient: get_monmap_and_config success 2024-04-26T04:20:47.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.045+0000 7ff035ffb640 1 -- 172.21.15.63:0/776619040 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0300a5cc0 con 0x7ff03813d720 2024-04-26T04:20:47.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.045+0000 7ff035ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:20:47.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.045+0000 7ff03d702640 4 set_mon_vals no callback set 2024-04-26T04:20:47.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.045+0000 7ff035ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T04:20:47.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.045+0000 7ff035ffb640 10 monclient: dump: 2024-04-26T04:20:47.052 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:20:47.052 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:47.052 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:47.052 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:47.052 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:20:47.052 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:47.052 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:20:47.052 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:20:47.052 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:20:47.052 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:47.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.049+0000 7ff03d702640 10 monclient: shutdown 2024-04-26T04:20:47.053 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.049+0000 7ff03d702640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:20:47.053 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.049+0000 7ff03d702640 1 -- 172.21.15.63:0/776619040 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff03813d720 msgr2=0x7ff03813fb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:47.053 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.049+0000 7ff03d702640 1 --2- 172.21.15.63:0/776619040 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff03813d720 0x7ff03813fb10 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7ff030009d90 tx=0x7ff03001c840 comp rx=0 tx=0).stop 2024-04-26T04:20:47.053 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.049+0000 7ff03d702640 1 -- 172.21.15.63:0/776619040 shutdown_connections 2024-04-26T04:20:47.053 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.049+0000 7ff03d702640 1 --2- 172.21.15.63:0/776619040 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff03813d720 0x7ff03813fb10 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:47.053 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.049+0000 7ff03d702640 1 -- 172.21.15.63:0/776619040 >> 172.21.15.63:0/776619040 conn(0x7ff0380a5f40 msgr2=0x7ff0380a6320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:20:47.053 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.049+0000 7ff03d702640 1 -- 172.21.15.63:0/776619040 shutdown_connections 2024-04-26T04:20:47.053 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.049+0000 7ff03d702640 1 -- 172.21.15.63:0/776619040 wait complete. 2024-04-26T04:20:47.053 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.049+0000 7ff03d702640 10 monclient: build_initial_monmap 2024-04-26T04:20:47.053 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.049+0000 7ff03d702640 10 monclient: monmap: 2024-04-26T04:20:47.053 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:20:47.053 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:20:47.053 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:20:47.053 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:20:47.053 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:20:47.054 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:47.054 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:20:47.054 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:20:47.054 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:20:47.054 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:47.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.049+0000 7ff03d702640 1 Processor -- start 2024-04-26T04:20:47.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.049+0000 7ff03d702640 1 -- start start 2024-04-26T04:20:47.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.049+0000 7ff03d702640 10 monclient: init 2024-04-26T04:20:47.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.049+0000 7ff03d702640 10 monclient: _reopen_session rank -1 2024-04-26T04:20:47.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.049+0000 7ff03d702640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T04:20:47.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.049+0000 7ff03d702640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff0380a2cd0 0x7ff038132840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:47.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.049+0000 7ff03d702640 10 monclient(hunting): picked mon.noname-c con 0x7ff0380a2cd0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:20:47.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.049+0000 7ff03d702640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff0380a30b0 0x7ff038132d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:47.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.049+0000 7ff03d702640 10 monclient(hunting): picked mon.noname-a con 0x7ff0380a30b0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:20:47.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.049+0000 7ff03d702640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff03813d720 0x7ff0381332c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:47.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.049+0000 7ff03d702640 10 monclient(hunting): picked mon.noname-b con 0x7ff03813d720 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:20:47.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.049+0000 7ff03d702640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:47.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.049+0000 7ff03d702640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff038142c00 con 0x7ff0380a30b0 2024-04-26T04:20:47.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.049+0000 7ff03d702640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:47.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.049+0000 7ff03d702640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff038142a80 con 0x7ff03813d720 2024-04-26T04:20:47.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.049+0000 7ff03d702640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:47.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.049+0000 7ff03d702640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7ff038142d80 con 0x7ff0380a2cd0 2024-04-26T04:20:47.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.049+0000 7ff03d702640 10 monclient(hunting): _renew_subs 2024-04-26T04:20:47.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.049+0000 7ff03d702640 10 monclient(hunting): authenticate will time out at 1078.757718s 2024-04-26T04:20:47.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.049+0000 7ff036ffd640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff0380a2cd0 0x7ff038132840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:47.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.049+0000 7ff0367fc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff0380a30b0 0x7ff038132d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:47.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.049+0000 7ff036ffd640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff0380a2cd0 0x7ff038132840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:37306/0 (socket says 172.21.15.63:37306) 2024-04-26T04:20:47.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.049+0000 7ff0367fc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff0380a30b0 0x7ff038132d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:36582/0 (socket says 172.21.15.63:36582) 2024-04-26T04:20:47.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.049+0000 7ff036ffd640 1 -- 172.21.15.63:0/2976292459 learned_addr learned my addr 172.21.15.63:0/2976292459 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:20:47.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.049+0000 7ff036ffd640 10 monclient(hunting): get_auth_request con 0x7ff0380a2cd0 auth_method 0 2024-04-26T04:20:47.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.049+0000 7ff036ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:47.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.049+0000 7ff036ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:47.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.049+0000 7ff036ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:20:47.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.049+0000 7ff0367fc640 10 monclient(hunting): get_auth_request con 0x7ff0380a30b0 auth_method 0 2024-04-26T04:20:47.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.049+0000 7ff0367fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:47.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.049+0000 7ff0367fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:47.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.049+0000 7ff0367fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:20:47.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.049+0000 7ff0367fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:20:47.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.049+0000 7ff0367fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:20:47.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.049+0000 7ff0367fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:20:47.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.049+0000 7ff036ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:20:47.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.049+0000 7ff036ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:20:47.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.049+0000 7ff036ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:20:47.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.049+0000 7ff0377fe640 1 --2- 172.21.15.63:0/2976292459 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff03813d720 0x7ff0381332c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:47.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.049+0000 7ff0377fe640 10 monclient(hunting): get_auth_request con 0x7ff03813d720 auth_method 0 2024-04-26T04:20:47.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.049+0000 7ff0377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:47.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.049+0000 7ff0377fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:47.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.049+0000 7ff0377fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:20:47.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.049+0000 7ff0367fc640 10 monclient(hunting): handle_auth_done global_id 4821 payload 824 2024-04-26T04:20:47.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.049+0000 7ff0367fc640 1 -- 172.21.15.63:0/2976292459 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff0380a2cd0 msgr2=0x7ff038132840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:47.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.049+0000 7ff0367fc640 1 --2- 172.21.15.63:0/2976292459 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff0380a2cd0 0x7ff038132840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:47.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.049+0000 7ff0367fc640 1 -- 172.21.15.63:0/2976292459 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff03813d720 msgr2=0x7ff0381332c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:47.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.049+0000 7ff0367fc640 1 --2- 172.21.15.63:0/2976292459 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff03813d720 0x7ff0381332c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:47.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.049+0000 7ff0367fc640 10 monclient: _finish_hunting 0 2024-04-26T04:20:47.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.049+0000 7ff0367fc640 1 monclient: found mon.noname-a 2024-04-26T04:20:47.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.049+0000 7ff0367fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:20:47.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.049+0000 7ff0367fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:20:47.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.049+0000 7ff0367fc640 1 -- 172.21.15.63:0/2976292459 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff038134cc0 con 0x7ff0380a30b0 2024-04-26T04:20:47.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.049+0000 7ff036ffd640 1 --2- 172.21.15.63:0/2976292459 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff0380a2cd0 0x7ff038132840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T04:20:47.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.053+0000 7ff0367fc640 1 -- 172.21.15.63:0/2976292459 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff0380a30b0 msgr2=0x7ff038132d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:20:47.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.053+0000 7ff0367fc640 1 --2- 172.21.15.63:0/2976292459 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff0380a30b0 0x7ff038132d80 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff02800c3e0 tx=0x7ff02800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:47.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.053+0000 7ff02f7fe640 1 -- 172.21.15.63:0/2976292459 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0280119a0 con 0x7ff0380a30b0 2024-04-26T04:20:47.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.053+0000 7ff02f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:20:47.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.053+0000 7ff02f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:20:47.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.053+0000 7ff02f7fe640 10 monclient: dump: 2024-04-26T04:20:47.063 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:20:47.063 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:47.063 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:47.063 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:47.063 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:20:47.064 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:47.064 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:20:47.064 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:20:47.064 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:20:47.064 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:47.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.053+0000 7ff02f7fe640 10 monclient: _finish_auth 0 2024-04-26T04:20:47.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.053+0000 7ff02f7fe640 10 monclient: _check_auth_tickets 2024-04-26T04:20:47.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.053+0000 7ff02f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:20:47.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.053+0000 7ff02f7fe640 1 -- 172.21.15.63:0/2976292459 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff028017070 con 0x7ff0380a30b0 2024-04-26T04:20:47.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.053+0000 7ff02f7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T04:20:47.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.053+0000 7ff02f7fe640 1 -- 172.21.15.63:0/2976292459 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff028011dc0 con 0x7ff0380a30b0 2024-04-26T04:20:47.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.053+0000 7ff03d702640 5 monclient: authenticate success, global_id 4821 2024-04-26T04:20:47.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.053+0000 7ff03d702640 10 monclient: _renew_subs 2024-04-26T04:20:47.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.053+0000 7ff03d702640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:20:47.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.053+0000 7ff03d702640 1 -- 172.21.15.63:0/2976292459 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff038134fa0 con 0x7ff0380a30b0 2024-04-26T04:20:47.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.053+0000 7ff0357fa640 4 set_mon_vals no callback set 2024-04-26T04:20:47.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.053+0000 7ff02f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:20:47.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.053+0000 7ff02f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:20:47.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.053+0000 7ff02f7fe640 10 monclient: dump: 2024-04-26T04:20:47.065 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:20:47.065 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:47.065 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:47.065 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:47.065 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:20:47.065 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:47.065 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:20:47.065 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:20:47.065 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:20:47.065 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:47.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.053+0000 7ff03d702640 10 monclient: _renew_subs 2024-04-26T04:20:47.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.053+0000 7ff03d702640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:20:47.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.053+0000 7ff03d702640 1 -- 172.21.15.63:0/2976292459 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff038133d60 con 0x7ff0380a30b0 2024-04-26T04:20:47.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.053+0000 7ff03d702640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:20:47.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.053+0000 7ff03d702640 1 --2- 172.21.15.63:0/2976292459 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff0381364f0 0x7ff03812ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:47.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.053+0000 7ff03d702640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:47.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.053+0000 7ff03d702640 1 -- 172.21.15.63:0/2976292459 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff0380a3840 con 0x7ff0381364f0 2024-04-26T04:20:47.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.053+0000 7ff02f7fe640 1 -- 172.21.15.63:0/2976292459 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7ff02801c460 con 0x7ff0380a30b0 2024-04-26T04:20:47.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.053+0000 7ff02f7fe640 1 --2- 172.21.15.63:0/2976292459 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7ff018035ec0 0x7ff018038380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:47.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.053+0000 7ff036ffd640 1 --2- 172.21.15.63:0/2976292459 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff0381364f0 0x7ff03812ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:47.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.053+0000 7ff02f7fe640 1 -- 172.21.15.63:0/2976292459 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 4738+0+0 (secure 0 0 0) 0x7ff028048d00 con 0x7ff0380a30b0 2024-04-26T04:20:47.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.053+0000 7ff036ffd640 10 monclient: get_auth_request con 0x7ff0381364f0 auth_method 0 2024-04-26T04:20:47.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.053+0000 7ff036ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:47.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.053+0000 7ff036ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:47.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.053+0000 7ff036ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:20:47.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.053+0000 7ff036ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:20:47.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.053+0000 7ff036ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:20:47.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.053+0000 7ff036ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:20:47.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.053+0000 7ff0377fe640 1 --2- 172.21.15.63:0/2976292459 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7ff018035ec0 0x7ff018038380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:47.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.053+0000 7ff0377fe640 10 monclient: get_auth_request con 0x7ff018035ec0 auth_method 0 2024-04-26T04:20:47.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.053+0000 7ff036ffd640 10 monclient(hunting): handle_auth_done global_id 4759 payload 824 2024-04-26T04:20:47.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.053+0000 7ff036ffd640 1 -- 172.21.15.63:0/2976292459 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff03813d4a0 con 0x7ff0381364f0 2024-04-26T04:20:47.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.057+0000 7ff036ffd640 1 -- 172.21.15.63:0/2976292459 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff0381364f0 msgr2=0x7ff03812ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:20:47.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.057+0000 7ff036ffd640 1 --2- 172.21.15.63:0/2976292459 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff0381364f0 0x7ff03812ab40 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7ff0200072a0 tx=0x7ff020007760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:47.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.057+0000 7ff02f7fe640 1 -- 172.21.15.63:0/2976292459 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff02000fde0 con 0x7ff0381364f0 2024-04-26T04:20:47.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.057+0000 7ff02f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:20:47.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.057+0000 7ff02f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T04:20:47.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.057+0000 7ff02f7fe640 10 monclient: dump: 2024-04-26T04:20:47.070 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:20:47.070 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:47.070 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:47.070 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:47.070 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:20:47.070 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:47.070 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:20:47.071 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:20:47.071 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:20:47.071 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:47.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.057+0000 7ff0377fe640 1 --2- 172.21.15.63:0/2976292459 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7ff018035ec0 0x7ff018038380 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff03806c760 tx=0x7ff0300a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:47.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.057+0000 7ff02f7fe640 1 -- 172.21.15.63:0/2976292459 <== mon.1 v2:172.21.15.186:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff02000fde0 con 0x7ff0381364f0 2024-04-26T04:20:47.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.057+0000 7ff02f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:20:47.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.057+0000 7ff02f7fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:20:47.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.057+0000 7ff02f7fe640 1 -- 172.21.15.63:0/2976292459 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff0381364f0 msgr2=0x7ff03812ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:47.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.057+0000 7ff02f7fe640 1 --2- 172.21.15.63:0/2976292459 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff0381364f0 0x7ff03812ab40 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7ff0200072a0 tx=0x7ff020007760 comp rx=0 tx=0).stop 2024-04-26T04:20:47.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.057+0000 7ff02f7fe640 1 -- 172.21.15.63:0/2976292459 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff0381364f0 msgr2=0x7ff03812ab40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:20:47.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.057+0000 7ff02f7fe640 1 --2- 172.21.15.63:0/2976292459 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff0381364f0 0x7ff03812ab40 secure :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7ff0200072a0 tx=0x7ff020007760 comp rx=0 tx=0).stop 2024-04-26T04:20:47.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.061+0000 7ff03d702640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T04:20:47.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.061+0000 7ff03d702640 1 --2- 172.21.15.63:0/2976292459 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff03812b080 0x7ff03812b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:47.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.061+0000 7ff03d702640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:47.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.061+0000 7ff03d702640 1 -- 172.21.15.63:0/2976292459 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff03812b9d0 con 0x7ff03812b080 2024-04-26T04:20:47.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.065+0000 7ff036ffd640 1 --2- 172.21.15.63:0/2976292459 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff03812b080 0x7ff03812b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:47.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.065+0000 7ff036ffd640 10 monclient: get_auth_request con 0x7ff03812b080 auth_method 0 2024-04-26T04:20:47.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.065+0000 7ff036ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:47.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.065+0000 7ff036ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:47.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.065+0000 7ff036ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:20:47.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.065+0000 7ff036ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:20:47.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.065+0000 7ff036ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:20:47.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.065+0000 7ff036ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:20:47.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.065+0000 7ff036ffd640 10 monclient(hunting): handle_auth_done global_id 4762 payload 824 2024-04-26T04:20:47.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.065+0000 7ff036ffd640 1 -- 172.21.15.63:0/2976292459 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff038062e10 con 0x7ff03812b080 2024-04-26T04:20:47.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.065+0000 7ff036ffd640 1 -- 172.21.15.63:0/2976292459 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff03812b080 msgr2=0x7ff03812b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:20:47.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.065+0000 7ff036ffd640 1 --2- 172.21.15.63:0/2976292459 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff03812b080 0x7ff03812b490 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7ff020017000 tx=0x7ff020004950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:47.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.065+0000 7ff02f7fe640 1 -- 172.21.15.63:0/2976292459 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff02001ee40 con 0x7ff03812b080 2024-04-26T04:20:47.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.065+0000 7ff02f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:20:47.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.065+0000 7ff02f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T04:20:47.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.065+0000 7ff02f7fe640 10 monclient: dump: 2024-04-26T04:20:47.073 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:20:47.073 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:47.073 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:47.073 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:47.073 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:20:47.073 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:47.073 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:20:47.073 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:20:47.073 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:20:47.074 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:47.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.069+0000 7ff02f7fe640 1 -- 172.21.15.63:0/2976292459 <== mon.1 v2:172.21.15.186:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4891 (secure 0 0 0) 0x7ff020006240 con 0x7ff03812b080 2024-04-26T04:20:47.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.069+0000 7ff02f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T04:20:47.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.069+0000 7ff02f7fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:20:47.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.069+0000 7ff02f7fe640 1 -- 172.21.15.63:0/2976292459 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff03812b080 msgr2=0x7ff03812b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:47.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.069+0000 7ff02f7fe640 1 --2- 172.21.15.63:0/2976292459 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff03812b080 0x7ff03812b490 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7ff020017000 tx=0x7ff020004950 comp rx=0 tx=0).stop 2024-04-26T04:20:47.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.069+0000 7ff02f7fe640 1 -- 172.21.15.63:0/2976292459 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff03812b080 msgr2=0x7ff03812b490 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:20:47.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.069+0000 7ff02f7fe640 1 --2- 172.21.15.63:0/2976292459 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff03812b080 0x7ff03812b490 secure :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7ff020017000 tx=0x7ff020004950 comp rx=0 tx=0).stop 2024-04-26T04:20:47.075 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-26T04:20:47.075 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b", 2024-04-26T04:20:47.075 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 1, 2024-04-26T04:20:47.075 INFO:teuthology.orchestra.run.smithi063.stdout: "state": "peon", 2024-04-26T04:20:47.075 INFO:teuthology.orchestra.run.smithi063.stdout: "election_epoch": 14, 2024-04-26T04:20:47.075 INFO:teuthology.orchestra.run.smithi063.stdout: "uptime": 138176, 2024-04-26T04:20:47.075 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum": [ 2024-04-26T04:20:47.075 INFO:teuthology.orchestra.run.smithi063.stdout: 0, 2024-04-26T04:20:47.075 INFO:teuthology.orchestra.run.smithi063.stdout: 1, 2024-04-26T04:20:47.075 INFO:teuthology.orchestra.run.smithi063.stdout: 2 2024-04-26T04:20:47.076 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:20:47.076 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_age": 7, 2024-04-26T04:20:47.076 INFO:teuthology.orchestra.run.smithi063.stdout: "features": { 2024-04-26T04:20:47.076 INFO:teuthology.orchestra.run.smithi063.stdout: "required_con": "2450521708007866372", 2024-04-26T04:20:47.076 INFO:teuthology.orchestra.run.smithi063.stdout: "required_mon": [ 2024-04-26T04:20:47.076 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:20:47.076 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:20:47.076 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:20:47.076 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:20:47.076 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:20:47.077 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:20:47.077 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:20:47.077 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:20:47.077 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:20:47.077 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:20:47.077 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:20:47.077 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:20:47.077 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_con": "4540701547738038271", 2024-04-26T04:20:47.077 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_mon": [ 2024-04-26T04:20:47.077 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:20:47.077 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:20:47.077 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:20:47.077 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:20:47.077 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:20:47.078 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:20:47.078 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:20:47.078 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:20:47.078 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:20:47.078 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:20:47.078 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:20:47.078 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:20:47.078 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:47.078 INFO:teuthology.orchestra.run.smithi063.stdout: "outside_quorum": [], 2024-04-26T04:20:47.079 INFO:teuthology.orchestra.run.smithi063.stdout: "extra_probe_peers": [], 2024-04-26T04:20:47.079 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_provider": [], 2024-04-26T04:20:47.079 INFO:teuthology.orchestra.run.smithi063.stdout: "monmap": { 2024-04-26T04:20:47.079 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 1, 2024-04-26T04:20:47.079 INFO:teuthology.orchestra.run.smithi063.stdout: "fsid": "cd1c6a0c-4e4b-4014-89c0-2e535479ef39", 2024-04-26T04:20:47.079 INFO:teuthology.orchestra.run.smithi063.stdout: "modified": "2024-04-26T04:18:03.702339Z", 2024-04-26T04:20:47.079 INFO:teuthology.orchestra.run.smithi063.stdout: "created": "2024-04-26T04:18:03.702339Z", 2024-04-26T04:20:47.079 INFO:teuthology.orchestra.run.smithi063.stdout: "min_mon_release": 19, 2024-04-26T04:20:47.079 INFO:teuthology.orchestra.run.smithi063.stdout: "min_mon_release_name": "squid", 2024-04-26T04:20:47.079 INFO:teuthology.orchestra.run.smithi063.stdout: "election_strategy": 1, 2024-04-26T04:20:47.079 INFO:teuthology.orchestra.run.smithi063.stdout: "disallowed_leaders: ": "", 2024-04-26T04:20:47.079 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": false, 2024-04-26T04:20:47.080 INFO:teuthology.orchestra.run.smithi063.stdout: "tiebreaker_mon": "", 2024-04-26T04:20:47.080 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_ranks: ": "", 2024-04-26T04:20:47.080 INFO:teuthology.orchestra.run.smithi063.stdout: "features": { 2024-04-26T04:20:47.080 INFO:teuthology.orchestra.run.smithi063.stdout: "persistent": [ 2024-04-26T04:20:47.080 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:20:47.080 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:20:47.080 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:20:47.080 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:20:47.080 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:20:47.080 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:20:47.080 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:20:47.081 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:20:47.081 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:20:47.081 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:20:47.081 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:20:47.081 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:20:47.081 INFO:teuthology.orchestra.run.smithi063.stdout: "optional": [] 2024-04-26T04:20:47.081 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:47.081 INFO:teuthology.orchestra.run.smithi063.stdout: "mons": [ 2024-04-26T04:20:47.081 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:47.081 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 0, 2024-04-26T04:20:47.081 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a", 2024-04-26T04:20:47.082 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:20:47.082 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:20:47.082 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:47.082 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:20:47.082 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:3300", 2024-04-26T04:20:47.082 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:20:47.082 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:47.082 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:47.082 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:20:47.082 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6789", 2024-04-26T04:20:47.082 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:20:47.083 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:20:47.083 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:20:47.083 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:47.083 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6789/0", 2024-04-26T04:20:47.083 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6789/0", 2024-04-26T04:20:47.083 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:20:47.083 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:20:47.083 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:20:47.083 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:47.084 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:47.084 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 1, 2024-04-26T04:20:47.084 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b", 2024-04-26T04:20:47.084 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:20:47.084 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:20:47.084 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:47.084 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:20:47.084 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:3300", 2024-04-26T04:20:47.084 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:20:47.084 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:47.084 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:47.085 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:20:47.085 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:6789", 2024-04-26T04:20:47.085 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:20:47.085 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:20:47.085 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:20:47.085 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:47.085 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:6789/0", 2024-04-26T04:20:47.085 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.186:6789/0", 2024-04-26T04:20:47.085 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:20:47.085 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:20:47.085 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:20:47.085 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:47.085 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:47.086 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 2, 2024-04-26T04:20:47.086 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "c", 2024-04-26T04:20:47.086 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:20:47.086 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:20:47.086 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:47.086 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:20:47.086 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:3301", 2024-04-26T04:20:47.086 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:20:47.086 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:47.086 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:47.086 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:20:47.086 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6790", 2024-04-26T04:20:47.087 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:20:47.087 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:20:47.087 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:20:47.087 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:47.087 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6790/0", 2024-04-26T04:20:47.087 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6790/0", 2024-04-26T04:20:47.087 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:20:47.087 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:20:47.087 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:20:47.087 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:20:47.087 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:20:47.087 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:47.087 INFO:teuthology.orchestra.run.smithi063.stdout: "feature_map": { 2024-04-26T04:20:47.087 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": [ 2024-04-26T04:20:47.088 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:47.088 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:20:47.088 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:20:47.088 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 1 2024-04-26T04:20:47.088 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:20:47.088 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:20:47.088 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": [ 2024-04-26T04:20:47.088 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:47.088 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:20:47.088 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:20:47.088 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 2 2024-04-26T04:20:47.088 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:20:47.088 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:20:47.088 INFO:teuthology.orchestra.run.smithi063.stdout: "client": [ 2024-04-26T04:20:47.089 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:47.089 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:20:47.089 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:20:47.089 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 6 2024-04-26T04:20:47.089 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:20:47.089 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:20:47.089 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": [ 2024-04-26T04:20:47.089 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:20:47.089 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:20:47.090 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:20:47.090 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 1 2024-04-26T04:20:47.090 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:20:47.090 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:20:47.090 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:20:47.090 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": false 2024-04-26T04:20:47.090 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-26T04:20:47.091 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.069+0000 7ff03d702640 1 -- 172.21.15.63:0/2976292459 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7ff018035ec0 msgr2=0x7ff018038380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:47.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.069+0000 7ff03d702640 1 --2- 172.21.15.63:0/2976292459 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7ff018035ec0 0x7ff018038380 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff03806c760 tx=0x7ff0300a7070 comp rx=0 tx=0).stop 2024-04-26T04:20:47.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.069+0000 7ff03d702640 10 monclient: shutdown 2024-04-26T04:20:47.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.069+0000 7ff03d702640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:20:47.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.069+0000 7ff03d702640 1 -- 172.21.15.63:0/2976292459 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff0380a30b0 msgr2=0x7ff038132d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:47.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.069+0000 7ff03d702640 1 --2- 172.21.15.63:0/2976292459 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff0380a30b0 0x7ff038132d80 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff02800c3e0 tx=0x7ff02800c8a0 comp rx=0 tx=0).stop 2024-04-26T04:20:47.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.069+0000 7ff0377fe640 1 -- 172.21.15.63:0/2976292459 reap_dead start 2024-04-26T04:20:47.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.069+0000 7ff0377fe640 1 -- 172.21.15.63:0/2976292459 reap_dead start 2024-04-26T04:20:47.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.069+0000 7ff03d702640 1 -- 172.21.15.63:0/2976292459 shutdown_connections 2024-04-26T04:20:47.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.069+0000 7ff03d702640 1 -- 172.21.15.63:0/2976292459 >> 172.21.15.63:0/2976292459 conn(0x7ff0380a5f40 msgr2=0x7ff0380a1a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:20:47.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.069+0000 7ff03d702640 1 -- 172.21.15.63:0/2976292459 shutdown_connections 2024-04-26T04:20:47.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.069+0000 7ff03d702640 1 -- 172.21.15.63:0/2976292459 wait complete. 2024-04-26T04:20:47.093 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T04:20:47.093 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-04-26T04:20:47.093 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-04-26T04:20:47.093 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-04-26T04:20:47.093 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-04-26T04:20:47.093 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-04-26T04:20:47.093 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T04:20:47.103 INFO:tasks.ceph.mon.c:Stopped 2024-04-26T04:20:47.103 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-04-26T04:20:47.103 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T04:20:47.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.257+0000 7f83dd6c6640 10 monclient: get_monmap_and_config 2024-04-26T04:20:47.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.257+0000 7f83dd6c6640 10 monclient: build_initial_monmap 2024-04-26T04:20:47.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.257+0000 7f83dd6c6640 10 monclient: monmap: 2024-04-26T04:20:47.259 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:20:47.259 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:47.259 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:20:47.258608+0000 2024-04-26T04:20:47.259 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:20:47.258608+0000 2024-04-26T04:20:47.259 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:20:47.259 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:47.259 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:20:47.259 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:20:47.259 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:20:47.260 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:20:47.260 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:20:47.260 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:47.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.257+0000 7f83dd6c6640 1 Processor -- start 2024-04-26T04:20:47.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.257+0000 7f83dd6c6640 1 -- start start 2024-04-26T04:20:47.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.257+0000 7f83dd6c6640 10 monclient: init 2024-04-26T04:20:47.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.257+0000 7f83dd6c6640 10 monclient: _reopen_session rank -1 2024-04-26T04:20:47.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.257+0000 7f83dd6c6640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2024-04-26T04:20:47.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.257+0000 7f83dd6c6640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f83d80a2cd0 0x7f83d80a30b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:47.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.257+0000 7f83dd6c6640 10 monclient(hunting): picked mon.noname-b con 0x7f83d80a2cd0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:20:47.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.257+0000 7f83dd6c6640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f83d80a47d0 0x7f83d8136ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:47.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.257+0000 7f83dd6c6640 10 monclient(hunting): picked mon.noname-c con 0x7f83d80a47d0 addr v2:172.21.15.186:3300/0 2024-04-26T04:20:47.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.257+0000 7f83dd6c6640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f83d80a4bd0 0x7f83d8137220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:47.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.257+0000 7f83dd6c6640 10 monclient(hunting): picked mon.noname-a con 0x7f83d80a4bd0 addr v2:172.21.15.63:3300/0 2024-04-26T04:20:47.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.257+0000 7f83dd6c6640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:47.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.257+0000 7f83dd6c6640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7f83d80a35f0 con 0x7f83d80a4bd0 2024-04-26T04:20:47.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.257+0000 7f83dd6c6640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:47.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.257+0000 7f83dd6c6640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f83d80a0fb0 con 0x7f83d80a47d0 2024-04-26T04:20:47.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.257+0000 7f83dd6c6640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:47.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.257+0000 7f83dd6c6640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f83d80a5380 con 0x7f83d80a2cd0 2024-04-26T04:20:47.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.257+0000 7f83dd6c6640 10 monclient(hunting): _renew_subs 2024-04-26T04:20:47.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.257+0000 7f83dd6c6640 10 monclient(hunting): authenticate will time out at 1078.966350s 2024-04-26T04:20:47.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.257+0000 7f83d6ffd640 1 -- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f83d80a2cd0 msgr2=0x7f83d80a30b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.63:3301/0 2024-04-26T04:20:47.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.257+0000 7f83d6ffd640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f83d80a2cd0 0x7f83d80a30b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:20:47.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.257+0000 7f83d77fe640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f83d80a4bd0 0x7f83d8137220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:47.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.257+0000 7f83d77fe640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f83d80a4bd0 0x7f83d8137220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:36584/0 (socket says 172.21.15.63:36584) 2024-04-26T04:20:47.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.257+0000 7f83d77fe640 1 -- 172.21.15.63:0/2951569571 learned_addr learned my addr 172.21.15.63:0/2951569571 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:20:47.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.257+0000 7f83d77fe640 10 monclient(hunting): get_auth_request con 0x7f83d80a4bd0 auth_method 0 2024-04-26T04:20:47.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.257+0000 7f83d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:47.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.257+0000 7f83d77fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:47.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.257+0000 7f83d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:20:47.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.257+0000 7f83d77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:20:47.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.257+0000 7f83d77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:20:47.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.257+0000 7f83d77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:20:47.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.257+0000 7f83d67fc640 1 --2- 172.21.15.63:0/2951569571 >> v2:172.21.15.186:3300/0 conn(0x7f83d80a47d0 0x7f83d8136ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:47.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.257+0000 7f83d67fc640 10 monclient(hunting): get_auth_request con 0x7f83d80a47d0 auth_method 0 2024-04-26T04:20:47.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.257+0000 7f83d67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:47.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.257+0000 7f83d67fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:47.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.257+0000 7f83d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:20:47.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.257+0000 7f83d77fe640 10 monclient(hunting): handle_auth_done global_id 4824 payload 274 2024-04-26T04:20:47.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.257+0000 7f83d77fe640 1 -- 172.21.15.63:0/2951569571 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f83d80a2cd0 msgr2=0x7f83d80a30b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T04:20:47.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.257+0000 7f83d77fe640 1 --2- 172.21.15.63:0/2951569571 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f83d80a2cd0 0x7f83d80a30b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:47.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.257+0000 7f83d77fe640 1 -- 172.21.15.63:0/2951569571 >> v2:172.21.15.186:3300/0 conn(0x7f83d80a47d0 msgr2=0x7f83d8136ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:20:47.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.257+0000 7f83d77fe640 1 --2- 172.21.15.63:0/2951569571 >> v2:172.21.15.186:3300/0 conn(0x7f83d80a47d0 0x7f83d8136ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:47.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.257+0000 7f83d77fe640 10 monclient: _finish_hunting 0 2024-04-26T04:20:47.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.257+0000 7f83d77fe640 1 monclient: found mon.noname-a 2024-04-26T04:20:47.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.257+0000 7f83d77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:20:47.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.257+0000 7f83d77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:20:47.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.257+0000 7f83d77fe640 1 -- 172.21.15.63:0/2951569571 --> v2:172.21.15.63:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83d80a1290 con 0x7f83d80a4bd0 2024-04-26T04:20:47.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.261+0000 7f83d77fe640 1 -- 172.21.15.63:0/2951569571 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f83d80a4bd0 msgr2=0x7f83d8137220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:20:47.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.261+0000 7f83d77fe640 1 --2- 172.21.15.63:0/2951569571 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f83d80a4bd0 0x7f83d8137220 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f83d806d7c0 tx=0x7f83d001c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=648a82ff71144bbb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:47.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.261+0000 7f83d5ffb640 1 -- 172.21.15.63:0/2951569571 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d00a55e0 con 0x7f83d80a4bd0 2024-04-26T04:20:47.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.261+0000 7f83d5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:20:47.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.261+0000 7f83d5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:20:47.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.261+0000 7f83d5ffb640 10 monclient: dump: 2024-04-26T04:20:47.273 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:20:47.273 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:47.273 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:47.273 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:47.273 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:20:47.273 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:47.273 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:20:47.273 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:20:47.274 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:20:47.274 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:47.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.261+0000 7f83d5ffb640 10 monclient: _finish_auth 0 2024-04-26T04:20:47.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.261+0000 7f83d5ffb640 10 monclient: _check_auth_tickets 2024-04-26T04:20:47.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.261+0000 7f83d5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:20:47.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.261+0000 7f83d5ffb640 1 -- 172.21.15.63:0/2951569571 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83d00a5780 con 0x7f83d80a4bd0 2024-04-26T04:20:47.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.261+0000 7f83d5ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T04:20:47.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.261+0000 7f83d5ffb640 1 -- 172.21.15.63:0/2951569571 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d00a5b50 con 0x7f83d80a4bd0 2024-04-26T04:20:47.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.261+0000 7f83dd6c6640 5 monclient: authenticate success, global_id 4824 2024-04-26T04:20:47.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.261+0000 7f83dd6c6640 10 monclient: get_monmap_and_config success 2024-04-26T04:20:47.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.261+0000 7f83dd6c6640 4 set_mon_vals no callback set 2024-04-26T04:20:47.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.261+0000 7f83d5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:20:47.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.261+0000 7f83d5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:20:47.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.261+0000 7f83d5ffb640 10 monclient: dump: 2024-04-26T04:20:47.274 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:20:47.275 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:47.275 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:47.275 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:47.275 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:20:47.275 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:47.275 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:20:47.275 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:20:47.275 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:20:47.275 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:47.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.261+0000 7f83dd6c6640 10 monclient: shutdown 2024-04-26T04:20:47.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.261+0000 7f83dd6c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:20:47.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.261+0000 7f83dd6c6640 1 -- 172.21.15.63:0/2951569571 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f83d80a4bd0 msgr2=0x7f83d8137220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:47.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.261+0000 7f83dd6c6640 1 --2- 172.21.15.63:0/2951569571 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f83d80a4bd0 0x7f83d8137220 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f83d806d7c0 tx=0x7f83d001c960 comp rx=0 tx=0).stop 2024-04-26T04:20:47.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.261+0000 7f83dd6c6640 1 -- 172.21.15.63:0/2951569571 shutdown_connections 2024-04-26T04:20:47.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.261+0000 7f83dd6c6640 1 --2- 172.21.15.63:0/2951569571 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f83d80a4bd0 0x7f83d8137220 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:47.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.261+0000 7f83dd6c6640 1 --2- 172.21.15.63:0/2951569571 >> v2:172.21.15.186:3300/0 conn(0x7f83d80a47d0 0x7f83d8136ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:47.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.261+0000 7f83dd6c6640 1 --2- 172.21.15.63:0/2951569571 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f83d80a2cd0 0x7f83d80a30b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:47.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.261+0000 7f83dd6c6640 1 -- 172.21.15.63:0/2951569571 >> 172.21.15.63:0/2951569571 conn(0x7f83d80a5f40 msgr2=0x7f83d80a6320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:20:47.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.265+0000 7f83dd6c6640 1 -- 172.21.15.63:0/2951569571 shutdown_connections 2024-04-26T04:20:47.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.265+0000 7f83dd6c6640 1 -- 172.21.15.63:0/2951569571 wait complete. 2024-04-26T04:20:47.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.265+0000 7f83dd6c6640 10 monclient: build_initial_monmap 2024-04-26T04:20:47.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.265+0000 7f83dd6c6640 10 monclient: monmap: 2024-04-26T04:20:47.276 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:20:47.276 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:20:47.276 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:20:47.276 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:20:47.276 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:20:47.276 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:47.276 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:20:47.277 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:20:47.277 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:20:47.277 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:47.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.265+0000 7f83dd6c6640 1 Processor -- start 2024-04-26T04:20:47.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.265+0000 7f83dd6c6640 1 -- start start 2024-04-26T04:20:47.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.265+0000 7f83dd6c6640 10 monclient: init 2024-04-26T04:20:47.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.265+0000 7f83dd6c6640 10 monclient: _reopen_session rank -1 2024-04-26T04:20:47.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.265+0000 7f83dd6c6640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T04:20:47.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.265+0000 7f83dd6c6640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f83d80a2cd0 0x7f83d811ff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:47.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.265+0000 7f83dd6c6640 10 monclient(hunting): picked mon.noname-c con 0x7f83d80a2cd0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:20:47.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.265+0000 7f83dd6c6640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83d80a47d0 0x7f83d813c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:47.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.265+0000 7f83dd6c6640 10 monclient(hunting): picked mon.noname-b con 0x7f83d80a47d0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:20:47.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.265+0000 7f83dd6c6640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f83d80a4bd0 0x7f83d8135d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:47.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.265+0000 7f83dd6c6640 10 monclient(hunting): picked mon.noname-a con 0x7f83d80a4bd0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:20:47.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.265+0000 7f83dd6c6640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:47.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.265+0000 7f83dd6c6640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f83d8140a40 con 0x7f83d80a4bd0 2024-04-26T04:20:47.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.265+0000 7f83dd6c6640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:47.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.265+0000 7f83dd6c6640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f83d8137b70 con 0x7f83d80a47d0 2024-04-26T04:20:47.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.265+0000 7f83dd6c6640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:47.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.265+0000 7f83dd6c6640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f83d8140bc0 con 0x7f83d80a2cd0 2024-04-26T04:20:47.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.265+0000 7f83dd6c6640 10 monclient(hunting): _renew_subs 2024-04-26T04:20:47.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.265+0000 7f83dd6c6640 10 monclient(hunting): authenticate will time out at 1078.973436s 2024-04-26T04:20:47.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.265+0000 7f83d6ffd640 1 -- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f83d80a2cd0 msgr2=0x7f83d811ff50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:3301/0 2024-04-26T04:20:47.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.265+0000 7f83d6ffd640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f83d80a2cd0 0x7f83d811ff50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:20:47.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.265+0000 7f83d77fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f83d80a4bd0 0x7f83d8135d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:47.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.265+0000 7f83d77fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f83d80a4bd0 0x7f83d8135d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:36590/0 (socket says 172.21.15.63:36590) 2024-04-26T04:20:47.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.265+0000 7f83d77fe640 1 -- 172.21.15.63:0/879813391 learned_addr learned my addr 172.21.15.63:0/879813391 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:20:47.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.265+0000 7f83d77fe640 10 monclient(hunting): get_auth_request con 0x7f83d80a4bd0 auth_method 0 2024-04-26T04:20:47.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.265+0000 7f83d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:47.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.265+0000 7f83d77fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:47.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.265+0000 7f83d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:20:47.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.265+0000 7f83d77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:20:47.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.265+0000 7f83d77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:20:47.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.265+0000 7f83d67fc640 1 --2- 172.21.15.63:0/879813391 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83d80a47d0 0x7f83d813c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:47.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.265+0000 7f83d77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:20:47.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.265+0000 7f83d67fc640 10 monclient(hunting): get_auth_request con 0x7f83d80a47d0 auth_method 0 2024-04-26T04:20:47.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.265+0000 7f83d67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:47.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.265+0000 7f83d67fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:47.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.265+0000 7f83d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:20:47.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.265+0000 7f83d77fe640 10 monclient(hunting): handle_auth_done global_id 4827 payload 824 2024-04-26T04:20:47.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.265+0000 7f83d77fe640 1 -- 172.21.15.63:0/879813391 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f83d80a2cd0 msgr2=0x7f83d811ff50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T04:20:47.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.265+0000 7f83d77fe640 1 --2- 172.21.15.63:0/879813391 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f83d80a2cd0 0x7f83d811ff50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:47.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.265+0000 7f83d77fe640 1 -- 172.21.15.63:0/879813391 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83d80a47d0 msgr2=0x7f83d813c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:47.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.265+0000 7f83d77fe640 1 --2- 172.21.15.63:0/879813391 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83d80a47d0 0x7f83d813c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:47.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.265+0000 7f83d77fe640 10 monclient: _finish_hunting 0 2024-04-26T04:20:47.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.265+0000 7f83d77fe640 1 monclient: found mon.noname-a 2024-04-26T04:20:47.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.265+0000 7f83d77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:20:47.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.265+0000 7f83d77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:20:47.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.265+0000 7f83d77fe640 1 -- 172.21.15.63:0/879813391 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83d81216e0 con 0x7f83d80a4bd0 2024-04-26T04:20:47.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.269+0000 7f83d77fe640 1 -- 172.21.15.63:0/879813391 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f83d80a4bd0 msgr2=0x7f83d8135d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:20:47.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.269+0000 7f83d77fe640 1 --2- 172.21.15.63:0/879813391 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f83d80a4bd0 0x7f83d8135d40 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f83d001b010 tx=0x7f83d0019700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:47.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.269+0000 7f83cf7fe640 1 -- 172.21.15.63:0/879813391 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d00a55e0 con 0x7f83d80a4bd0 2024-04-26T04:20:47.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.269+0000 7f83cf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:20:47.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.269+0000 7f83cf7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:20:47.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.269+0000 7f83cf7fe640 10 monclient: dump: 2024-04-26T04:20:47.280 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:20:47.281 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:47.281 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:47.281 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:47.281 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:20:47.281 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:47.281 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:20:47.281 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:20:47.281 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:20:47.281 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:47.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.269+0000 7f83cf7fe640 10 monclient: _finish_auth 0 2024-04-26T04:20:47.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.269+0000 7f83cf7fe640 10 monclient: _check_auth_tickets 2024-04-26T04:20:47.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.269+0000 7f83cf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:20:47.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.269+0000 7f83cf7fe640 1 -- 172.21.15.63:0/879813391 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83d00bc070 con 0x7f83d80a4bd0 2024-04-26T04:20:47.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.269+0000 7f83cf7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T04:20:47.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.269+0000 7f83cf7fe640 1 -- 172.21.15.63:0/879813391 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d00b6d00 con 0x7f83d80a4bd0 2024-04-26T04:20:47.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.269+0000 7f83cf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:20:47.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.269+0000 7f83cf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:20:47.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.269+0000 7f83cf7fe640 10 monclient: dump: 2024-04-26T04:20:47.282 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:20:47.282 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:47.282 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:47.282 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:47.282 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:20:47.282 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:47.282 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:20:47.282 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:20:47.282 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:20:47.282 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:47.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.269+0000 7f83dd6c6640 5 monclient: authenticate success, global_id 4827 2024-04-26T04:20:47.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.269+0000 7f83d57fa640 4 set_mon_vals no callback set 2024-04-26T04:20:47.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.269+0000 7f83dd6c6640 10 monclient: _renew_subs 2024-04-26T04:20:47.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.269+0000 7f83dd6c6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:20:47.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.269+0000 7f83dd6c6640 1 -- 172.21.15.63:0/879813391 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83d813d920 con 0x7f83d80a4bd0 2024-04-26T04:20:47.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.269+0000 7f83dd6c6640 10 monclient: _renew_subs 2024-04-26T04:20:47.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.269+0000 7f83dd6c6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:20:47.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.269+0000 7f83dd6c6640 1 -- 172.21.15.63:0/879813391 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83d8120e40 con 0x7f83d80a4bd0 2024-04-26T04:20:47.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.269+0000 7f83dd6c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:20:47.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.269+0000 7f83dd6c6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:20:47.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.269+0000 7f83dd6c6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:20:47.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.269+0000 7f83dd6c6640 1 -- 172.21.15.63:0/879813391 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83d8062e10 con 0x7f83d80a4bd0 2024-04-26T04:20:47.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.269+0000 7f83cf7fe640 1 -- 172.21.15.63:0/879813391 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f83d00c1460 con 0x7f83d80a4bd0 2024-04-26T04:20:47.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.269+0000 7f83cf7fe640 1 --2- 172.21.15.63:0/879813391 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f83b8035ec0 0x7f83b8038380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:47.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.269+0000 7f83cf7fe640 1 -- 172.21.15.63:0/879813391 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 4738+0+0 (secure 0 0 0) 0x7f83d00e3d40 con 0x7f83d80a4bd0 2024-04-26T04:20:47.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.269+0000 7f83d6ffd640 1 --2- 172.21.15.63:0/879813391 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f83b8035ec0 0x7f83b8038380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:47.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.269+0000 7f83d6ffd640 10 monclient: get_auth_request con 0x7f83b8035ec0 auth_method 0 2024-04-26T04:20:47.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.273+0000 7f83d6ffd640 1 --2- 172.21.15.63:0/879813391 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f83b8035ec0 0x7f83b8038380 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f83c0006fd0 tx=0x7f83c0006d40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:47.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.273+0000 7f83cf7fe640 1 -- 172.21.15.63:0/879813391 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f83d00165e0 con 0x7f83d80a4bd0 2024-04-26T04:20:47.285 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.273+0000 7f83cf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:20:47.285 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.273+0000 7f83cf7fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:20:47.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.489+0000 7f83dd6c6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T04:20:47.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.489+0000 7f83dd6c6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T04:20:47.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.489+0000 7f83dd6c6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:20:47.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.489+0000 7f83dd6c6640 1 -- 172.21.15.63:0/879813391 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f83d80683d0 con 0x7f83d80a4bd0 2024-04-26T04:20:47.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.489+0000 7f83cf7fe640 1 -- 172.21.15.63:0/879813391 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f83d00ace30 con 0x7f83d80a4bd0 2024-04-26T04:20:47.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.489+0000 7f83cf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T04:20:47.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.489+0000 7f83cf7fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:20:47.492 INFO:teuthology.orchestra.run.smithi063.stdout:{"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cd1c6a0c-4e4b-4014-89c0-2e535479ef39","modified":"2024-04-26T04:18:03.702339Z","created":"2024-04-26T04:18:03.702339Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3300","nonce":0},{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:3300","nonce":0},{"type":"v1","addr":"172.21.15.186:6789","nonce":0}]},"addr":"172.21.15.186:6789/0","public_addr":"172.21.15.186:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3301","nonce":0},{"type":"v1","addr":"172.21.15.63:6790","nonce":0}]},"addr":"172.21.15.63:6790/0","public_addr":"172.21.15.63:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T04:20:47.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.493+0000 7f83dd6c6640 1 -- 172.21.15.63:0/879813391 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f83b8035ec0 msgr2=0x7f83b8038380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:47.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.493+0000 7f83dd6c6640 1 --2- 172.21.15.63:0/879813391 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f83b8035ec0 0x7f83b8038380 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f83c0006fd0 tx=0x7f83c0006d40 comp rx=0 tx=0).stop 2024-04-26T04:20:47.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.493+0000 7f83dd6c6640 10 monclient: shutdown 2024-04-26T04:20:47.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.493+0000 7f83dd6c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:20:47.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.493+0000 7f83dd6c6640 1 -- 172.21.15.63:0/879813391 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f83d80a4bd0 msgr2=0x7f83d8135d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:47.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.493+0000 7f83dd6c6640 1 --2- 172.21.15.63:0/879813391 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f83d80a4bd0 0x7f83d8135d40 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f83d001b010 tx=0x7f83d0019700 comp rx=0 tx=0).stop 2024-04-26T04:20:47.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.493+0000 7f83dd6c6640 1 -- 172.21.15.63:0/879813391 shutdown_connections 2024-04-26T04:20:47.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.493+0000 7f83dd6c6640 1 --2- 172.21.15.63:0/879813391 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f83b8035ec0 0x7f83b8038380 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:47.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.493+0000 7f83dd6c6640 1 --2- 172.21.15.63:0/879813391 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f83d80a4bd0 0x7f83d8135d40 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:47.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.493+0000 7f83dd6c6640 1 --2- 172.21.15.63:0/879813391 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83d80a47d0 0x7f83d813c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:47.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.493+0000 7f83dd6c6640 1 --2- 172.21.15.63:0/879813391 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f83d80a2cd0 0x7f83d811ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:47.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.493+0000 7f83dd6c6640 1 -- 172.21.15.63:0/879813391 >> 172.21.15.63:0/879813391 conn(0x7f83d80a5f40 msgr2=0x7f83d8072c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:20:47.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.493+0000 7f83dd6c6640 1 -- 172.21.15.63:0/879813391 shutdown_connections 2024-04-26T04:20:47.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:47.493+0000 7f83dd6c6640 1 -- 172.21.15.63:0/879813391 wait complete. 2024-04-26T04:20:50.512 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T04:20:50.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.641+0000 7f32d4dce640 10 monclient: get_monmap_and_config 2024-04-26T04:20:50.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.641+0000 7f32d4dce640 10 monclient: build_initial_monmap 2024-04-26T04:20:50.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.641+0000 7f32d4dce640 10 monclient: monmap: 2024-04-26T04:20:50.644 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:20:50.644 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:50.644 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:20:50.644123+0000 2024-04-26T04:20:50.644 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:20:50.644123+0000 2024-04-26T04:20:50.644 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:20:50.644 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:50.645 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:20:50.645 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:20:50.645 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:20:50.645 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:20:50.645 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:20:50.645 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:50.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.641+0000 7f32d4dce640 1 Processor -- start 2024-04-26T04:20:50.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.641+0000 7f32d4dce640 1 -- start start 2024-04-26T04:20:50.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.641+0000 7f32d4dce640 10 monclient: init 2024-04-26T04:20:50.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.645+0000 7f32d4dce640 10 monclient: _reopen_session rank -1 2024-04-26T04:20:50.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.645+0000 7f32d4dce640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2024-04-26T04:20:50.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.645+0000 7f32d4dce640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f32d009fcb0 0x7f32d00a1740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:50.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.645+0000 7f32d4dce640 10 monclient(hunting): picked mon.noname-c con 0x7f32d009fcb0 addr v2:172.21.15.186:3300/0 2024-04-26T04:20:50.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.645+0000 7f32d4dce640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f32d00a0090 0x7f32d00a0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:50.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.645+0000 7f32d4dce640 10 monclient(hunting): picked mon.noname-b con 0x7f32d00a0090 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:20:50.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.645+0000 7f32d4dce640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f32d0133a80 addr v1:172.21.15.186:6789/0 2024-04-26T04:20:50.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.645+0000 7f32d4dce640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32d009f010 con 0x7f32d0133a80 2024-04-26T04:20:50.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.645+0000 7f32d4dce640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:50.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.645+0000 7f32d4dce640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f32d00a1040 con 0x7f32d009fcb0 2024-04-26T04:20:50.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.645+0000 7f32d4dce640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:50.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.645+0000 7f32d4dce640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f32d006b970 con 0x7f32d00a0090 2024-04-26T04:20:50.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.645+0000 7f32d4dce640 10 monclient(hunting): _renew_subs 2024-04-26T04:20:50.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.645+0000 7f32d4dce640 10 monclient(hunting): authenticate will time out at 1082.352613s 2024-04-26T04:20:50.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.645+0000 7f32c7fff640 1 -- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f32d00a0090 msgr2=0x7f32d00a0470 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.63:3301/0 2024-04-26T04:20:50.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.645+0000 7f32c7fff640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f32d00a0090 0x7f32d00a0470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:20:50.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.645+0000 7f32ce59c640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f32d009fcb0 0x7f32d00a1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:50.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.645+0000 7f32ce59c640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f32d009fcb0 0x7f32d00a1740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.63:41814/0 (socket says 172.21.15.63:41814) 2024-04-26T04:20:50.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.645+0000 7f32ce59c640 1 -- 172.21.15.63:0/2600028074 learned_addr learned my addr 172.21.15.63:0/2600028074 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:20:50.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.645+0000 7f32ce59c640 10 monclient(hunting): get_auth_request con 0x7f32d009fcb0 auth_method 0 2024-04-26T04:20:50.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.645+0000 7f32ce59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:50.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.645+0000 7f32ce59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:50.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.645+0000 7f32ce59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:20:50.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.645+0000 7f32ce59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:20:50.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.645+0000 7f32ce59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:20:50.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.645+0000 7f32ce59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:20:50.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.645+0000 7f32ce59c640 10 monclient(hunting): handle_auth_done global_id 4771 payload 274 2024-04-26T04:20:50.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.645+0000 7f32ce59c640 1 -- 172.21.15.63:0/2600028074 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f32d00a0090 msgr2=0x7f32d00a0470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T04:20:50.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.645+0000 7f32ce59c640 1 --2- 172.21.15.63:0/2600028074 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f32d00a0090 0x7f32d00a0470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:50.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.645+0000 7f32ce59c640 1 -- 172.21.15.63:0/2600028074 >> v1:172.21.15.186:6789/0 conn(0x7f32d0133a80 legacy=0x7f32d0133ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:20:50.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.645+0000 7f32ce59c640 10 monclient: _finish_hunting 0 2024-04-26T04:20:50.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.645+0000 7f32ce59c640 1 monclient: found mon.noname-c 2024-04-26T04:20:50.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.645+0000 7f32ce59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:20:50.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.645+0000 7f32ce59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.186:3300/0 2024-04-26T04:20:50.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.645+0000 7f32ce59c640 1 -- 172.21.15.63:0/2600028074 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32d00a0d60 con 0x7f32d009fcb0 2024-04-26T04:20:50.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.649+0000 7f32ce59c640 1 -- 172.21.15.63:0/2600028074 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f32d009fcb0 msgr2=0x7f32d00a1740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:20:50.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.649+0000 7f32ce59c640 1 --2- 172.21.15.63:0/2600028074 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f32d009fcb0 0x7f32d00a1740 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f32c0009870 tx=0x7f32c001c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f1d5d2f952c872d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:50.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.649+0000 7f32cdd9b640 1 -- 172.21.15.63:0/2600028074 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32c00b0070 con 0x7f32d009fcb0 2024-04-26T04:20:50.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.649+0000 7f32cdd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:20:50.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.649+0000 7f32cdd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:20:50.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.649+0000 7f32cdd9b640 10 monclient: dump: 2024-04-26T04:20:50.652 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:20:50.652 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:50.653 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:50.653 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:50.653 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:20:50.653 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:50.653 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:20:50.653 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:20:50.653 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:20:50.653 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:50.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.649+0000 7f32cdd9b640 10 monclient: _finish_auth 0 2024-04-26T04:20:50.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.649+0000 7f32cdd9b640 10 monclient: _check_auth_tickets 2024-04-26T04:20:50.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.649+0000 7f32cdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:20:50.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.649+0000 7f32cdd9b640 1 -- 172.21.15.63:0/2600028074 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32c00ab600 con 0x7f32d009fcb0 2024-04-26T04:20:50.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.649+0000 7f32cdd9b640 10 monclient: handle_config config(0 keys) 2024-04-26T04:20:50.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.649+0000 7f32cdd9b640 1 -- 172.21.15.63:0/2600028074 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32c00ab9d0 con 0x7f32d009fcb0 2024-04-26T04:20:50.654 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.649+0000 7f32cdd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:20:50.654 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.649+0000 7f32cdd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T04:20:50.654 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.649+0000 7f32cdd9b640 10 monclient: dump: 2024-04-26T04:20:50.654 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:20:50.654 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:50.654 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:50.654 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:50.654 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:20:50.654 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:50.654 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:20:50.654 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:20:50.654 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:20:50.654 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:50.654 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.649+0000 7f32d4dce640 5 monclient: authenticate success, global_id 4771 2024-04-26T04:20:50.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.649+0000 7f32d4dce640 10 monclient: get_monmap_and_config success 2024-04-26T04:20:50.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.649+0000 7f32d4dce640 4 set_mon_vals no callback set 2024-04-26T04:20:50.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.649+0000 7f32d4dce640 10 monclient: shutdown 2024-04-26T04:20:50.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.649+0000 7f32d4dce640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:20:50.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.649+0000 7f32d4dce640 1 -- 172.21.15.63:0/2600028074 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f32d009fcb0 msgr2=0x7f32d00a1740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:50.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.649+0000 7f32d4dce640 1 --2- 172.21.15.63:0/2600028074 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f32d009fcb0 0x7f32d00a1740 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f32c0009870 tx=0x7f32c001c490 comp rx=0 tx=0).stop 2024-04-26T04:20:50.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.649+0000 7f32d4dce640 1 -- 172.21.15.63:0/2600028074 shutdown_connections 2024-04-26T04:20:50.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.649+0000 7f32d4dce640 1 --2- 172.21.15.63:0/2600028074 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f32d00a0090 0x7f32d00a0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:50.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.649+0000 7f32d4dce640 1 --2- 172.21.15.63:0/2600028074 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f32d009fcb0 0x7f32d00a1740 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:50.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.649+0000 7f32d4dce640 1 -- 172.21.15.63:0/2600028074 >> 172.21.15.63:0/2600028074 conn(0x7f32d00a1db0 msgr2=0x7f32d00a2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:20:50.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.649+0000 7f32d4dce640 1 -- 172.21.15.63:0/2600028074 shutdown_connections 2024-04-26T04:20:50.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.649+0000 7f32d4dce640 1 -- 172.21.15.63:0/2600028074 wait complete. 2024-04-26T04:20:50.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.653+0000 7f32d4dce640 10 monclient: build_initial_monmap 2024-04-26T04:20:50.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.653+0000 7f32d4dce640 10 monclient: monmap: 2024-04-26T04:20:50.660 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:20:50.660 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:20:50.660 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:20:50.660 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:20:50.660 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:20:50.660 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:50.661 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:20:50.661 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:20:50.661 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:20:50.661 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:50.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.653+0000 7f32d4dce640 1 Processor -- start 2024-04-26T04:20:50.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.653+0000 7f32d4dce640 1 -- start start 2024-04-26T04:20:50.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.653+0000 7f32d4dce640 10 monclient: init 2024-04-26T04:20:50.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.653+0000 7f32d4dce640 10 monclient: _reopen_session rank -1 2024-04-26T04:20:50.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.653+0000 7f32d4dce640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T04:20:50.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.653+0000 7f32d4dce640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f32d009fcb0 0x7f32d01cef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:50.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.653+0000 7f32d4dce640 10 monclient(hunting): picked mon.noname-b con 0x7f32d009fcb0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:20:50.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.653+0000 7f32d4dce640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f32d00a0090 0x7f32d0132960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:50.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.653+0000 7f32d4dce640 10 monclient(hunting): picked mon.noname-a con 0x7f32d00a0090 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:20:50.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.653+0000 7f32d4dce640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f32d0133a80 0x7f32d013cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:50.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.653+0000 7f32d4dce640 10 monclient(hunting): picked mon.noname-c con 0x7f32d0133a80 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:20:50.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.653+0000 7f32d4dce640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:50.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.653+0000 7f32d4dce640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f32d0141bf0 con 0x7f32d00a0090 2024-04-26T04:20:50.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.653+0000 7f32d4dce640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:50.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.653+0000 7f32d4dce640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f32d0141a70 con 0x7f32d009fcb0 2024-04-26T04:20:50.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.653+0000 7f32d4dce640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:50.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.653+0000 7f32d4dce640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f32d0141d70 con 0x7f32d0133a80 2024-04-26T04:20:50.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.653+0000 7f32d4dce640 10 monclient(hunting): _renew_subs 2024-04-26T04:20:50.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.653+0000 7f32d4dce640 10 monclient(hunting): authenticate will time out at 1082.360982s 2024-04-26T04:20:50.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.653+0000 7f32ced9d640 1 -- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f32d0133a80 msgr2=0x7f32d013cab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:3301/0 2024-04-26T04:20:50.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.653+0000 7f32ced9d640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f32d0133a80 0x7f32d013cab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:20:50.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.653+0000 7f32c7fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f32d00a0090 0x7f32d0132960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:50.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.653+0000 7f32c7fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f32d00a0090 0x7f32d0132960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:50010/0 (socket says 172.21.15.63:50010) 2024-04-26T04:20:50.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.653+0000 7f32c7fff640 1 -- 172.21.15.63:0/1099216580 learned_addr learned my addr 172.21.15.63:0/1099216580 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:20:50.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.653+0000 7f32c7fff640 10 monclient(hunting): get_auth_request con 0x7f32d00a0090 auth_method 0 2024-04-26T04:20:50.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.653+0000 7f32c7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:50.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.653+0000 7f32c7fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:50.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.653+0000 7f32c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:20:50.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.653+0000 7f32c7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:20:50.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.653+0000 7f32c7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:20:50.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.653+0000 7f32c7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:20:50.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.653+0000 7f32ce59c640 1 --2- 172.21.15.63:0/1099216580 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f32d009fcb0 0x7f32d01cef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:50.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.653+0000 7f32ce59c640 10 monclient(hunting): get_auth_request con 0x7f32d009fcb0 auth_method 0 2024-04-26T04:20:50.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.653+0000 7f32ce59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:50.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.653+0000 7f32ce59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:50.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.653+0000 7f32ce59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:20:50.663 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.653+0000 7f32c7fff640 10 monclient(hunting): handle_auth_done global_id 4830 payload 824 2024-04-26T04:20:50.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.653+0000 7f32c7fff640 1 -- 172.21.15.63:0/1099216580 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f32d0133a80 msgr2=0x7f32d013cab0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T04:20:50.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.653+0000 7f32c7fff640 1 --2- 172.21.15.63:0/1099216580 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f32d0133a80 0x7f32d013cab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:50.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.653+0000 7f32c7fff640 1 -- 172.21.15.63:0/1099216580 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f32d009fcb0 msgr2=0x7f32d01cef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:50.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.653+0000 7f32c7fff640 1 --2- 172.21.15.63:0/1099216580 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f32d009fcb0 0x7f32d01cef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:50.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.653+0000 7f32c7fff640 10 monclient: _finish_hunting 0 2024-04-26T04:20:50.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.653+0000 7f32c7fff640 1 monclient: found mon.noname-a 2024-04-26T04:20:50.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.653+0000 7f32c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:20:50.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.653+0000 7f32c7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:20:50.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.653+0000 7f32c7fff640 1 -- 172.21.15.63:0/1099216580 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32d013e750 con 0x7f32d00a0090 2024-04-26T04:20:50.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.653+0000 7f32ce59c640 1 --2- 172.21.15.63:0/1099216580 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f32d009fcb0 0x7f32d01cef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T04:20:50.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.653+0000 7f32c7fff640 1 -- 172.21.15.63:0/1099216580 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f32d00a0090 msgr2=0x7f32d0132960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:20:50.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.653+0000 7f32c7fff640 1 --2- 172.21.15.63:0/1099216580 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f32d00a0090 0x7f32d0132960 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f32b800b580 tx=0x7f32b800ba40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:50.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.657+0000 7f32c7fff640 1 --2- 172.21.15.63:0/1099216580 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f32d00a0090 0x7f32d0132960 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f32b800b580 tx=0x7f32b800ba40 comp rx=0 tx=0).queue_received will delay after 1.71411e+09 on 0x7f32b8016070 mon_map magic: 0 2024-04-26T04:20:50.664 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.657+0000 7f32c6ffd640 1 -- 172.21.15.63:0/1099216580 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32b8016070 con 0x7f32d00a0090 2024-04-26T04:20:50.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.657+0000 7f32c6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:20:50.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.657+0000 7f32c6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:20:50.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.657+0000 7f32c6ffd640 10 monclient: dump: 2024-04-26T04:20:50.665 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:20:50.665 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:50.665 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:50.665 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:50.665 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:20:50.665 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:50.665 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:20:50.665 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:20:50.665 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:20:50.665 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:50.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.657+0000 7f32c6ffd640 10 monclient: _finish_auth 0 2024-04-26T04:20:50.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.657+0000 7f32c6ffd640 10 monclient: _check_auth_tickets 2024-04-26T04:20:50.665 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.657+0000 7f32c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:20:50.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.657+0000 7f32c6ffd640 1 -- 172.21.15.63:0/1099216580 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32b80050d0 con 0x7f32d00a0090 2024-04-26T04:20:50.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.657+0000 7f32c6ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T04:20:50.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.657+0000 7f32d4dce640 5 monclient: authenticate success, global_id 4830 2024-04-26T04:20:50.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.657+0000 7f32d4dce640 10 monclient: _renew_subs 2024-04-26T04:20:50.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.657+0000 7f32d4dce640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:20:50.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.657+0000 7f32d4dce640 1 -- 172.21.15.63:0/1099216580 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32d011ca50 con 0x7f32d00a0090 2024-04-26T04:20:50.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.657+0000 7f32cd59a640 4 set_mon_vals no callback set 2024-04-26T04:20:50.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.657+0000 7f32d4dce640 10 monclient: _renew_subs 2024-04-26T04:20:50.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.657+0000 7f32d4dce640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:20:50.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.657+0000 7f32d4dce640 1 -- 172.21.15.63:0/1099216580 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32d013edf0 con 0x7f32d00a0090 2024-04-26T04:20:50.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.657+0000 7f32d4dce640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:20:50.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.657+0000 7f32d4dce640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:20:50.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.657+0000 7f32d4dce640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:20:50.666 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.657+0000 7f32c6ffd640 1 -- 172.21.15.63:0/1099216580 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32b80054c0 con 0x7f32d00a0090 2024-04-26T04:20:50.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.657+0000 7f32d4dce640 1 -- 172.21.15.63:0/1099216580 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32d0062e10 con 0x7f32d00a0090 2024-04-26T04:20:50.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.657+0000 7f32c6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:20:50.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.657+0000 7f32c6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:20:50.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.657+0000 7f32c6ffd640 10 monclient: dump: 2024-04-26T04:20:50.667 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:20:50.667 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:50.667 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:50.667 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:50.667 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:20:50.667 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:50.667 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:20:50.667 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:20:50.667 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:20:50.667 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:50.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.657+0000 7f32c6ffd640 1 -- 172.21.15.63:0/1099216580 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f32b8019070 con 0x7f32d00a0090 2024-04-26T04:20:50.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.657+0000 7f32c6ffd640 1 --2- 172.21.15.63:0/1099216580 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f32b0035e70 0x7f32b0038330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:50.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.657+0000 7f32ce59c640 1 --2- 172.21.15.63:0/1099216580 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f32b0035e70 0x7f32b0038330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:50.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.657+0000 7f32ce59c640 10 monclient: get_auth_request con 0x7f32b0035e70 auth_method 0 2024-04-26T04:20:50.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.661+0000 7f32ce59c640 1 --2- 172.21.15.63:0/1099216580 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f32b0035e70 0x7f32b0038330 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f32c00ab000 tx=0x7f32c0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:50.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.661+0000 7f32c6ffd640 1 -- 172.21.15.63:0/1099216580 DispatchQueue::entry inject delay of 0.002000 2024-04-26T04:20:50.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:50.661+0000 7f32c6ffd640 1 -- 172.21.15.63:0/1099216580 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 4738+0+0 (secure 0 0 0) 0x7f32b8048240 con 0x7f32d00a0090 2024-04-26T04:20:51.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:51.281+0000 7f32c6ffd640 1 -- 172.21.15.63:0/1099216580 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f32d011ca50 con 0x7f32d00a0090 2024-04-26T04:20:51.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:51.281+0000 7f32c6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:20:51.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:51.281+0000 7f32c6ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:20:51.522 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:51.521+0000 7f32d4dce640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T04:20:51.522 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:51.521+0000 7f32d4dce640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T04:20:51.522 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:51.521+0000 7f32d4dce640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:20:51.522 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:51.521+0000 7f32d4dce640 1 -- 172.21.15.63:0/1099216580 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f32d01cfdc0 con 0x7f32d00a0090 2024-04-26T04:20:51.523 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:51.521+0000 7f32c6ffd640 1 -- 172.21.15.63:0/1099216580 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f32b8046c60 con 0x7f32d00a0090 2024-04-26T04:20:51.523 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:51.521+0000 7f32c6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T04:20:51.523 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:51.521+0000 7f32c6ffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:20:51.523 INFO:teuthology.orchestra.run.smithi063.stdout:{"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cd1c6a0c-4e4b-4014-89c0-2e535479ef39","modified":"2024-04-26T04:18:03.702339Z","created":"2024-04-26T04:18:03.702339Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3300","nonce":0},{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:3300","nonce":0},{"type":"v1","addr":"172.21.15.186:6789","nonce":0}]},"addr":"172.21.15.186:6789/0","public_addr":"172.21.15.186:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3301","nonce":0},{"type":"v1","addr":"172.21.15.63:6790","nonce":0}]},"addr":"172.21.15.63:6790/0","public_addr":"172.21.15.63:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T04:20:51.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:51.521+0000 7f32d4dce640 1 -- 172.21.15.63:0/1099216580 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f32b0035e70 msgr2=0x7f32b0038330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:51.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:51.521+0000 7f32d4dce640 1 --2- 172.21.15.63:0/1099216580 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f32b0035e70 0x7f32b0038330 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f32c00ab000 tx=0x7f32c0008040 comp rx=0 tx=0).stop 2024-04-26T04:20:51.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:51.521+0000 7f32d4dce640 10 monclient: shutdown 2024-04-26T04:20:51.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:51.521+0000 7f32d4dce640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:20:51.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:51.521+0000 7f32d4dce640 1 -- 172.21.15.63:0/1099216580 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f32d00a0090 msgr2=0x7f32d0132960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:51.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:51.521+0000 7f32d4dce640 1 --2- 172.21.15.63:0/1099216580 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f32d00a0090 0x7f32d0132960 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f32b800b580 tx=0x7f32b800ba40 comp rx=0 tx=0).stop 2024-04-26T04:20:51.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:51.525+0000 7f32d4dce640 1 -- 172.21.15.63:0/1099216580 shutdown_connections 2024-04-26T04:20:51.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:51.525+0000 7f32d4dce640 1 --2- 172.21.15.63:0/1099216580 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f32b0035e70 0x7f32b0038330 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:51.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:51.525+0000 7f32d4dce640 1 --2- 172.21.15.63:0/1099216580 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f32d0133a80 0x7f32d013cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:51.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:51.525+0000 7f32d4dce640 1 --2- 172.21.15.63:0/1099216580 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f32d00a0090 0x7f32d0132960 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:51.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:51.525+0000 7f32d4dce640 1 --2- 172.21.15.63:0/1099216580 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f32d009fcb0 0x7f32d01cef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:51.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:51.525+0000 7f32d4dce640 1 -- 172.21.15.63:0/1099216580 >> 172.21.15.63:0/1099216580 conn(0x7f32d00a1db0 msgr2=0x7f32d0134e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:20:51.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:51.525+0000 7f32d4dce640 1 -- 172.21.15.63:0/1099216580 shutdown_connections 2024-04-26T04:20:51.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:51.525+0000 7f32d4dce640 1 -- 172.21.15.63:0/1099216580 wait complete. 2024-04-26T04:20:54.544 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T04:20:54.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.673+0000 7f0b0f1a9640 10 monclient: get_monmap_and_config 2024-04-26T04:20:54.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.673+0000 7f0b0f1a9640 10 monclient: build_initial_monmap 2024-04-26T04:20:54.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.673+0000 7f0b0f1a9640 10 monclient: monmap: 2024-04-26T04:20:54.676 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:20:54.676 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:54.676 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:20:54.675840+0000 2024-04-26T04:20:54.676 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:20:54.675840+0000 2024-04-26T04:20:54.677 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:20:54.677 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:54.677 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:20:54.677 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:20:54.677 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:20:54.677 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:20:54.677 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:20:54.677 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:54.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.673+0000 7f0b0f1a9640 1 Processor -- start 2024-04-26T04:20:54.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.673+0000 7f0b0f1a9640 1 -- start start 2024-04-26T04:20:54.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.673+0000 7f0b0f1a9640 10 monclient: init 2024-04-26T04:20:54.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.673+0000 7f0b0f1a9640 10 monclient: _reopen_session rank -1 2024-04-26T04:20:54.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.673+0000 7f0b0f1a9640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-04-26T04:20:54.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.673+0000 7f0b0f1a9640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f0b080a2cd0 0x7f0b080a30b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:54.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.673+0000 7f0b0f1a9640 10 monclient(hunting): picked mon.noname-c con 0x7f0b080a2cd0 addr v2:172.21.15.186:3300/0 2024-04-26T04:20:54.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.673+0000 7f0b0f1a9640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f0b080a47d0 0x7f0b08136ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:54.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.673+0000 7f0b0f1a9640 10 monclient(hunting): picked mon.noname-a con 0x7f0b080a47d0 addr v2:172.21.15.63:3300/0 2024-04-26T04:20:54.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.673+0000 7f0b0f1a9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0b080a4c00 addr v1:172.21.15.186:6789/0 2024-04-26T04:20:54.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.673+0000 7f0b0f1a9640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b080a0fb0 con 0x7f0b080a4c00 2024-04-26T04:20:54.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.673+0000 7f0b0f1a9640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:54.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.673+0000 7f0b0f1a9640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7f0b080a35f0 con 0x7f0b080a47d0 2024-04-26T04:20:54.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.673+0000 7f0b0f1a9640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:54.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.673+0000 7f0b0f1a9640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f0b080a5380 con 0x7f0b080a2cd0 2024-04-26T04:20:54.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.673+0000 7f0b0f1a9640 10 monclient(hunting): _renew_subs 2024-04-26T04:20:54.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.673+0000 7f0b0f1a9640 10 monclient(hunting): authenticate will time out at 1086.383188s 2024-04-26T04:20:54.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.673+0000 7f0b07fff640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f0b080a47d0 0x7f0b08136ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:54.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.673+0000 7f0b0cf45640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f0b080a2cd0 0x7f0b080a30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:54.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.673+0000 7f0b07fff640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f0b080a47d0 0x7f0b08136ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:50020/0 (socket says 172.21.15.63:50020) 2024-04-26T04:20:54.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.673+0000 7f0b07fff640 1 -- 172.21.15.63:0/2857619629 learned_addr learned my addr 172.21.15.63:0/2857619629 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:20:54.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.673+0000 7f0b07fff640 10 monclient(hunting): get_auth_request con 0x7f0b080a47d0 auth_method 0 2024-04-26T04:20:54.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.673+0000 7f0b07fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:54.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.673+0000 7f0b07fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:54.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.673+0000 7f0b07fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:20:54.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.673+0000 7f0b0cf45640 10 monclient(hunting): get_auth_request con 0x7f0b080a2cd0 auth_method 0 2024-04-26T04:20:54.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.673+0000 7f0b0cf45640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:54.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.673+0000 7f0b0cf45640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:54.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.673+0000 7f0b0cf45640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:20:54.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.673+0000 7f0b07fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:20:54.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.673+0000 7f0b07fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:20:54.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.673+0000 7f0b07fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:20:54.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.673+0000 7f0b0cf45640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:20:54.682 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.673+0000 7f0b0cf45640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:20:54.682 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.673+0000 7f0b0cf45640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:20:54.682 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.673+0000 7f0b07fff640 10 monclient(hunting): handle_auth_done global_id 4833 payload 274 2024-04-26T04:20:54.682 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.673+0000 7f0b07fff640 1 -- 172.21.15.63:0/2857619629 >> v2:172.21.15.186:3300/0 conn(0x7f0b080a2cd0 msgr2=0x7f0b080a30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:20:54.682 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.673+0000 7f0b07fff640 1 --2- 172.21.15.63:0/2857619629 >> v2:172.21.15.186:3300/0 conn(0x7f0b080a2cd0 0x7f0b080a30b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:54.682 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.673+0000 7f0b07fff640 1 -- 172.21.15.63:0/2857619629 >> v1:172.21.15.186:6789/0 conn(0x7f0b080a4c00 legacy=0x7f0b08137220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:20:54.682 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.673+0000 7f0b07fff640 10 monclient: _finish_hunting 0 2024-04-26T04:20:54.682 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.673+0000 7f0b07fff640 1 monclient: found mon.noname-a 2024-04-26T04:20:54.682 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.673+0000 7f0b07fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:20:54.682 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.673+0000 7f0b07fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:20:54.682 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.673+0000 7f0b07fff640 1 -- 172.21.15.63:0/2857619629 --> v2:172.21.15.63:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b080a1290 con 0x7f0b080a47d0 2024-04-26T04:20:54.685 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b07fff640 1 -- 172.21.15.63:0/2857619629 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b080a47d0 msgr2=0x7f0b08136ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:20:54.686 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b07fff640 1 --2- 172.21.15.63:0/2857619629 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b080a47d0 0x7f0b08136ce0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f0b0806c760 tx=0x7f0af001c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c3778b3cacec7b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:54.686 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b077fe640 1 -- 172.21.15.63:0/2857619629 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0af00a5650 con 0x7f0b080a47d0 2024-04-26T04:20:54.686 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b077fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:20:54.686 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b077fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:20:54.686 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b077fe640 10 monclient: dump: 2024-04-26T04:20:54.686 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:20:54.686 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:54.687 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:54.687 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:54.687 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:20:54.687 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:54.687 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:20:54.687 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:20:54.687 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:20:54.687 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:54.687 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b077fe640 10 monclient: _finish_auth 0 2024-04-26T04:20:54.687 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b077fe640 10 monclient: _check_auth_tickets 2024-04-26T04:20:54.687 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:20:54.687 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b077fe640 1 -- 172.21.15.63:0/2857619629 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0af00a57f0 con 0x7f0b080a47d0 2024-04-26T04:20:54.687 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b077fe640 10 monclient: handle_config config(0 keys) 2024-04-26T04:20:54.687 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b077fe640 1 -- 172.21.15.63:0/2857619629 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0af00a5bc0 con 0x7f0b080a47d0 2024-04-26T04:20:54.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b0f1a9640 5 monclient: authenticate success, global_id 4833 2024-04-26T04:20:54.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b0f1a9640 10 monclient: get_monmap_and_config success 2024-04-26T04:20:54.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b0f1a9640 4 set_mon_vals no callback set 2024-04-26T04:20:54.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b077fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:20:54.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b077fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:20:54.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b077fe640 10 monclient: dump: 2024-04-26T04:20:54.688 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:20:54.688 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:54.688 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:54.688 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:54.688 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:20:54.688 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:54.688 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:20:54.688 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:20:54.688 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:20:54.689 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:54.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b0f1a9640 10 monclient: shutdown 2024-04-26T04:20:54.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b0f1a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:20:54.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b0f1a9640 1 -- 172.21.15.63:0/2857619629 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b080a47d0 msgr2=0x7f0b08136ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:54.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b0f1a9640 1 --2- 172.21.15.63:0/2857619629 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b080a47d0 0x7f0b08136ce0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f0b0806c760 tx=0x7f0af001c990 comp rx=0 tx=0).stop 2024-04-26T04:20:54.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b0f1a9640 1 -- 172.21.15.63:0/2857619629 shutdown_connections 2024-04-26T04:20:54.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b0f1a9640 1 --2- 172.21.15.63:0/2857619629 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b080a47d0 0x7f0b08136ce0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:54.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b0f1a9640 1 --2- 172.21.15.63:0/2857619629 >> v2:172.21.15.186:3300/0 conn(0x7f0b080a2cd0 0x7f0b080a30b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:54.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b0f1a9640 1 -- 172.21.15.63:0/2857619629 >> 172.21.15.63:0/2857619629 conn(0x7f0b080a5f40 msgr2=0x7f0b080a6320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:20:54.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b0f1a9640 1 -- 172.21.15.63:0/2857619629 shutdown_connections 2024-04-26T04:20:54.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b0f1a9640 1 -- 172.21.15.63:0/2857619629 wait complete. 2024-04-26T04:20:54.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b0f1a9640 10 monclient: build_initial_monmap 2024-04-26T04:20:54.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b0f1a9640 10 monclient: monmap: 2024-04-26T04:20:54.689 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:20:54.690 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:20:54.690 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:20:54.690 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:20:54.690 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:20:54.690 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:54.690 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:20:54.690 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:20:54.690 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:20:54.690 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:54.690 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b0f1a9640 1 Processor -- start 2024-04-26T04:20:54.690 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b0f1a9640 1 -- start start 2024-04-26T04:20:54.690 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b0f1a9640 10 monclient: init 2024-04-26T04:20:54.690 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b0f1a9640 10 monclient: _reopen_session rank -1 2024-04-26T04:20:54.690 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b0f1a9640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T04:20:54.690 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b0f1a9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b080a2cd0 0x7f0b081ced70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:54.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b0f1a9640 10 monclient(hunting): picked mon.noname-b con 0x7f0b080a2cd0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:20:54.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b0f1a9640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f0b080a47d0 0x7f0b081cf540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:54.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b0f1a9640 10 monclient(hunting): picked mon.noname-c con 0x7f0b080a47d0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:20:54.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b0f1a9640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b080a4c00 0x7f0b0813bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:54.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b0f1a9640 10 monclient(hunting): picked mon.noname-a con 0x7f0b080a4c00 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:20:54.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b0f1a9640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:54.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b0f1a9640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f0b08141a50 con 0x7f0b080a4c00 2024-04-26T04:20:54.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b0f1a9640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:54.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b0f1a9640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0b08137b40 con 0x7f0b080a2cd0 2024-04-26T04:20:54.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b0f1a9640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:54.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b0f1a9640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f0b08141bd0 con 0x7f0b080a47d0 2024-04-26T04:20:54.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b0f1a9640 10 monclient(hunting): _renew_subs 2024-04-26T04:20:54.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b0f1a9640 10 monclient(hunting): authenticate will time out at 1086.390393s 2024-04-26T04:20:54.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b07fff640 1 -- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f0b080a47d0 msgr2=0x7f0b081cf540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:3301/0 2024-04-26T04:20:54.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b07fff640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f0b080a47d0 0x7f0b081cf540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:20:54.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b0d746640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b080a4c00 0x7f0b0813bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:54.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b0d746640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b080a4c00 0x7f0b0813bb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:50026/0 (socket says 172.21.15.63:50026) 2024-04-26T04:20:54.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b0d746640 1 -- 172.21.15.63:0/1939009827 learned_addr learned my addr 172.21.15.63:0/1939009827 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:20:54.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b0d746640 10 monclient(hunting): get_auth_request con 0x7f0b080a4c00 auth_method 0 2024-04-26T04:20:54.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b0d746640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:54.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b0d746640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:54.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b0d746640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:20:54.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b0d746640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:20:54.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b0d746640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:20:54.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b0d746640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:20:54.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b0cf45640 1 --2- 172.21.15.63:0/1939009827 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b080a2cd0 0x7f0b081ced70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:54.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b0d746640 10 monclient(hunting): handle_auth_done global_id 4836 payload 824 2024-04-26T04:20:54.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b0d746640 1 -- 172.21.15.63:0/1939009827 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f0b080a47d0 msgr2=0x7f0b081cf540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T04:20:54.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b0d746640 1 --2- 172.21.15.63:0/1939009827 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f0b080a47d0 0x7f0b081cf540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:54.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b0d746640 1 -- 172.21.15.63:0/1939009827 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b080a2cd0 msgr2=0x7f0b081ced70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:54.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b0d746640 1 --2- 172.21.15.63:0/1939009827 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b080a2cd0 0x7f0b081ced70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:54.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b0d746640 10 monclient: _finish_hunting 0 2024-04-26T04:20:54.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b0d746640 1 monclient: found mon.noname-a 2024-04-26T04:20:54.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b0d746640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:20:54.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b0d746640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:20:54.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b0d746640 1 -- 172.21.15.63:0/1939009827 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b080a37d0 con 0x7f0b080a4c00 2024-04-26T04:20:54.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b0cf45640 10 monclient: get_auth_request con 0x7f0b080a2cd0 auth_method 0 2024-04-26T04:20:54.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.681+0000 7f0b0cf45640 1 --2- 172.21.15.63:0/1939009827 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b080a2cd0 0x7f0b081ced70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T04:20:54.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.685+0000 7f0b0d746640 1 -- 172.21.15.63:0/1939009827 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b080a4c00 msgr2=0x7f0b0813bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:20:54.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.685+0000 7f0b0d746640 1 --2- 172.21.15.63:0/1939009827 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b080a4c00 0x7f0b0813bb20 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0b0000c3c0 tx=0x7f0b0000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:54.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.685+0000 7f0b057fa640 1 -- 172.21.15.63:0/1939009827 DispatchQueue::entry inject delay of 0.002000 2024-04-26T04:20:54.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.685+0000 7f0b057fa640 1 -- 172.21.15.63:0/1939009827 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b000119a0 con 0x7f0b080a4c00 2024-04-26T04:20:54.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.685+0000 7f0b057fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:20:54.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.685+0000 7f0b057fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:20:54.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.685+0000 7f0b057fa640 10 monclient: dump: 2024-04-26T04:20:54.695 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:20:54.695 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:54.695 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:54.696 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:54.696 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:20:54.696 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:54.696 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:20:54.696 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:20:54.696 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:20:54.696 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:54.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.685+0000 7f0b057fa640 10 monclient: _finish_auth 0 2024-04-26T04:20:54.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.685+0000 7f0b057fa640 10 monclient: _check_auth_tickets 2024-04-26T04:20:54.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.685+0000 7f0b057fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:20:54.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.685+0000 7f0b057fa640 1 -- 172.21.15.63:0/1939009827 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b00017070 con 0x7f0b080a4c00 2024-04-26T04:20:54.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.685+0000 7f0b057fa640 10 monclient: handle_config config(0 keys) 2024-04-26T04:20:54.697 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.685+0000 7f0b057fa640 1 -- 172.21.15.63:0/1939009827 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b00011d80 con 0x7f0b080a4c00 2024-04-26T04:20:54.697 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.685+0000 7f0b057fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:20:54.697 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.685+0000 7f0b057fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:20:54.697 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.685+0000 7f0b057fa640 10 monclient: dump: 2024-04-26T04:20:54.697 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:20:54.697 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:54.697 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:54.697 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:20:54.697 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:20:54.697 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:54.697 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:20:54.697 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:20:54.697 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:20:54.698 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:54.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.685+0000 7f0b0f1a9640 5 monclient: authenticate success, global_id 4836 2024-04-26T04:20:54.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.685+0000 7f0b06ffd640 4 set_mon_vals no callback set 2024-04-26T04:20:54.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.685+0000 7f0b0f1a9640 10 monclient: _renew_subs 2024-04-26T04:20:54.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.685+0000 7f0b0f1a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:20:54.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.685+0000 7f0b0f1a9640 1 -- 172.21.15.63:0/1939009827 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b08072ae0 con 0x7f0b080a4c00 2024-04-26T04:20:54.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.685+0000 7f0b0f1a9640 10 monclient: _renew_subs 2024-04-26T04:20:54.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.685+0000 7f0b0f1a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:20:54.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.685+0000 7f0b0f1a9640 1 -- 172.21.15.63:0/1939009827 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b081201f0 con 0x7f0b080a4c00 2024-04-26T04:20:54.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.689+0000 7f0b0f1a9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:20:54.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.689+0000 7f0b0f1a9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:20:54.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.689+0000 7f0b0f1a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:20:54.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.689+0000 7f0b0f1a9640 1 -- 172.21.15.63:0/1939009827 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b08136240 con 0x7f0b080a4c00 2024-04-26T04:20:54.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.689+0000 7f0b057fa640 1 -- 172.21.15.63:0/1939009827 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f0b0001c460 con 0x7f0b080a4c00 2024-04-26T04:20:54.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.689+0000 7f0b057fa640 1 --2- 172.21.15.63:0/1939009827 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f0ae8035ec0 0x7f0ae8038380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:54.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.689+0000 7f0b057fa640 1 -- 172.21.15.63:0/1939009827 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(114..114 src has 1..114) ==== 4738+0+0 (secure 0 0 0) 0x7f0b00048d20 con 0x7f0b080a4c00 2024-04-26T04:20:54.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.689+0000 7f0b0cf45640 1 --2- 172.21.15.63:0/1939009827 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f0ae8035ec0 0x7f0ae8038380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:54.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.689+0000 7f0b0cf45640 10 monclient: get_auth_request con 0x7f0ae8035ec0 auth_method 0 2024-04-26T04:20:54.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.689+0000 7f0b0cf45640 1 --2- 172.21.15.63:0/1939009827 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f0ae8035ec0 0x7f0ae8038380 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f0af8009a40 tx=0x7f0af8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:20:54.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.693+0000 7f0b057fa640 1 -- 172.21.15.63:0/1939009827 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f0b08136240 con 0x7f0b080a4c00 2024-04-26T04:20:54.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.693+0000 7f0b057fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:20:54.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.693+0000 7f0b057fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:20:54.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.913+0000 7f0b0f1a9640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T04:20:54.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.913+0000 7f0b0f1a9640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T04:20:54.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.913+0000 7f0b0f1a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:20:54.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.913+0000 7f0b0f1a9640 1 -- 172.21.15.63:0/1939009827 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f0b080683d0 con 0x7f0b080a4c00 2024-04-26T04:20:54.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.913+0000 7f0b057fa640 1 -- 172.21.15.63:0/1939009827 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f0b0001a720 con 0x7f0b080a4c00 2024-04-26T04:20:54.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.913+0000 7f0b057fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T04:20:54.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.913+0000 7f0b057fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:20:54.916 INFO:teuthology.orchestra.run.smithi063.stdout:{"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cd1c6a0c-4e4b-4014-89c0-2e535479ef39","modified":"2024-04-26T04:18:03.702339Z","created":"2024-04-26T04:18:03.702339Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3300","nonce":0},{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:3300","nonce":0},{"type":"v1","addr":"172.21.15.186:6789","nonce":0}]},"addr":"172.21.15.186:6789/0","public_addr":"172.21.15.186:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3301","nonce":0},{"type":"v1","addr":"172.21.15.63:6790","nonce":0}]},"addr":"172.21.15.63:6790/0","public_addr":"172.21.15.63:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T04:20:54.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.917+0000 7f0b0f1a9640 1 -- 172.21.15.63:0/1939009827 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f0ae8035ec0 msgr2=0x7f0ae8038380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:54.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.917+0000 7f0b0f1a9640 1 --2- 172.21.15.63:0/1939009827 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f0ae8035ec0 0x7f0ae8038380 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f0af8009a40 tx=0x7f0af8008040 comp rx=0 tx=0).stop 2024-04-26T04:20:54.919 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.917+0000 7f0b0f1a9640 10 monclient: shutdown 2024-04-26T04:20:54.919 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.917+0000 7f0b0f1a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:20:54.919 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.917+0000 7f0b0f1a9640 1 -- 172.21.15.63:0/1939009827 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b080a4c00 msgr2=0x7f0b0813bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:20:54.919 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.917+0000 7f0b0f1a9640 1 --2- 172.21.15.63:0/1939009827 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b080a4c00 0x7f0b0813bb20 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0b0000c3c0 tx=0x7f0b0000c880 comp rx=0 tx=0).stop 2024-04-26T04:20:54.919 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.917+0000 7f0b0f1a9640 1 -- 172.21.15.63:0/1939009827 shutdown_connections 2024-04-26T04:20:54.919 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.917+0000 7f0b0f1a9640 1 --2- 172.21.15.63:0/1939009827 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f0ae8035ec0 0x7f0ae8038380 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:54.919 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.917+0000 7f0b0f1a9640 1 --2- 172.21.15.63:0/1939009827 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b080a4c00 0x7f0b0813bb20 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:54.919 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.917+0000 7f0b0f1a9640 1 --2- 172.21.15.63:0/1939009827 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f0b080a47d0 0x7f0b081cf540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:54.919 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.917+0000 7f0b0f1a9640 1 --2- 172.21.15.63:0/1939009827 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b080a2cd0 0x7f0b081ced70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:20:54.919 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.917+0000 7f0b0f1a9640 1 -- 172.21.15.63:0/1939009827 >> 172.21.15.63:0/1939009827 conn(0x7f0b080a5f40 msgr2=0x7f0b080a7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:20:54.919 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.917+0000 7f0b0f1a9640 1 -- 172.21.15.63:0/1939009827 shutdown_connections 2024-04-26T04:20:54.919 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:54.917+0000 7f0b0f1a9640 1 -- 172.21.15.63:0/1939009827 wait complete. 2024-04-26T04:20:57.936 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T04:20:58.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:58.065+0000 7f9463cf0640 10 monclient: get_monmap_and_config 2024-04-26T04:20:58.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:58.065+0000 7f9463cf0640 10 monclient: build_initial_monmap 2024-04-26T04:20:58.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:58.065+0000 7f9463cf0640 10 monclient: monmap: 2024-04-26T04:20:58.068 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:20:58.068 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:20:58.068 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:20:58.067564+0000 2024-04-26T04:20:58.068 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:20:58.067564+0000 2024-04-26T04:20:58.068 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:20:58.068 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:20:58.068 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:20:58.068 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:20:58.068 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:20:58.068 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:20:58.068 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:20:58.068 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:20:58.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:58.065+0000 7f9463cf0640 1 Processor -- start 2024-04-26T04:20:58.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:58.065+0000 7f9463cf0640 1 -- start start 2024-04-26T04:20:58.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:58.065+0000 7f9463cf0640 10 monclient: init 2024-04-26T04:20:58.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:58.065+0000 7f9463cf0640 10 monclient: _reopen_session rank -1 2024-04-26T04:20:58.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:58.065+0000 7f9463cf0640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-04-26T04:20:58.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:58.065+0000 7f9463cf0640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f945c0a0bb0 0x7f945c09eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:58.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:58.065+0000 7f9463cf0640 10 monclient(hunting): picked mon.noname-b con 0x7f945c0a0bb0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:20:58.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:58.065+0000 7f9463cf0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f945c09d7b0 addr v1:172.21.15.186:6789/0 2024-04-26T04:20:58.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:58.065+0000 7f9463cf0640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f945c09e2a0 0x7f945c13b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:20:58.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:58.065+0000 7f9463cf0640 10 monclient(hunting): picked mon.noname-c con 0x7f945c09e2a0 addr v2:172.21.15.186:3300/0 2024-04-26T04:20:58.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:58.065+0000 7f9463cf0640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f945c0a1930 con 0x7f945c09d7b0 2024-04-26T04:20:58.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:58.065+0000 7f9463cf0640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:58.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:58.065+0000 7f9463cf0640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f945c0a0f90 con 0x7f945c09e2a0 2024-04-26T04:20:58.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:58.065+0000 7f9463cf0640 10 monclient(hunting): start opening mon connection 2024-04-26T04:20:58.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:58.065+0000 7f9463cf0640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f945c099240 con 0x7f945c0a0bb0 2024-04-26T04:20:58.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:58.065+0000 7f9463cf0640 10 monclient(hunting): _renew_subs 2024-04-26T04:20:58.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:58.065+0000 7f9463cf0640 10 monclient(hunting): authenticate will time out at 1089.775504s 2024-04-26T04:20:58.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:58.065+0000 7f9461a8c640 1 -- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f945c0a0bb0 msgr2=0x7f945c09eb60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.63:3301/0 2024-04-26T04:20:58.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:58.065+0000 7f9461a8c640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f945c0a0bb0 0x7f945c09eb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:20:58.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:58.065+0000 7f946228d640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f945c09e2a0 0x7f945c13b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:58.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:58.069+0000 7f946228d640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f945c09e2a0 0x7f945c13b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.63:41872/0 (socket says 172.21.15.63:41872) 2024-04-26T04:20:58.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:58.069+0000 7f946228d640 1 -- 172.21.15.63:0/3326135823 learned_addr learned my addr 172.21.15.63:0/3326135823 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:20:58.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:58.069+0000 7f946228d640 10 monclient(hunting): get_auth_request con 0x7f945c09e2a0 auth_method 0 2024-04-26T04:20:58.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:58.069+0000 7f946228d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:58.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:58.069+0000 7f946228d640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:58.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:58.069+0000 7f946228d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:20:58.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:58.069+0000 7f946228d640 1 -- 172.21.15.63:0/3326135823 >> v2:172.21.15.186:3300/0 conn(0x7f945c09e2a0 msgr2=0x7f945c13b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T04:20:58.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:58.069+0000 7f946228d640 1 -- 172.21.15.63:0/3326135823 >> v2:172.21.15.186:3300/0 conn(0x7f945c09e2a0 msgr2=0x7f945c13b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:20:58.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:58.069+0000 7f946228d640 1 --2- 172.21.15.63:0/3326135823 >> v2:172.21.15.186:3300/0 conn(0x7f945c09e2a0 0x7f945c13b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:20:58.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:58.069+0000 7f946228d640 1 --2- 172.21.15.63:0/3326135823 >> v2:172.21.15.186:3300/0 conn(0x7f945c09e2a0 0x7f945c13b340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:20:58.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:58.069+0000 7f946128b640 1 -- 172.21.15.63:0/3326135823 >> v1:172.21.15.186:6789/0 conn(0x7f945c09d7b0 legacy=0x7f945c09db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:20:58.270 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:58.269+0000 7f9461a8c640 1 -- 172.21.15.63:0/3326135823 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f945c0a0bb0 msgr2=0x7f945c09eb60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.63:3301/0 2024-04-26T04:20:58.270 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:58.269+0000 7f9461a8c640 1 --2- 172.21.15.63:0/3326135823 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f945c0a0bb0 0x7f945c09eb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T04:20:58.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:58.273+0000 7f946228d640 1 --2- 172.21.15.63:0/3326135823 >> v2:172.21.15.186:3300/0 conn(0x7f945c09e2a0 0x7f945c13b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:58.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:58.273+0000 7f946228d640 10 monclient(hunting): get_auth_request con 0x7f945c09e2a0 auth_method 0 2024-04-26T04:20:58.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:58.273+0000 7f946228d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:58.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:58.273+0000 7f946228d640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:58.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:58.273+0000 7f946228d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:20:58.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:58.273+0000 7f946228d640 1 -- 172.21.15.63:0/3326135823 >> v2:172.21.15.186:3300/0 conn(0x7f945c09e2a0 msgr2=0x7f945c13b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T04:20:58.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:58.273+0000 7f946228d640 1 -- 172.21.15.63:0/3326135823 >> v2:172.21.15.186:3300/0 conn(0x7f945c09e2a0 msgr2=0x7f945c13b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:20:58.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:58.273+0000 7f946228d640 1 --2- 172.21.15.63:0/3326135823 >> v2:172.21.15.186:3300/0 conn(0x7f945c09e2a0 0x7f945c13b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:20:58.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:58.273+0000 7f946228d640 1 --2- 172.21.15.63:0/3326135823 >> v2:172.21.15.186:3300/0 conn(0x7f945c09e2a0 0x7f945c13b340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T04:20:58.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:58.673+0000 7f9461a8c640 1 -- 172.21.15.63:0/3326135823 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f945c0a0bb0 msgr2=0x7f945c09eb60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.63:3301/0 2024-04-26T04:20:58.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:58.673+0000 7f9461a8c640 1 --2- 172.21.15.63:0/3326135823 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f945c0a0bb0 0x7f945c09eb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T04:20:58.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:58.677+0000 7f946228d640 1 --2- 172.21.15.63:0/3326135823 >> v2:172.21.15.186:3300/0 conn(0x7f945c09e2a0 0x7f945c13b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:58.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:58.677+0000 7f946228d640 10 monclient(hunting): get_auth_request con 0x7f945c09e2a0 auth_method 0 2024-04-26T04:20:58.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:58.677+0000 7f946228d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:58.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:58.677+0000 7f946228d640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:58.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:58.677+0000 7f946228d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:20:58.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:58.677+0000 7f946228d640 1 -- 172.21.15.63:0/3326135823 >> v2:172.21.15.186:3300/0 conn(0x7f945c09e2a0 msgr2=0x7f945c13b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T04:20:58.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:58.677+0000 7f946228d640 1 -- 172.21.15.63:0/3326135823 >> v2:172.21.15.186:3300/0 conn(0x7f945c09e2a0 msgr2=0x7f945c13b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:20:58.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:58.677+0000 7f946228d640 1 --2- 172.21.15.63:0/3326135823 >> v2:172.21.15.186:3300/0 conn(0x7f945c09e2a0 0x7f945c13b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:20:58.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:58.677+0000 7f946228d640 1 --2- 172.21.15.63:0/3326135823 >> v2:172.21.15.186:3300/0 conn(0x7f945c09e2a0 0x7f945c13b340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T04:20:59.478 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:59.477+0000 7f9461a8c640 1 -- 172.21.15.63:0/3326135823 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f945c0a0bb0 msgr2=0x7f945c09eb60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.63:3301/0 2024-04-26T04:20:59.478 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:59.477+0000 7f9461a8c640 1 --2- 172.21.15.63:0/3326135823 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f945c0a0bb0 0x7f945c09eb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T04:20:59.482 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:59.481+0000 7f946228d640 1 --2- 172.21.15.63:0/3326135823 >> v2:172.21.15.186:3300/0 conn(0x7f945c09e2a0 0x7f945c13b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:20:59.482 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:59.481+0000 7f946228d640 10 monclient(hunting): get_auth_request con 0x7f945c09e2a0 auth_method 0 2024-04-26T04:20:59.482 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:59.481+0000 7f946228d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:20:59.483 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:59.481+0000 7f946228d640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:20:59.483 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:59.481+0000 7f946228d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:20:59.483 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:59.481+0000 7f946228d640 1 -- 172.21.15.63:0/3326135823 >> v2:172.21.15.186:3300/0 conn(0x7f945c09e2a0 msgr2=0x7f945c13b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T04:20:59.483 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:59.481+0000 7f946228d640 1 -- 172.21.15.63:0/3326135823 >> v2:172.21.15.186:3300/0 conn(0x7f945c09e2a0 msgr2=0x7f945c13b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:20:59.483 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:59.481+0000 7f946228d640 1 --2- 172.21.15.63:0/3326135823 >> v2:172.21.15.186:3300/0 conn(0x7f945c09e2a0 0x7f945c13b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:20:59.483 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:20:59.481+0000 7f946228d640 1 --2- 172.21.15.63:0/3326135823 >> v2:172.21.15.186:3300/0 conn(0x7f945c09e2a0 0x7f945c13b340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T04:21:01.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.065+0000 7f945b7fe640 10 monclient(hunting): tick 2024-04-26T04:21:01.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.065+0000 7f945b7fe640 10 monclient(hunting): _check_auth_tickets 2024-04-26T04:21:01.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.065+0000 7f945b7fe640 1 monclient(hunting): continuing hunt 2024-04-26T04:21:01.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.065+0000 7f945b7fe640 10 monclient(hunting): _reopen_session rank -1 2024-04-26T04:21:01.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.065+0000 7f945b7fe640 1 -- 172.21.15.63:0/3326135823 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f945c0a0bb0 msgr2=0x7f945c09eb60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T04:21:01.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.065+0000 7f945b7fe640 1 --2- 172.21.15.63:0/3326135823 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f945c0a0bb0 0x7f945c09eb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:21:01.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.065+0000 7f945b7fe640 1 -- 172.21.15.63:0/3326135823 >> v2:172.21.15.186:3300/0 conn(0x7f945c09e2a0 msgr2=0x7f945c13b340 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T04:21:01.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.065+0000 7f945b7fe640 1 --2- 172.21.15.63:0/3326135823 >> v2:172.21.15.186:3300/0 conn(0x7f945c09e2a0 0x7f945c13b340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:21:01.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.065+0000 7f945b7fe640 1 -- 172.21.15.63:0/3326135823 >> v1:172.21.15.186:6789/0 conn(0x7f945c09d7b0 legacy=0x7f945c09db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:21:01.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.065+0000 7f945b7fe640 10 monclient: _add_conns ranks=[3,0] 2024-04-26T04:21:01.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.065+0000 7f945b7fe640 1 --2- 172.21.15.63:0/3326135823 >> v2:172.21.15.63:3300/0 conn(0x7f9448003770 0x7f9448005bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:21:01.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.065+0000 7f945b7fe640 10 monclient(hunting): picked mon.noname-a con 0x7f9448003770 addr v2:172.21.15.63:3300/0 2024-04-26T04:21:01.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.065+0000 7f945b7fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f94480062c0 addr v1:172.21.15.63:6789/0 2024-04-26T04:21:01.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.065+0000 7f945b7fe640 1 -- 172.21.15.63:0/3326135823 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f945c09b820 con 0x7f94480062c0 2024-04-26T04:21:01.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.065+0000 7f945b7fe640 10 monclient(hunting): start opening mon connection 2024-04-26T04:21:01.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.065+0000 7f945b7fe640 1 -- 172.21.15.63:0/3326135823 --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7f945c0a0f90 con 0x7f9448003770 2024-04-26T04:21:01.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.065+0000 7f945b7fe640 10 monclient(hunting): _renew_subs 2024-04-26T04:21:01.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.069+0000 7f9461a8c640 1 --2- 172.21.15.63:0/3326135823 >> v2:172.21.15.63:3300/0 conn(0x7f9448003770 0x7f9448005bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:21:01.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.069+0000 7f9461a8c640 10 monclient(hunting): get_auth_request con 0x7f9448003770 auth_method 0 2024-04-26T04:21:01.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.069+0000 7f9461a8c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:21:01.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.069+0000 7f9461a8c640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:21:01.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.069+0000 7f9461a8c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:21:01.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.069+0000 7f9461a8c640 1 -- 172.21.15.63:0/3326135823 >> v2:172.21.15.63:3300/0 conn(0x7f9448003770 msgr2=0x7f9448005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T04:21:01.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.069+0000 7f9461a8c640 1 -- 172.21.15.63:0/3326135823 >> v2:172.21.15.63:3300/0 conn(0x7f9448003770 msgr2=0x7f9448005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:21:01.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.069+0000 7f9461a8c640 1 --2- 172.21.15.63:0/3326135823 >> v2:172.21.15.63:3300/0 conn(0x7f9448003770 0x7f9448005bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:21:01.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.069+0000 7f9461a8c640 1 --2- 172.21.15.63:0/3326135823 >> v2:172.21.15.63:3300/0 conn(0x7f9448003770 0x7f9448005bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:21:01.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.069+0000 7f946128b640 1 -- 172.21.15.63:0/3326135823 >> v1:172.21.15.63:6789/0 conn(0x7f94480062c0 legacy=0x7f9448008780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:21:01.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.273+0000 7f9461a8c640 1 --2- 172.21.15.63:0/3326135823 >> v2:172.21.15.63:3300/0 conn(0x7f9448003770 0x7f9448005bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:21:01.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.273+0000 7f9461a8c640 10 monclient(hunting): get_auth_request con 0x7f9448003770 auth_method 0 2024-04-26T04:21:01.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.273+0000 7f9461a8c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:21:01.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.273+0000 7f9461a8c640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:21:01.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.273+0000 7f9461a8c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:21:01.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.273+0000 7f9461a8c640 1 -- 172.21.15.63:0/3326135823 >> v2:172.21.15.63:3300/0 conn(0x7f9448003770 msgr2=0x7f9448005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T04:21:01.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.273+0000 7f9461a8c640 1 -- 172.21.15.63:0/3326135823 >> v2:172.21.15.63:3300/0 conn(0x7f9448003770 msgr2=0x7f9448005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:21:01.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.273+0000 7f9461a8c640 1 --2- 172.21.15.63:0/3326135823 >> v2:172.21.15.63:3300/0 conn(0x7f9448003770 0x7f9448005bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:21:01.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.273+0000 7f9461a8c640 1 --2- 172.21.15.63:0/3326135823 >> v2:172.21.15.63:3300/0 conn(0x7f9448003770 0x7f9448005bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T04:21:01.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.677+0000 7f9461a8c640 1 --2- 172.21.15.63:0/3326135823 >> v2:172.21.15.63:3300/0 conn(0x7f9448003770 0x7f9448005bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:21:01.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.677+0000 7f9461a8c640 10 monclient(hunting): get_auth_request con 0x7f9448003770 auth_method 0 2024-04-26T04:21:01.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.677+0000 7f9461a8c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:21:01.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.677+0000 7f9461a8c640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:21:01.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.677+0000 7f9461a8c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:21:01.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.677+0000 7f9461a8c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:21:01.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.677+0000 7f9461a8c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:21:01.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.677+0000 7f9461a8c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:21:01.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.677+0000 7f9461a8c640 10 monclient(hunting): handle_auth_done global_id 4839 payload 274 2024-04-26T04:21:01.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.677+0000 7f9461a8c640 1 -- 172.21.15.63:0/3326135823 >> v1:172.21.15.63:6789/0 conn(0x7f94480062c0 legacy=0x7f9448008780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:21:01.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.677+0000 7f9461a8c640 10 monclient: _finish_hunting 0 2024-04-26T04:21:01.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.677+0000 7f9461a8c640 1 monclient: found mon.noname-a 2024-04-26T04:21:01.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.677+0000 7f9461a8c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:21:01.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.677+0000 7f9461a8c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:21:01.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.677+0000 7f9461a8c640 1 -- 172.21.15.63:0/3326135823 --> v2:172.21.15.63:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f944800b0d0 con 0x7f9448003770 2024-04-26T04:21:01.686 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.685+0000 7f9461a8c640 1 -- 172.21.15.63:0/3326135823 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9448003770 msgr2=0x7f9448005bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:21:01.686 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.685+0000 7f9461a8c640 1 --2- 172.21.15.63:0/3326135823 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9448003770 0x7f9448005bd0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f944c003990 tx=0x7f944c01d080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9593a0f1480a7f48 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:21:01.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.697+0000 7f9460a8a640 1 -- 172.21.15.63:0/3326135823 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f944c0b1070 con 0x7f9448003770 2024-04-26T04:21:01.702 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.697+0000 7f9460a8a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:21:01.702 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.697+0000 7f9460a8a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:21:01.702 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.697+0000 7f9460a8a640 10 monclient: dump: 2024-04-26T04:21:01.702 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:21:01.702 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:21:01.702 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:21:01.702 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:21:01.702 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:21:01.702 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:21:01.703 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:21:01.703 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:21:01.703 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:21:01.703 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:21:01.703 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.697+0000 7f9460a8a640 10 monclient: _finish_auth 0 2024-04-26T04:21:01.703 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.697+0000 7f9460a8a640 10 monclient: _check_auth_tickets 2024-04-26T04:21:01.703 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.697+0000 7f9460a8a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:21:01.703 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.697+0000 7f9460a8a640 1 -- 172.21.15.63:0/3326135823 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f944c0ac600 con 0x7f9448003770 2024-04-26T04:21:01.703 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.697+0000 7f9460a8a640 10 monclient: handle_config config(0 keys) 2024-04-26T04:21:01.703 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.697+0000 7f9460a8a640 1 -- 172.21.15.63:0/3326135823 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f944c0ac9d0 con 0x7f9448003770 2024-04-26T04:21:01.703 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.697+0000 7f9463cf0640 5 monclient: authenticate success, global_id 4839 2024-04-26T04:21:01.703 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.697+0000 7f9463cf0640 10 monclient: get_monmap_and_config success 2024-04-26T04:21:01.703 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.697+0000 7f9463cf0640 4 set_mon_vals no callback set 2024-04-26T04:21:01.704 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.697+0000 7f9460a8a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:21:01.704 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.697+0000 7f9460a8a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:21:01.704 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.697+0000 7f9460a8a640 10 monclient: dump: 2024-04-26T04:21:01.704 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:21:01.704 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:21:01.704 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:21:01.704 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:21:01.704 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:21:01.704 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:21:01.704 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:21:01.704 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:21:01.704 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:21:01.704 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:21:01.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.701+0000 7f9463cf0640 10 monclient: shutdown 2024-04-26T04:21:01.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.701+0000 7f9463cf0640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:21:01.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.701+0000 7f9463cf0640 1 -- 172.21.15.63:0/3326135823 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9448003770 msgr2=0x7f9448005bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:21:01.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.701+0000 7f9463cf0640 1 --2- 172.21.15.63:0/3326135823 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9448003770 0x7f9448005bd0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f944c003990 tx=0x7f944c01d080 comp rx=0 tx=0).stop 2024-04-26T04:21:01.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.701+0000 7f946228d640 1 -- 172.21.15.63:0/3326135823 reap_dead start 2024-04-26T04:21:01.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.701+0000 7f9463cf0640 1 -- 172.21.15.63:0/3326135823 shutdown_connections 2024-04-26T04:21:01.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.701+0000 7f9463cf0640 1 --2- 172.21.15.63:0/3326135823 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9448003770 0x7f9448005bd0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:21:01.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.701+0000 7f9463cf0640 1 -- 172.21.15.63:0/3326135823 >> 172.21.15.63:0/3326135823 conn(0x7f945c0a1e70 msgr2=0x7f945c0a2280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:21:01.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.701+0000 7f9463cf0640 1 -- 172.21.15.63:0/3326135823 shutdown_connections 2024-04-26T04:21:01.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.701+0000 7f9463cf0640 1 -- 172.21.15.63:0/3326135823 wait complete. 2024-04-26T04:21:01.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.701+0000 7f9463cf0640 10 monclient: build_initial_monmap 2024-04-26T04:21:01.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.701+0000 7f9463cf0640 10 monclient: monmap: 2024-04-26T04:21:01.709 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:21:01.709 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:21:01.709 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:21:01.709 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:21:01.709 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:21:01.709 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:21:01.709 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:21:01.709 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:21:01.709 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:21:01.709 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:21:01.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.701+0000 7f9463cf0640 1 Processor -- start 2024-04-26T04:21:01.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.701+0000 7f9463cf0640 1 -- start start 2024-04-26T04:21:01.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.701+0000 7f9463cf0640 10 monclient: init 2024-04-26T04:21:01.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.705+0000 7f9463cf0640 10 monclient: _reopen_session rank -1 2024-04-26T04:21:01.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.705+0000 7f9463cf0640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T04:21:01.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.705+0000 7f9463cf0640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9448003770 0x7f945c13a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:21:01.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.705+0000 7f9463cf0640 10 monclient(hunting): picked mon.noname-c con 0x7f9448003770 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:21:01.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.705+0000 7f9463cf0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f945c13ac60 0x7f945c1ce6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:21:01.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.705+0000 7f9463cf0640 10 monclient(hunting): picked mon.noname-b con 0x7f945c13ac60 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:21:01.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.705+0000 7f9463cf0640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f945c1cec10 0x7f945c1cf090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:21:01.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.705+0000 7f9463cf0640 10 monclient(hunting): picked mon.noname-a con 0x7f945c1cec10 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:21:01.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.705+0000 7f9463cf0640 10 monclient(hunting): start opening mon connection 2024-04-26T04:21:01.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.705+0000 7f9463cf0640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f945c09d030 con 0x7f945c1cec10 2024-04-26T04:21:01.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.705+0000 7f9463cf0640 10 monclient(hunting): start opening mon connection 2024-04-26T04:21:01.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.705+0000 7f9463cf0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f945c09ceb0 con 0x7f945c13ac60 2024-04-26T04:21:01.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.705+0000 7f9463cf0640 10 monclient(hunting): start opening mon connection 2024-04-26T04:21:01.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.705+0000 7f9463cf0640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f945c09d1b0 con 0x7f9448003770 2024-04-26T04:21:01.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.705+0000 7f9463cf0640 10 monclient(hunting): _renew_subs 2024-04-26T04:21:01.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.705+0000 7f9463cf0640 10 monclient(hunting): authenticate will time out at 1093.412578s 2024-04-26T04:21:01.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.705+0000 7f9461a8c640 1 -- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9448003770 msgr2=0x7f945c13a720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:3301/0 2024-04-26T04:21:01.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.705+0000 7f9461a8c640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9448003770 0x7f945c13a720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:21:01.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.705+0000 7f946228d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f945c1cec10 0x7f945c1cf090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:21:01.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.705+0000 7f946228d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f945c1cec10 0x7f945c1cf090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:56750/0 (socket says 172.21.15.63:56750) 2024-04-26T04:21:01.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.705+0000 7f946228d640 1 -- 172.21.15.63:0/241960967 learned_addr learned my addr 172.21.15.63:0/241960967 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:21:01.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.705+0000 7f946228d640 10 monclient(hunting): get_auth_request con 0x7f945c1cec10 auth_method 0 2024-04-26T04:21:01.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.705+0000 7f946228d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:21:01.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.705+0000 7f946228d640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:21:01.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.705+0000 7f946228d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:21:01.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.705+0000 7f946228d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:21:01.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.705+0000 7f946228d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:21:01.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.705+0000 7f946228d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:21:01.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.705+0000 7f946128b640 1 --2- 172.21.15.63:0/241960967 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f945c13ac60 0x7f945c1ce6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:21:01.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.705+0000 7f946128b640 10 monclient(hunting): get_auth_request con 0x7f945c13ac60 auth_method 0 2024-04-26T04:21:01.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.705+0000 7f946128b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:21:01.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.705+0000 7f946128b640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:21:01.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.705+0000 7f946128b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:21:01.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.705+0000 7f946228d640 10 monclient(hunting): handle_auth_done global_id 4842 payload 824 2024-04-26T04:21:01.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.705+0000 7f946228d640 1 -- 172.21.15.63:0/241960967 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9448003770 msgr2=0x7f945c13a720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T04:21:01.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.705+0000 7f946228d640 1 --2- 172.21.15.63:0/241960967 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9448003770 0x7f945c13a720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:21:01.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.705+0000 7f946228d640 1 -- 172.21.15.63:0/241960967 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f945c13ac60 msgr2=0x7f945c1ce6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:21:01.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.705+0000 7f946228d640 1 --2- 172.21.15.63:0/241960967 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f945c13ac60 0x7f945c1ce6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:21:01.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.705+0000 7f946228d640 10 monclient: _finish_hunting 0 2024-04-26T04:21:01.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.705+0000 7f946228d640 1 monclient: found mon.noname-a 2024-04-26T04:21:01.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.705+0000 7f946228d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:21:01.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.705+0000 7f946228d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:21:01.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.705+0000 7f946228d640 1 -- 172.21.15.63:0/241960967 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f945c0723e0 con 0x7f945c1cec10 2024-04-26T04:21:01.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.705+0000 7f946128b640 1 --2- 172.21.15.63:0/241960967 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f945c13ac60 0x7f945c1ce6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T04:21:01.714 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[29110]: starting. 2024-04-26T04:21:01.714 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[29110]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-29108 2024-04-26T04:21:01.714 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[29110]: shutting down. 2024-04-26T04:21:01.716 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.705+0000 7f946228d640 1 -- 172.21.15.63:0/241960967 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f945c1cec10 msgr2=0x7f945c1cf090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:21:01.716 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.705+0000 7f946228d640 1 --2- 172.21.15.63:0/241960967 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f945c1cec10 0x7f945c1cf090 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f945400e5f0 tx=0x7f945400eab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:21:01.716 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.709+0000 7f945a7fc640 1 -- 172.21.15.63:0/241960967 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94540109a0 con 0x7f945c1cec10 2024-04-26T04:21:01.716 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.709+0000 7f945a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:21:01.716 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.709+0000 7f945a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:21:01.716 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.709+0000 7f945a7fc640 10 monclient: dump: 2024-04-26T04:21:01.716 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:21:01.716 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:21:01.716 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:21:01.717 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:21:01.717 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:21:01.717 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:21:01.717 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:21:01.717 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:21:01.717 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:21:01.717 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:21:01.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.709+0000 7f945a7fc640 10 monclient: _finish_auth 0 2024-04-26T04:21:01.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.709+0000 7f945a7fc640 10 monclient: _check_auth_tickets 2024-04-26T04:21:01.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.709+0000 7f945a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:21:01.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.709+0000 7f9463cf0640 5 monclient: authenticate success, global_id 4842 2024-04-26T04:21:01.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.709+0000 7f9463cf0640 10 monclient: _renew_subs 2024-04-26T04:21:01.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.709+0000 7f9463cf0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:21:01.718 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.709+0000 7f9463cf0640 1 -- 172.21.15.63:0/241960967 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f945c0971e0 con 0x7f945c1cec10 2024-04-26T04:21:01.718 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.709+0000 7f9463cf0640 10 monclient: _renew_subs 2024-04-26T04:21:01.718 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.709+0000 7f9463cf0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:21:01.718 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.709+0000 7f9463cf0640 1 -- 172.21.15.63:0/241960967 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f945c096940 con 0x7f945c1cec10 2024-04-26T04:21:01.718 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.709+0000 7f945a7fc640 1 -- 172.21.15.63:0/241960967 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9454010b40 con 0x7f945c1cec10 2024-04-26T04:21:01.718 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.709+0000 7f945a7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T04:21:01.718 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.709+0000 7f945a7fc640 1 -- 172.21.15.63:0/241960967 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9454024430 con 0x7f945c1cec10 2024-04-26T04:21:01.718 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.709+0000 7f945a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:21:01.718 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.709+0000 7f945a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:21:01.718 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.709+0000 7f945a7fc640 10 monclient: dump: 2024-04-26T04:21:01.718 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:21:01.718 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:21:01.719 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:21:01.719 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:21:01.719 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:21:01.719 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:21:01.719 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:21:01.719 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:21:01.719 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:21:01.719 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:21:01.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.709+0000 7f945b7fe640 4 set_mon_vals no callback set 2024-04-26T04:21:01.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.709+0000 7f9463cf0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:21:01.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.709+0000 7f9463cf0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:21:01.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.709+0000 7f9463cf0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:21:01.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.709+0000 7f9463cf0640 1 -- 172.21.15.63:0/241960967 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f943c005090 con 0x7f945c1cec10 2024-04-26T04:21:01.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.709+0000 7f945a7fc640 1 -- 172.21.15.63:0/241960967 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f94540245d0 con 0x7f945c1cec10 2024-04-26T04:21:01.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.709+0000 7f945a7fc640 1 --2- 172.21.15.63:0/241960967 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f9448036e30 0x7f94480392f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:21:01.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.709+0000 7f945a7fc640 1 -- 172.21.15.63:0/241960967 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 4346+0+0 (secure 0 0 0) 0x7f9454049580 con 0x7f945c1cec10 2024-04-26T04:21:01.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.709+0000 7f9461a8c640 1 --2- 172.21.15.63:0/241960967 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f9448036e30 0x7f94480392f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:21:01.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.709+0000 7f9461a8c640 10 monclient: get_auth_request con 0x7f9448036e30 auth_method 0 2024-04-26T04:21:01.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.713+0000 7f9461a8c640 1 --2- 172.21.15.63:0/241960967 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f9448036e30 0x7f94480392f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9448006290 tx=0x7f944c0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:21:01.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.717+0000 7f945a7fc640 1 -- 172.21.15.63:0/241960967 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f945c0723e0 con 0x7f945c1cec10 2024-04-26T04:21:01.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.717+0000 7f945a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:21:01.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.717+0000 7f945a7fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:21:01.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.957+0000 7f9463cf0640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T04:21:01.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.957+0000 7f9463cf0640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T04:21:01.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.957+0000 7f9463cf0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:21:01.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.957+0000 7f9463cf0640 1 -- 172.21.15.63:0/241960967 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f943c005380 con 0x7f945c1cec10 2024-04-26T04:21:01.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.957+0000 7f945a7fc640 1 -- 172.21.15.63:0/241960967 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1554 (secure 0 0 0) 0x7f9454048130 con 0x7f945c1cec10 2024-04-26T04:21:01.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.957+0000 7f945a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T04:21:01.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.957+0000 7f945a7fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:21:01.960 INFO:teuthology.orchestra.run.smithi063.stdout:{"election_epoch":18,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cd1c6a0c-4e4b-4014-89c0-2e535479ef39","modified":"2024-04-26T04:18:03.702339Z","created":"2024-04-26T04:18:03.702339Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3300","nonce":0},{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:3300","nonce":0},{"type":"v1","addr":"172.21.15.186:6789","nonce":0}]},"addr":"172.21.15.186:6789/0","public_addr":"172.21.15.186:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3301","nonce":0},{"type":"v1","addr":"172.21.15.63:6790","nonce":0}]},"addr":"172.21.15.63:6790/0","public_addr":"172.21.15.63:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T04:21:01.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.961+0000 7f9463cf0640 1 -- 172.21.15.63:0/241960967 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f9448036e30 msgr2=0x7f94480392f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:21:01.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.961+0000 7f9463cf0640 1 --2- 172.21.15.63:0/241960967 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f9448036e30 0x7f94480392f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9448006290 tx=0x7f944c0a7000 comp rx=0 tx=0).stop 2024-04-26T04:21:01.962 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.961+0000 7f9463cf0640 10 monclient: shutdown 2024-04-26T04:21:01.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.961+0000 7f9463cf0640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:21:01.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.961+0000 7f9463cf0640 1 -- 172.21.15.63:0/241960967 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f945c1cec10 msgr2=0x7f945c1cf090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:21:01.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.961+0000 7f9463cf0640 1 --2- 172.21.15.63:0/241960967 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f945c1cec10 0x7f945c1cf090 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f945400e5f0 tx=0x7f945400eab0 comp rx=0 tx=0).stop 2024-04-26T04:21:01.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.961+0000 7f9463cf0640 1 -- 172.21.15.63:0/241960967 shutdown_connections 2024-04-26T04:21:01.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.961+0000 7f9463cf0640 1 --2- 172.21.15.63:0/241960967 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f9448036e30 0x7f94480392f0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:21:01.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.961+0000 7f9463cf0640 1 --2- 172.21.15.63:0/241960967 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f945c1cec10 0x7f945c1cf090 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:21:01.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.961+0000 7f9463cf0640 1 --2- 172.21.15.63:0/241960967 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f945c13ac60 0x7f945c1ce6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:21:01.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.961+0000 7f9463cf0640 1 --2- 172.21.15.63:0/241960967 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9448003770 0x7f945c13a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:21:01.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.961+0000 7f9463cf0640 1 -- 172.21.15.63:0/241960967 >> 172.21.15.63:0/241960967 conn(0x7f945c0a1e70 msgr2=0x7f945c099f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:21:01.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.961+0000 7f9463cf0640 1 -- 172.21.15.63:0/241960967 shutdown_connections 2024-04-26T04:21:01.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:01.961+0000 7f9463cf0640 1 -- 172.21.15.63:0/241960967 wait complete. 2024-04-26T04:21:01.979 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-04-26T04:21:01.980 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-04-26T04:21:01.980 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T04:21:02.098 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.097+0000 7f4fdff52640 10 monclient: get_monmap_and_config 2024-04-26T04:21:02.098 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.097+0000 7f4fdff52640 10 monclient: build_initial_monmap 2024-04-26T04:21:02.098 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.097+0000 7f4fdff52640 10 monclient: monmap: 2024-04-26T04:21:02.098 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:21:02.099 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:21:02.099 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:21:02.098465+0000 2024-04-26T04:21:02.099 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:21:02.098465+0000 2024-04-26T04:21:02.099 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:21:02.099 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:21:02.099 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:21:02.099 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:21:02.099 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:21:02.099 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:21:02.099 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:21:02.099 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:21:02.100 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.097+0000 7f4fdff52640 1 Processor -- start 2024-04-26T04:21:02.100 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.097+0000 7f4fdff52640 1 -- start start 2024-04-26T04:21:02.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.097+0000 7f4fdff52640 10 monclient: init 2024-04-26T04:21:02.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.097+0000 7f4fdff52640 10 monclient: _reopen_session rank -1 2024-04-26T04:21:02.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.097+0000 7f4fdff52640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-04-26T04:21:02.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.097+0000 7f4fdff52640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4fd809fe60 addr v1:172.21.15.63:6789/0 2024-04-26T04:21:02.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.097+0000 7f4fdff52640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4fd80a0240 0x7f4fd80a2a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:21:02.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.097+0000 7f4fdff52640 10 monclient(hunting): picked mon.noname-b con 0x7f4fd80a0240 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:21:02.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.097+0000 7f4fdff52640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4fd80a2f70 addr v1:172.21.15.186:6789/0 2024-04-26T04:21:02.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.097+0000 7f4fdff52640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4fd80a0c60 con 0x7f4fd809fe60 2024-04-26T04:21:02.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.097+0000 7f4fdff52640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4fd80a0f40 con 0x7f4fd80a2f70 2024-04-26T04:21:02.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.097+0000 7f4fdff52640 10 monclient(hunting): start opening mon connection 2024-04-26T04:21:02.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.097+0000 7f4fdff52640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f4fd80a3730 con 0x7f4fd80a0240 2024-04-26T04:21:02.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.097+0000 7f4fdff52640 10 monclient(hunting): _renew_subs 2024-04-26T04:21:02.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.097+0000 7f4fdff52640 10 monclient(hunting): authenticate will time out at 1093.806127s 2024-04-26T04:21:02.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.097+0000 7f4fdd4ed640 1 -- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4fd80a0240 msgr2=0x7f4fd80a2a30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.63:3301/0 2024-04-26T04:21:02.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.097+0000 7f4fdd4ed640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4fd80a0240 0x7f4fd80a2a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:21:02.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.097+0000 7f4fddcee640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f4fd809fe60 0x7f4fd81372f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:36988/0 (socket says 172.21.15.63:36988) 2024-04-26T04:21:02.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.097+0000 7f4fddcee640 1 -- 172.21.15.63:0/1229865905 learned_addr learned my addr 172.21.15.63:0/1229865905 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-04-26T04:21:02.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.101+0000 7f4fddcee640 1 -- 172.21.15.63:0/1229865905 >> v1:172.21.15.63:6789/0 conn(0x7f4fd809fe60 legacy=0x7f4fd81372f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:21:02.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.101+0000 7f4fdccec640 1 -- 172.21.15.63:0/1229865905 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1755963902 0 0) 0x7f4fd80a0c60 con 0x7f4fd809fe60 2024-04-26T04:21:02.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.101+0000 7f4fdccec640 10 monclient(hunting): _negotiate 2024-04-26T04:21:02.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.101+0000 7f4fdccec640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:21:02.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.101+0000 7f4fdccec640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:21:02.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.101+0000 7f4fdccec640 10 monclient(hunting): my global_id is 4857 2024-04-26T04:21:02.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.101+0000 7f4fdccec640 1 -- 172.21.15.63:0/1229865905 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4fc4003530 con 0x7f4fd809fe60 2024-04-26T04:21:02.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.101+0000 7f4fde4ef640 1 -- 172.21.15.63:0/1229865905 >> v1:172.21.15.186:6789/0 conn(0x7f4fd80a2f70 legacy=0x7f4fd813e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:21:02.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.101+0000 7f4fdccec640 1 -- 172.21.15.63:0/1229865905 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1331756024 0 0) 0x7f4fc4003530 con 0x7f4fd809fe60 2024-04-26T04:21:02.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.101+0000 7f4fdccec640 1 -- 172.21.15.63:0/1229865905 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4fd80a0240 msgr2=0x7f4fd80a2a30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T04:21:02.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.101+0000 7f4fdccec640 1 --2- 172.21.15.63:0/1229865905 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4fd80a0240 0x7f4fd80a2a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:21:02.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.101+0000 7f4fdccec640 1 -- 172.21.15.63:0/1229865905 >> v1:172.21.15.186:6789/0 conn(0x7f4fd80a2f70 legacy=0x7f4fd813e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:21:02.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.101+0000 7f4fdccec640 10 monclient: _finish_hunting 0 2024-04-26T04:21:02.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.101+0000 7f4fdccec640 1 monclient: found mon.noname-a-legacy 2024-04-26T04:21:02.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.101+0000 7f4fdccec640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:21:02.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.101+0000 7f4fdccec640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.63:6789/0 2024-04-26T04:21:02.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.101+0000 7f4fdccec640 1 -- 172.21.15.63:0/1229865905 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fd80a1220 con 0x7f4fd809fe60 2024-04-26T04:21:02.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.101+0000 7f4fdccec640 10 monclient: _finish_auth 0 2024-04-26T04:21:02.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.101+0000 7f4fdccec640 10 monclient: _check_auth_tickets 2024-04-26T04:21:02.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.101+0000 7f4fdccec640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:21:02.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.101+0000 7f4fdccec640 1 -- 172.21.15.63:0/1229865905 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2612289768 0 0) 0x7f4fd0001930 con 0x7f4fd809fe60 2024-04-26T04:21:02.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.101+0000 7f4fdccec640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:21:02.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.101+0000 7f4fdccec640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-26T04:21:02.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.101+0000 7f4fdccec640 10 monclient: dump: 2024-04-26T04:21:02.106 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:21:02.106 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:21:02.106 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:21:02.106 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:21:02.106 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:21:02.106 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:21:02.106 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:21:02.107 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:21:02.107 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:21:02.107 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:21:02.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.101+0000 7f4fdccec640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] but i'm connected to v1:172.21.15.63:6789/0, reconnecting 2024-04-26T04:21:02.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.101+0000 7f4fdccec640 10 monclient: _reopen_session rank -1 2024-04-26T04:21:02.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.101+0000 7f4fdccec640 1 -- 172.21.15.63:0/1229865905 >> v1:172.21.15.63:6789/0 conn(0x7f4fd809fe60 legacy=0x7f4fd81372f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:21:02.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.101+0000 7f4fdccec640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T04:21:02.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.101+0000 7f4fdccec640 1 --2- 172.21.15.63:0/1229865905 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4fc4003ad0 0x7f4fc4003ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:21:02.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.101+0000 7f4fdccec640 10 monclient(hunting): picked mon.a con 0x7f4fc4003ad0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:21:02.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.101+0000 7f4fdccec640 1 --2- 172.21.15.63:0/1229865905 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4fd80a0240 0x7f4fc4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:21:02.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.101+0000 7f4fdccec640 10 monclient(hunting): picked mon.c con 0x7f4fd80a0240 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:21:02.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.101+0000 7f4fdccec640 1 --2- 172.21.15.63:0/1229865905 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4fc4004f70 0x7f4fc400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:21:02.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.101+0000 7f4fdccec640 10 monclient(hunting): picked mon.b con 0x7f4fc4004f70 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:21:02.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.101+0000 7f4fdccec640 10 monclient(hunting): start opening mon connection 2024-04-26T04:21:02.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.101+0000 7f4fdccec640 1 -- 172.21.15.63:0/1229865905 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd80a3730 con 0x7f4fc4003ad0 2024-04-26T04:21:02.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.101+0000 7f4fdccec640 10 monclient(hunting): start opening mon connection 2024-04-26T04:21:02.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.101+0000 7f4fdccec640 1 -- 172.21.15.63:0/1229865905 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4fc400bf10 con 0x7f4fc4004f70 2024-04-26T04:21:02.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.101+0000 7f4fdccec640 10 monclient(hunting): start opening mon connection 2024-04-26T04:21:02.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.101+0000 7f4fdccec640 1 -- 172.21.15.63:0/1229865905 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f4fc400c090 con 0x7f4fd80a0240 2024-04-26T04:21:02.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.101+0000 7f4fdccec640 10 monclient(hunting): _renew_subs 2024-04-26T04:21:02.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.101+0000 7f4fdccec640 10 monclient(hunting): _finish_auth 0 2024-04-26T04:21:02.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.101+0000 7f4fdd4ed640 1 -- 172.21.15.63:0/1229865905 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4fd80a0240 msgr2=0x7f4fc4004810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.63:3301/0 2024-04-26T04:21:02.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.101+0000 7f4fdd4ed640 1 --2- 172.21.15.63:0/1229865905 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4fd80a0240 0x7f4fc4004810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:21:02.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.101+0000 7f4fddcee640 1 --2- 172.21.15.63:0/1229865905 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4fc4003ad0 0x7f4fc4003ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:21:02.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.101+0000 7f4fddcee640 10 monclient(hunting): get_auth_request con 0x7f4fc4003ad0 auth_method 0 2024-04-26T04:21:02.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.101+0000 7f4fddcee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:21:02.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.101+0000 7f4fddcee640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:21:02.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.101+0000 7f4fddcee640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:21:02.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.101+0000 7f4fddcee640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:21:02.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.101+0000 7f4fddcee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:21:02.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.101+0000 7f4fddcee640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T04:21:02.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.101+0000 7f4fde4ef640 1 --2- 172.21.15.63:0/1229865905 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4fc4004f70 0x7f4fc400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:21:02.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.101+0000 7f4fddcee640 10 monclient(hunting): handle_auth_done global_id 4857 payload 293 2024-04-26T04:21:02.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.101+0000 7f4fddcee640 1 -- 172.21.15.63:0/1229865905 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4fd80a0240 msgr2=0x7f4fc4004810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T04:21:02.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.101+0000 7f4fddcee640 1 --2- 172.21.15.63:0/1229865905 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4fd80a0240 0x7f4fc4004810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:21:02.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.101+0000 7f4fddcee640 1 -- 172.21.15.63:0/1229865905 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4fc4004f70 msgr2=0x7f4fc400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:21:02.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.101+0000 7f4fddcee640 1 --2- 172.21.15.63:0/1229865905 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4fc4004f70 0x7f4fc400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:21:02.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.101+0000 7f4fddcee640 10 monclient: _finish_hunting 0 2024-04-26T04:21:02.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.101+0000 7f4fddcee640 1 monclient: found mon.a 2024-04-26T04:21:02.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.101+0000 7f4fddcee640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:21:02.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.101+0000 7f4fddcee640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:21:02.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.101+0000 7f4fddcee640 1 -- 172.21.15.63:0/1229865905 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fc4003530 con 0x7f4fc4003ad0 2024-04-26T04:21:02.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.101+0000 7f4fddcee640 10 monclient: _finish_auth 0 2024-04-26T04:21:02.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.101+0000 7f4fddcee640 10 monclient: _check_auth_tickets 2024-04-26T04:21:02.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.101+0000 7f4fddcee640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:21:02.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.101+0000 7f4fdff52640 5 monclient: authenticate success, global_id 4857 2024-04-26T04:21:02.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.101+0000 7f4fdff52640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T04:21:02.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.101+0000 7f4fde4ef640 10 monclient: get_auth_request con 0x7f4fc4004f70 auth_method 0 2024-04-26T04:21:02.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.101+0000 7f4fde4ef640 1 --2- 172.21.15.63:0/1229865905 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4fc4004f70 0x7f4fc400b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T04:21:02.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.105+0000 7f4fddcee640 1 -- 172.21.15.63:0/1229865905 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4fc4003ad0 msgr2=0x7f4fc4003ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:21:02.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.105+0000 7f4fddcee640 1 --2- 172.21.15.63:0/1229865905 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4fc4003ad0 0x7f4fc4003ef0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f4fd00014a0 tx=0x7f4fd001cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fcaec480791c8b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:21:02.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.105+0000 7f4fdccec640 1 -- 172.21.15.63:0/1229865905 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fd000c0e0 con 0x7f4fc4003ad0 2024-04-26T04:21:02.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.105+0000 7f4fdccec640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:21:02.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.105+0000 7f4fdccec640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:21:02.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.105+0000 7f4fdccec640 10 monclient: dump: 2024-04-26T04:21:02.115 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:21:02.115 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:21:02.116 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:21:02.116 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:21:02.116 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:21:02.116 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:21:02.116 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:21:02.116 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:21:02.116 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:21:02.116 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:21:02.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.105+0000 7f4fdff52640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T04:21:02.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.105+0000 7f4fdccec640 1 -- 172.21.15.63:0/1229865905 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fd00a9070 con 0x7f4fc4003ad0 2024-04-26T04:21:02.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.105+0000 7f4fdccec640 10 monclient: handle_config config(0 keys) 2024-04-26T04:21:02.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.105+0000 7f4fdccec640 1 -- 172.21.15.63:0/1229865905 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fd00a5d50 con 0x7f4fc4003ad0 2024-04-26T04:21:02.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.105+0000 7f4fdccec640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:21:02.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.105+0000 7f4fdccec640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:21:02.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.105+0000 7f4fdccec640 10 monclient: dump: 2024-04-26T04:21:02.117 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:21:02.117 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:21:02.117 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:21:02.117 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:21:02.117 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:21:02.117 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:21:02.117 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:21:02.117 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:21:02.117 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:21:02.117 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:21:02.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.105+0000 7f4fdff52640 10 monclient: get_monmap_and_config success 2024-04-26T04:21:02.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.105+0000 7f4fdff52640 4 set_mon_vals no callback set 2024-04-26T04:21:02.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.105+0000 7f4fdff52640 10 monclient: shutdown 2024-04-26T04:21:02.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.105+0000 7f4fdff52640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:21:02.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.105+0000 7f4fdff52640 1 -- 172.21.15.63:0/1229865905 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4fc4003ad0 msgr2=0x7f4fc4003ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:21:02.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.105+0000 7f4fdff52640 1 --2- 172.21.15.63:0/1229865905 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4fc4003ad0 0x7f4fc4003ef0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f4fd00014a0 tx=0x7f4fd001cd30 comp rx=0 tx=0).stop 2024-04-26T04:21:02.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.105+0000 7f4fde4ef640 1 -- 172.21.15.63:0/1229865905 reap_dead start 2024-04-26T04:21:02.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.105+0000 7f4fdff52640 1 -- 172.21.15.63:0/1229865905 shutdown_connections 2024-04-26T04:21:02.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.105+0000 7f4fdff52640 1 -- 172.21.15.63:0/1229865905 >> 172.21.15.63:0/1229865905 conn(0x7f4fd80a5fb0 msgr2=0x7f4fd80a63c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:21:02.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.105+0000 7f4fdff52640 1 -- 172.21.15.63:0/1229865905 shutdown_connections 2024-04-26T04:21:02.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.105+0000 7f4fdff52640 1 -- 172.21.15.63:0/1229865905 wait complete. 2024-04-26T04:21:02.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.105+0000 7f4fdff52640 10 monclient: build_initial_monmap 2024-04-26T04:21:02.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.105+0000 7f4fdff52640 10 monclient: monmap: 2024-04-26T04:21:02.118 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:21:02.118 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:21:02.118 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:21:02.118 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:21:02.118 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:21:02.118 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:21:02.118 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:21:02.119 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:21:02.119 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:21:02.119 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:21:02.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.105+0000 7f4fdff52640 1 Processor -- start 2024-04-26T04:21:02.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.105+0000 7f4fdff52640 1 -- start start 2024-04-26T04:21:02.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.105+0000 7f4fdff52640 10 monclient: init 2024-04-26T04:21:02.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.109+0000 7f4fdff52640 10 monclient: _reopen_session rank -1 2024-04-26T04:21:02.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.109+0000 7f4fdff52640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T04:21:02.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.109+0000 7f4fdff52640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4fc4003ad0 0x7f4fd81cec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:21:02.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.109+0000 7f4fdff52640 10 monclient(hunting): picked mon.noname-b con 0x7f4fc4003ad0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:21:02.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.109+0000 7f4fdff52640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4fd81cf180 0x7f4fd8072930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:21:02.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.109+0000 7f4fdff52640 10 monclient(hunting): picked mon.noname-c con 0x7f4fd81cf180 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:21:02.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.109+0000 7f4fdff52640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4fd8072e70 0x7f4fd813d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:21:02.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.109+0000 7f4fdff52640 10 monclient(hunting): picked mon.noname-a con 0x7f4fd8072e70 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:21:02.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.109+0000 7f4fdff52640 10 monclient(hunting): start opening mon connection 2024-04-26T04:21:02.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.109+0000 7f4fdff52640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd81404d0 con 0x7f4fd8072e70 2024-04-26T04:21:02.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.109+0000 7f4fdff52640 10 monclient(hunting): start opening mon connection 2024-04-26T04:21:02.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.109+0000 7f4fdff52640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd8140350 con 0x7f4fc4003ad0 2024-04-26T04:21:02.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.109+0000 7f4fdff52640 10 monclient(hunting): start opening mon connection 2024-04-26T04:21:02.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.109+0000 7f4fdff52640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f4fd8140650 con 0x7f4fd81cf180 2024-04-26T04:21:02.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.109+0000 7f4fdff52640 10 monclient(hunting): _renew_subs 2024-04-26T04:21:02.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.109+0000 7f4fdff52640 10 monclient(hunting): authenticate will time out at 1093.816304s 2024-04-26T04:21:02.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.109+0000 7f4fdd4ed640 1 -- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4fd81cf180 msgr2=0x7f4fd8072930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:3301/0 2024-04-26T04:21:02.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.109+0000 7f4fdd4ed640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4fd81cf180 0x7f4fd8072930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:21:02.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.109+0000 7f4fde4ef640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4fd8072e70 0x7f4fd813d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:21:02.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.109+0000 7f4fde4ef640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4fd8072e70 0x7f4fd813d2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:56758/0 (socket says 172.21.15.63:56758) 2024-04-26T04:21:02.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.109+0000 7f4fde4ef640 1 -- 172.21.15.63:0/2346238807 learned_addr learned my addr 172.21.15.63:0/2346238807 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:21:02.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.109+0000 7f4fde4ef640 10 monclient(hunting): get_auth_request con 0x7f4fd8072e70 auth_method 0 2024-04-26T04:21:02.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.109+0000 7f4fde4ef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:21:02.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.109+0000 7f4fde4ef640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:21:02.121 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.109+0000 7f4fde4ef640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:21:02.121 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.109+0000 7f4fde4ef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:21:02.121 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.109+0000 7f4fde4ef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:21:02.121 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.109+0000 7f4fde4ef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:21:02.121 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.109+0000 7f4fddcee640 1 --2- 172.21.15.63:0/2346238807 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4fc4003ad0 0x7f4fd81cec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:21:02.121 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.109+0000 7f4fddcee640 10 monclient(hunting): get_auth_request con 0x7f4fc4003ad0 auth_method 0 2024-04-26T04:21:02.121 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.109+0000 7f4fddcee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:21:02.121 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.109+0000 7f4fddcee640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:21:02.121 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.109+0000 7f4fddcee640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:21:02.121 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.109+0000 7f4fde4ef640 10 monclient(hunting): handle_auth_done global_id 4860 payload 824 2024-04-26T04:21:02.121 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.109+0000 7f4fde4ef640 1 -- 172.21.15.63:0/2346238807 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4fd81cf180 msgr2=0x7f4fd8072930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T04:21:02.121 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.109+0000 7f4fde4ef640 1 --2- 172.21.15.63:0/2346238807 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4fd81cf180 0x7f4fd8072930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:21:02.121 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.109+0000 7f4fde4ef640 1 -- 172.21.15.63:0/2346238807 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4fc4003ad0 msgr2=0x7f4fd81cec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:21:02.121 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.109+0000 7f4fde4ef640 1 --2- 172.21.15.63:0/2346238807 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4fc4003ad0 0x7f4fd81cec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:21:02.121 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.109+0000 7f4fde4ef640 10 monclient: _finish_hunting 0 2024-04-26T04:21:02.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.109+0000 7f4fde4ef640 1 monclient: found mon.noname-a 2024-04-26T04:21:02.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.109+0000 7f4fde4ef640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:21:02.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.109+0000 7f4fde4ef640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:21:02.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.109+0000 7f4fde4ef640 1 -- 172.21.15.63:0/2346238807 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fd80a38f0 con 0x7f4fd8072e70 2024-04-26T04:21:02.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.109+0000 7f4fde4ef640 1 -- 172.21.15.63:0/2346238807 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4fd8072e70 msgr2=0x7f4fd813d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:21:02.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.109+0000 7f4fde4ef640 1 --2- 172.21.15.63:0/2346238807 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4fd8072e70 0x7f4fd813d2f0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f4fd80a3350 tx=0x7f4fd400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:21:02.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.109+0000 7f4fce7fc640 1 -- 172.21.15.63:0/2346238807 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fd4018070 con 0x7f4fd8072e70 2024-04-26T04:21:02.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.109+0000 7f4fce7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:21:02.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.109+0000 7f4fce7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:21:02.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.109+0000 7f4fce7fc640 10 monclient: dump: 2024-04-26T04:21:02.122 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:21:02.122 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:21:02.122 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:21:02.122 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:21:02.122 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:21:02.123 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:21:02.123 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:21:02.123 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:21:02.123 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:21:02.123 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:21:02.123 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.109+0000 7f4fce7fc640 10 monclient: _finish_auth 0 2024-04-26T04:21:02.123 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.109+0000 7f4fce7fc640 10 monclient: _check_auth_tickets 2024-04-26T04:21:02.123 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.109+0000 7f4fce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:21:02.123 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.109+0000 7f4fce7fc640 1 -- 172.21.15.63:0/2346238807 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fd400b620 con 0x7f4fd8072e70 2024-04-26T04:21:02.123 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.109+0000 7f4fce7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T04:21:02.123 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.109+0000 7f4fce7fc640 1 -- 172.21.15.63:0/2346238807 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fd400b9d0 con 0x7f4fd8072e70 2024-04-26T04:21:02.123 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.109+0000 7f4fdff52640 5 monclient: authenticate success, global_id 4860 2024-04-26T04:21:02.123 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.109+0000 7f4fce7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:21:02.123 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.109+0000 7f4fce7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:21:02.123 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.109+0000 7f4fce7fc640 10 monclient: dump: 2024-04-26T04:21:02.124 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:21:02.124 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:21:02.124 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:21:02.124 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:21:02.124 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:21:02.124 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:21:02.124 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:21:02.124 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:21:02.124 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:21:02.124 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:21:02.124 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.109+0000 7f4fcffff640 4 set_mon_vals no callback set 2024-04-26T04:21:02.124 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.109+0000 7f4fdff52640 10 monclient: _renew_subs 2024-04-26T04:21:02.124 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.109+0000 7f4fdff52640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:21:02.124 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.109+0000 7f4fdff52640 1 -- 172.21.15.63:0/2346238807 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fd806fb50 con 0x7f4fd8072e70 2024-04-26T04:21:02.124 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.109+0000 7f4fdff52640 10 monclient: _renew_subs 2024-04-26T04:21:02.125 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.109+0000 7f4fdff52640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:21:02.125 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.109+0000 7f4fdff52640 1 -- 172.21.15.63:0/2346238807 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fd80732f0 con 0x7f4fd8072e70 2024-04-26T04:21:02.125 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.113+0000 7f4fdff52640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:21:02.125 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.113+0000 7f4fdff52640 1 --2- 172.21.15.63:0/2346238807 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4fd813d830 0x7f4fd813dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:21:02.125 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.113+0000 7f4fdff52640 10 monclient(hunting): start opening mon connection 2024-04-26T04:21:02.125 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.113+0000 7f4fdff52640 1 -- 172.21.15.63:0/2346238807 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd80a7f40 con 0x7f4fd813d830 2024-04-26T04:21:02.125 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.113+0000 7f4fddcee640 1 --2- 172.21.15.63:0/2346238807 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4fd813d830 0x7f4fd813dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:21:02.125 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.113+0000 7f4fce7fc640 1 -- 172.21.15.63:0/2346238807 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f4fd401b070 con 0x7f4fd8072e70 2024-04-26T04:21:02.125 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.113+0000 7f4fddcee640 10 monclient: get_auth_request con 0x7f4fd813d830 auth_method 0 2024-04-26T04:21:02.125 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.113+0000 7f4fddcee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:21:02.125 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.113+0000 7f4fddcee640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:21:02.125 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.113+0000 7f4fddcee640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:21:02.125 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.113+0000 7f4fddcee640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:21:02.125 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.113+0000 7f4fddcee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:21:02.125 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.113+0000 7f4fddcee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:21:02.126 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.113+0000 7f4fce7fc640 1 --2- 172.21.15.63:0/2346238807 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f4fbc035ec0 0x7f4fbc038380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:21:02.126 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.113+0000 7f4fddcee640 10 monclient(hunting): handle_auth_done global_id 4863 payload 824 2024-04-26T04:21:02.126 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.113+0000 7f4fce7fc640 1 -- 172.21.15.63:0/2346238807 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 4346+0+0 (secure 0 0 0) 0x7f4fd40482f0 con 0x7f4fd8072e70 2024-04-26T04:21:02.126 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.113+0000 7f4fddcee640 1 -- 172.21.15.63:0/2346238807 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4fd813e150 con 0x7f4fd813d830 2024-04-26T04:21:02.126 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.113+0000 7f4fdd4ed640 1 --2- 172.21.15.63:0/2346238807 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f4fbc035ec0 0x7f4fbc038380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:21:02.126 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.113+0000 7f4fdd4ed640 10 monclient: get_auth_request con 0x7f4fbc035ec0 auth_method 0 2024-04-26T04:21:02.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.113+0000 7f4fddcee640 1 -- 172.21.15.63:0/2346238807 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4fd813d830 msgr2=0x7f4fd813dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:21:02.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.113+0000 7f4fddcee640 1 --2- 172.21.15.63:0/2346238807 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4fd813d830 0x7f4fd813dc10 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f4fd00ab000 tx=0x7f4fc4005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:21:02.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.113+0000 7f4fce7fc640 1 -- 172.21.15.63:0/2346238807 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fd00047f0 con 0x7f4fd813d830 2024-04-26T04:21:02.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.113+0000 7f4fce7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:21:02.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.113+0000 7f4fce7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:21:02.129 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.113+0000 7f4fce7fc640 10 monclient: dump: 2024-04-26T04:21:02.129 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:21:02.129 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:21:02.129 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:21:02.129 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:21:02.129 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:21:02.129 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:21:02.129 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:21:02.129 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:21:02.129 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:21:02.129 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:21:02.129 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.113+0000 7f4fce7fc640 1 -- 172.21.15.63:0/2346238807 <== mon.0 v2:172.21.15.63:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4fd00a9030 con 0x7f4fd813d830 2024-04-26T04:21:02.129 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.113+0000 7f4fce7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:21:02.129 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.113+0000 7f4fce7fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:21:02.129 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.113+0000 7f4fce7fc640 1 -- 172.21.15.63:0/2346238807 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4fd813d830 msgr2=0x7f4fd813dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:21:02.130 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.113+0000 7f4fce7fc640 1 --2- 172.21.15.63:0/2346238807 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4fd813d830 0x7f4fd813dc10 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f4fd00ab000 tx=0x7f4fc4005840 comp rx=0 tx=0).stop 2024-04-26T04:21:02.130 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.113+0000 7f4fce7fc640 1 -- 172.21.15.63:0/2346238807 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4fd813d830 msgr2=0x7f4fd813dc10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:21:02.130 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.113+0000 7f4fce7fc640 1 --2- 172.21.15.63:0/2346238807 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4fd813d830 0x7f4fd813dc10 secure :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f4fd00ab000 tx=0x7f4fc4005840 comp rx=0 tx=0).stop 2024-04-26T04:21:02.130 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.117+0000 7f4fdd4ed640 1 --2- 172.21.15.63:0/2346238807 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f4fbc035ec0 0x7f4fbc038380 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f4fd806ee30 tx=0x7f4fc8006d70 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:21:02.130 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.121+0000 7f4fdff52640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T04:21:02.130 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.121+0000 7f4fdff52640 1 --2- 172.21.15.63:0/2346238807 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4fd81384b0 0x7f4fd81388c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:21:02.130 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.121+0000 7f4fdff52640 10 monclient(hunting): start opening mon connection 2024-04-26T04:21:02.130 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.121+0000 7f4fdff52640 1 -- 172.21.15.63:0/2346238807 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f4fd8062e10 con 0x7f4fd81384b0 2024-04-26T04:21:02.130 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.121+0000 7f4fddcee640 1 --2- 172.21.15.63:0/2346238807 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4fd81384b0 0x7f4fd81388c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:21:02.130 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.121+0000 7f4fddcee640 10 monclient: get_auth_request con 0x7f4fd81384b0 auth_method 0 2024-04-26T04:21:02.130 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.121+0000 7f4fddcee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:21:02.130 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.121+0000 7f4fddcee640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:21:02.130 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.121+0000 7f4fddcee640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:21:02.130 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.121+0000 7f4fddcee640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:21:02.130 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.121+0000 7f4fddcee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:21:02.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.121+0000 7f4fddcee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:21:02.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.121+0000 7f4fddcee640 10 monclient(hunting): handle_auth_done global_id 4866 payload 824 2024-04-26T04:21:02.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.121+0000 7f4fddcee640 1 -- 172.21.15.63:0/2346238807 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4fd806fd20 con 0x7f4fd81384b0 2024-04-26T04:21:02.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.125+0000 7f4fddcee640 1 -- 172.21.15.63:0/2346238807 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4fd81384b0 msgr2=0x7f4fd81388c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:21:02.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.125+0000 7f4fddcee640 1 --2- 172.21.15.63:0/2346238807 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4fd81384b0 0x7f4fd81388c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f4fd001cd70 tx=0x7f4fd00a7000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:21:02.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.125+0000 7f4fce7fc640 1 -- 172.21.15.63:0/2346238807 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fd00072b0 con 0x7f4fd81384b0 2024-04-26T04:21:02.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.125+0000 7f4fce7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:21:02.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.125+0000 7f4fce7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:21:02.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.125+0000 7f4fce7fc640 10 monclient: dump: 2024-04-26T04:21:02.131 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:21:02.131 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:21:02.131 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:21:02.131 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:21:02.131 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:21:02.131 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:21:02.132 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:21:02.132 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:21:02.132 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:21:02.132 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:21:02.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.125+0000 7f4fce7fc640 1 -- 172.21.15.63:0/2346238807 <== mon.0 v2:172.21.15.63:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4712 (secure 0 0 0) 0x7f4fd00072b0 con 0x7f4fd81384b0 2024-04-26T04:21:02.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.125+0000 7f4fce7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T04:21:02.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.125+0000 7f4fce7fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:21:02.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.125+0000 7f4fce7fc640 1 -- 172.21.15.63:0/2346238807 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4fd81384b0 msgr2=0x7f4fd81388c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:21:02.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.125+0000 7f4fce7fc640 1 --2- 172.21.15.63:0/2346238807 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4fd81384b0 0x7f4fd81388c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f4fd001cd70 tx=0x7f4fd00a7000 comp rx=0 tx=0).stop 2024-04-26T04:21:02.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.125+0000 7f4fce7fc640 1 -- 172.21.15.63:0/2346238807 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4fd81384b0 msgr2=0x7f4fd81388c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:21:02.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.125+0000 7f4fce7fc640 1 --2- 172.21.15.63:0/2346238807 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4fd81384b0 0x7f4fd81388c0 secure :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f4fd001cd70 tx=0x7f4fd00a7000 comp rx=0 tx=0).stop 2024-04-26T04:21:02.132 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-26T04:21:02.133 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a", 2024-04-26T04:21:02.133 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 0, 2024-04-26T04:21:02.133 INFO:teuthology.orchestra.run.smithi063.stdout: "state": "leader", 2024-04-26T04:21:02.133 INFO:teuthology.orchestra.run.smithi063.stdout: "election_epoch": 18, 2024-04-26T04:21:02.133 INFO:teuthology.orchestra.run.smithi063.stdout: "uptime": 153278, 2024-04-26T04:21:02.133 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum": [ 2024-04-26T04:21:02.133 INFO:teuthology.orchestra.run.smithi063.stdout: 0, 2024-04-26T04:21:02.133 INFO:teuthology.orchestra.run.smithi063.stdout: 1 2024-04-26T04:21:02.133 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:21:02.133 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_age": 0, 2024-04-26T04:21:02.133 INFO:teuthology.orchestra.run.smithi063.stdout: "features": { 2024-04-26T04:21:02.133 INFO:teuthology.orchestra.run.smithi063.stdout: "required_con": "2450521708007866372", 2024-04-26T04:21:02.133 INFO:teuthology.orchestra.run.smithi063.stdout: "required_mon": [ 2024-04-26T04:21:02.133 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:21:02.134 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:21:02.134 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:21:02.134 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:21:02.134 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:21:02.134 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:21:02.134 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:21:02.134 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:21:02.134 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:21:02.134 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:21:02.134 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:21:02.134 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:21:02.134 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_con": "4540701547738038271", 2024-04-26T04:21:02.134 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_mon": [ 2024-04-26T04:21:02.134 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:21:02.135 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:21:02.135 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:21:02.135 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:21:02.135 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:21:02.135 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:21:02.135 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:21:02.135 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:21:02.135 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:21:02.135 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:21:02.135 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:21:02.135 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:21:02.135 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:21:02.135 INFO:teuthology.orchestra.run.smithi063.stdout: "outside_quorum": [], 2024-04-26T04:21:02.135 INFO:teuthology.orchestra.run.smithi063.stdout: "extra_probe_peers": [], 2024-04-26T04:21:02.135 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_provider": [], 2024-04-26T04:21:02.136 INFO:teuthology.orchestra.run.smithi063.stdout: "monmap": { 2024-04-26T04:21:02.136 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 1, 2024-04-26T04:21:02.136 INFO:teuthology.orchestra.run.smithi063.stdout: "fsid": "cd1c6a0c-4e4b-4014-89c0-2e535479ef39", 2024-04-26T04:21:02.136 INFO:teuthology.orchestra.run.smithi063.stdout: "modified": "2024-04-26T04:18:03.702339Z", 2024-04-26T04:21:02.136 INFO:teuthology.orchestra.run.smithi063.stdout: "created": "2024-04-26T04:18:03.702339Z", 2024-04-26T04:21:02.136 INFO:teuthology.orchestra.run.smithi063.stdout: "min_mon_release": 19, 2024-04-26T04:21:02.136 INFO:teuthology.orchestra.run.smithi063.stdout: "min_mon_release_name": "squid", 2024-04-26T04:21:02.136 INFO:teuthology.orchestra.run.smithi063.stdout: "election_strategy": 1, 2024-04-26T04:21:02.136 INFO:teuthology.orchestra.run.smithi063.stdout: "disallowed_leaders: ": "", 2024-04-26T04:21:02.136 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": false, 2024-04-26T04:21:02.136 INFO:teuthology.orchestra.run.smithi063.stdout: "tiebreaker_mon": "", 2024-04-26T04:21:02.136 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_ranks: ": "", 2024-04-26T04:21:02.136 INFO:teuthology.orchestra.run.smithi063.stdout: "features": { 2024-04-26T04:21:02.136 INFO:teuthology.orchestra.run.smithi063.stdout: "persistent": [ 2024-04-26T04:21:02.136 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:21:02.137 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:21:02.137 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:21:02.137 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:21:02.137 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:21:02.137 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:21:02.137 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:21:02.137 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:21:02.137 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:21:02.137 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:21:02.137 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:21:02.137 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:21:02.137 INFO:teuthology.orchestra.run.smithi063.stdout: "optional": [] 2024-04-26T04:21:02.137 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:21:02.137 INFO:teuthology.orchestra.run.smithi063.stdout: "mons": [ 2024-04-26T04:21:02.137 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:21:02.138 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 0, 2024-04-26T04:21:02.138 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a", 2024-04-26T04:21:02.138 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:21:02.138 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:21:02.138 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:21:02.138 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:21:02.138 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:3300", 2024-04-26T04:21:02.138 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:21:02.138 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:21:02.138 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:21:02.138 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:21:02.138 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6789", 2024-04-26T04:21:02.138 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:21:02.138 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:21:02.138 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:21:02.139 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:21:02.139 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6789/0", 2024-04-26T04:21:02.139 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6789/0", 2024-04-26T04:21:02.139 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:21:02.139 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:21:02.139 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:21:02.139 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:21:02.139 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:21:02.139 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 1, 2024-04-26T04:21:02.139 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b", 2024-04-26T04:21:02.139 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:21:02.139 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:21:02.139 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:21:02.139 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:21:02.139 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:3300", 2024-04-26T04:21:02.140 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:21:02.140 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:21:02.140 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:21:02.140 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:21:02.140 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:6789", 2024-04-26T04:21:02.140 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:21:02.140 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:21:02.140 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:21:02.140 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:21:02.140 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:6789/0", 2024-04-26T04:21:02.140 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.186:6789/0", 2024-04-26T04:21:02.140 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:21:02.140 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:21:02.140 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:21:02.140 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:21:02.141 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:21:02.141 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 2, 2024-04-26T04:21:02.141 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "c", 2024-04-26T04:21:02.141 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:21:02.141 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:21:02.141 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:21:02.141 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:21:02.141 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:3301", 2024-04-26T04:21:02.141 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:21:02.141 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:21:02.141 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:21:02.141 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:21:02.141 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6790", 2024-04-26T04:21:02.141 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:21:02.141 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:21:02.142 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:21:02.142 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:21:02.142 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6790/0", 2024-04-26T04:21:02.142 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6790/0", 2024-04-26T04:21:02.142 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:21:02.142 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:21:02.142 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:21:02.142 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:21:02.142 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:21:02.142 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:21:02.142 INFO:teuthology.orchestra.run.smithi063.stdout: "feature_map": { 2024-04-26T04:21:02.142 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": [ 2024-04-26T04:21:02.142 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:21:02.142 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:21:02.142 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:21:02.143 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 1 2024-04-26T04:21:02.143 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:21:02.143 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:21:02.143 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": [ 2024-04-26T04:21:02.143 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:21:02.143 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:21:02.143 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:21:02.143 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 4 2024-04-26T04:21:02.143 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:21:02.143 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:21:02.143 INFO:teuthology.orchestra.run.smithi063.stdout: "client": [ 2024-04-26T04:21:02.143 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:21:02.143 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:21:02.143 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:21:02.143 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 4 2024-04-26T04:21:02.144 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:21:02.144 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:21:02.144 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:21:02.144 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": false 2024-04-26T04:21:02.144 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-26T04:21:02.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.125+0000 7f4fdff52640 1 -- 172.21.15.63:0/2346238807 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f4fbc035ec0 msgr2=0x7f4fbc038380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:21:02.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.125+0000 7f4fdff52640 1 --2- 172.21.15.63:0/2346238807 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f4fbc035ec0 0x7f4fbc038380 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f4fd806ee30 tx=0x7f4fc8006d70 comp rx=0 tx=0).stop 2024-04-26T04:21:02.145 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.125+0000 7f4fdff52640 10 monclient: shutdown 2024-04-26T04:21:02.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.125+0000 7f4fdff52640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:21:02.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.125+0000 7f4fdff52640 1 -- 172.21.15.63:0/2346238807 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4fd8072e70 msgr2=0x7f4fd813d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:21:02.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.125+0000 7f4fdff52640 1 --2- 172.21.15.63:0/2346238807 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4fd8072e70 0x7f4fd813d2f0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f4fd80a3350 tx=0x7f4fd400dc00 comp rx=0 tx=0).stop 2024-04-26T04:21:02.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.125+0000 7f4fde4ef640 1 -- 172.21.15.63:0/2346238807 reap_dead start 2024-04-26T04:21:02.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.125+0000 7f4fde4ef640 1 -- 172.21.15.63:0/2346238807 reap_dead start 2024-04-26T04:21:02.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.125+0000 7f4fdff52640 1 -- 172.21.15.63:0/2346238807 shutdown_connections 2024-04-26T04:21:02.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.125+0000 7f4fdff52640 1 -- 172.21.15.63:0/2346238807 >> 172.21.15.63:0/2346238807 conn(0x7f4fd80a5fb0 msgr2=0x7f4fd80a3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:21:02.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.125+0000 7f4fdff52640 1 -- 172.21.15.63:0/2346238807 shutdown_connections 2024-04-26T04:21:02.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.125+0000 7f4fdff52640 1 -- 172.21.15.63:0/2346238807 wait complete. 2024-04-26T04:21:02.146 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-26T04:21:02.146 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-04-26T04:21:02.147 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T04:21:02.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.257+0000 7f0333fd4640 10 monclient: get_monmap_and_config 2024-04-26T04:21:02.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.257+0000 7f0333fd4640 10 monclient: build_initial_monmap 2024-04-26T04:21:02.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.257+0000 7f0333fd4640 10 monclient: monmap: 2024-04-26T04:21:02.259 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:21:02.259 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:21:02.259 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:21:02.259051+0000 2024-04-26T04:21:02.260 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:21:02.259051+0000 2024-04-26T04:21:02.260 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:21:02.260 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:21:02.260 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:21:02.260 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:21:02.260 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:21:02.260 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:21:02.260 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:21:02.260 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:21:02.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.257+0000 7f0333fd4640 1 Processor -- start 2024-04-26T04:21:02.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.257+0000 7f0333fd4640 1 -- start start 2024-04-26T04:21:02.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.257+0000 7f0333fd4640 10 monclient: init 2024-04-26T04:21:02.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.257+0000 7f0333fd4640 10 monclient: _reopen_session rank -1 2024-04-26T04:21:02.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.257+0000 7f0333fd4640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-04-26T04:21:02.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.257+0000 7f0333fd4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f032c09b3b0 addr v1:172.21.15.186:6789/0 2024-04-26T04:21:02.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.257+0000 7f0333fd4640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f032c137320 0x7f032c139710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:21:02.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.257+0000 7f0333fd4640 10 monclient(hunting): picked mon.noname-b con 0x7f032c137320 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:21:02.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.257+0000 7f0333fd4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f032c139c50 addr v1:172.21.15.63:6789/0 2024-04-26T04:21:02.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.257+0000 7f0333fd4640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f032c09dde0 con 0x7f032c139c50 2024-04-26T04:21:02.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.257+0000 7f0333fd4640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f032c09c9b0 con 0x7f032c09b3b0 2024-04-26T04:21:02.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.257+0000 7f0333fd4640 10 monclient(hunting): start opening mon connection 2024-04-26T04:21:02.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.257+0000 7f0333fd4640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f032c09c100 con 0x7f032c137320 2024-04-26T04:21:02.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.257+0000 7f0333fd4640 10 monclient(hunting): _renew_subs 2024-04-26T04:21:02.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.257+0000 7f0333fd4640 10 monclient(hunting): authenticate will time out at 1093.966770s 2024-04-26T04:21:02.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.257+0000 7f033156f640 1 -- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f032c137320 msgr2=0x7f032c139710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.63:3301/0 2024-04-26T04:21:02.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.257+0000 7f033156f640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f032c137320 0x7f032c139710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:21:02.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.257+0000 7f0332571640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f032c139c50 0x7f032c13c040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:37002/0 (socket says 172.21.15.63:37002) 2024-04-26T04:21:02.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.257+0000 7f0332571640 1 -- 172.21.15.63:0/2617986603 learned_addr learned my addr 172.21.15.63:0/2617986603 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-04-26T04:21:02.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0332571640 1 -- 172.21.15.63:0/2617986603 >> v1:172.21.15.63:6789/0 conn(0x7f032c139c50 legacy=0x7f032c13c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:21:02.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0331d70640 1 -- 172.21.15.63:0/2617986603 >> v1:172.21.15.186:6789/0 conn(0x7f032c09b3b0 legacy=0x7f032c09e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:21:02.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0330d6e640 1 -- 172.21.15.63:0/2617986603 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 324656195 0 0) 0x7f031c001460 con 0x7f032c139c50 2024-04-26T04:21:02.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0330d6e640 10 monclient(hunting): _negotiate 2024-04-26T04:21:02.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0330d6e640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:21:02.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0330d6e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:21:02.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0330d6e640 10 monclient(hunting): my global_id is 4869 2024-04-26T04:21:02.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0330d6e640 1 -- 172.21.15.63:0/2617986603 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0314003530 con 0x7f032c139c50 2024-04-26T04:21:02.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0330d6e640 1 -- 172.21.15.63:0/2617986603 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3067364725 0 0) 0x7f0314003530 con 0x7f032c139c50 2024-04-26T04:21:02.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0330d6e640 1 -- 172.21.15.63:0/2617986603 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f032c137320 msgr2=0x7f032c139710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T04:21:02.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0330d6e640 1 --2- 172.21.15.63:0/2617986603 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f032c137320 0x7f032c139710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:21:02.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0330d6e640 1 -- 172.21.15.63:0/2617986603 >> v1:172.21.15.186:6789/0 conn(0x7f032c09b3b0 legacy=0x7f032c09e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:21:02.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0330d6e640 10 monclient: _finish_hunting 0 2024-04-26T04:21:02.268 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0330d6e640 1 monclient: found mon.noname-a-legacy 2024-04-26T04:21:02.268 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0330d6e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:21:02.268 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0330d6e640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.63:6789/0 2024-04-26T04:21:02.268 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0330d6e640 1 -- 172.21.15.63:0/2617986603 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f032c09db00 con 0x7f032c139c50 2024-04-26T04:21:02.268 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0330d6e640 10 monclient: _finish_auth 0 2024-04-26T04:21:02.268 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0330d6e640 10 monclient: _check_auth_tickets 2024-04-26T04:21:02.268 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0330d6e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:21:02.268 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0330d6e640 1 -- 172.21.15.63:0/2617986603 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2612289768 0 0) 0x7f031c001b00 con 0x7f032c139c50 2024-04-26T04:21:02.268 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0333fd4640 5 monclient: authenticate success, global_id 4869 2024-04-26T04:21:02.268 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0333fd4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T04:21:02.268 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0330d6e640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:21:02.268 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0330d6e640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-26T04:21:02.268 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0330d6e640 10 monclient: dump: 2024-04-26T04:21:02.268 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:21:02.269 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:21:02.269 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:21:02.269 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:21:02.269 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:21:02.269 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:21:02.269 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:21:02.269 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:21:02.269 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:21:02.269 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:21:02.269 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0330d6e640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] but i'm connected to v1:172.21.15.63:6789/0, reconnecting 2024-04-26T04:21:02.269 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0330d6e640 10 monclient: _reopen_session rank -1 2024-04-26T04:21:02.269 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0330d6e640 1 -- 172.21.15.63:0/2617986603 >> v1:172.21.15.63:6789/0 conn(0x7f032c139c50 legacy=0x7f032c13c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:21:02.269 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0330d6e640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T04:21:02.269 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0330d6e640 1 --2- 172.21.15.63:0/2617986603 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0314003ad0 0x7f0314003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:21:02.270 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0330d6e640 10 monclient(hunting): picked mon.a con 0x7f0314003ad0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:21:02.270 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0330d6e640 1 --2- 172.21.15.63:0/2617986603 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f032c137320 0x7f0314004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:21:02.270 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0330d6e640 10 monclient(hunting): picked mon.c con 0x7f032c137320 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:21:02.270 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0330d6e640 1 --2- 172.21.15.63:0/2617986603 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0314004f50 0x7f031400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:21:02.270 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0330d6e640 10 monclient(hunting): picked mon.b con 0x7f0314004f50 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:21:02.270 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0330d6e640 10 monclient(hunting): start opening mon connection 2024-04-26T04:21:02.270 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0330d6e640 1 -- 172.21.15.63:0/2617986603 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f032c09c100 con 0x7f0314003ad0 2024-04-26T04:21:02.270 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0330d6e640 10 monclient(hunting): start opening mon connection 2024-04-26T04:21:02.270 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0330d6e640 1 -- 172.21.15.63:0/2617986603 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f031400bf10 con 0x7f0314004f50 2024-04-26T04:21:02.270 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0330d6e640 10 monclient(hunting): start opening mon connection 2024-04-26T04:21:02.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0330d6e640 1 -- 172.21.15.63:0/2617986603 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f031400c090 con 0x7f032c137320 2024-04-26T04:21:02.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0330d6e640 10 monclient(hunting): _renew_subs 2024-04-26T04:21:02.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0330d6e640 10 monclient(hunting): _finish_auth 0 2024-04-26T04:21:02.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0333fd4640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-04-26T04:21:02.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f033156f640 1 -- 172.21.15.63:0/2617986603 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f032c137320 msgr2=0x7f0314004810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.63:3301/0 2024-04-26T04:21:02.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f033156f640 1 --2- 172.21.15.63:0/2617986603 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f032c137320 0x7f0314004810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:21:02.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0331d70640 1 --2- 172.21.15.63:0/2617986603 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0314003ad0 0x7f0314003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:21:02.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0331d70640 10 monclient(hunting): get_auth_request con 0x7f0314003ad0 auth_method 0 2024-04-26T04:21:02.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0331d70640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:21:02.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0331d70640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:21:02.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0331d70640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:21:02.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0331d70640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:21:02.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0331d70640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:21:02.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0331d70640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T04:21:02.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0332571640 1 --2- 172.21.15.63:0/2617986603 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0314004f50 0x7f031400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:21:02.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0332571640 10 monclient(hunting): get_auth_request con 0x7f0314004f50 auth_method 0 2024-04-26T04:21:02.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0332571640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:21:02.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0332571640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:21:02.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0332571640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:21:02.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0331d70640 10 monclient(hunting): handle_auth_done global_id 4869 payload 293 2024-04-26T04:21:02.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0331d70640 1 -- 172.21.15.63:0/2617986603 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f032c137320 msgr2=0x7f0314004810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T04:21:02.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0331d70640 1 --2- 172.21.15.63:0/2617986603 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f032c137320 0x7f0314004810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:21:02.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0331d70640 1 -- 172.21.15.63:0/2617986603 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0314004f50 msgr2=0x7f031400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:21:02.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0331d70640 1 --2- 172.21.15.63:0/2617986603 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0314004f50 0x7f031400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:21:02.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0331d70640 10 monclient: _finish_hunting 0 2024-04-26T04:21:02.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0331d70640 1 monclient: found mon.a 2024-04-26T04:21:02.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0331d70640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:21:02.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0331d70640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:21:02.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0331d70640 1 -- 172.21.15.63:0/2617986603 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0314003530 con 0x7f0314003ad0 2024-04-26T04:21:02.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0331d70640 10 monclient: _finish_auth 0 2024-04-26T04:21:02.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0331d70640 10 monclient: _check_auth_tickets 2024-04-26T04:21:02.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0331d70640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:21:02.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.261+0000 7f0332571640 1 --2- 172.21.15.63:0/2617986603 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0314004f50 0x7f031400b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T04:21:02.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.265+0000 7f0331d70640 1 -- 172.21.15.63:0/2617986603 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0314003ad0 msgr2=0x7f0314003eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:21:02.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.265+0000 7f0331d70640 1 --2- 172.21.15.63:0/2617986603 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0314003ad0 0x7f0314003eb0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f0324001a70 tx=0x7f032401c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3283efb21e00ea3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:21:02.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.265+0000 7f0330d6e640 1 -- 172.21.15.63:0/2617986603 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03240017a0 con 0x7f0314003ad0 2024-04-26T04:21:02.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.265+0000 7f0330d6e640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:21:02.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.265+0000 7f0330d6e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:21:02.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.265+0000 7f0330d6e640 10 monclient: dump: 2024-04-26T04:21:02.283 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:21:02.283 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:21:02.283 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:21:02.283 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:21:02.283 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:21:02.283 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:21:02.283 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:21:02.283 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:21:02.283 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:21:02.283 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:21:02.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.265+0000 7f0333fd4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T04:21:02.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.265+0000 7f0330d6e640 1 -- 172.21.15.63:0/2617986603 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0324007330 con 0x7f0314003ad0 2024-04-26T04:21:02.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.265+0000 7f0330d6e640 10 monclient: handle_config config(0 keys) 2024-04-26T04:21:02.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.265+0000 7f0333fd4640 10 monclient: get_monmap_and_config success 2024-04-26T04:21:02.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.265+0000 7f0333fd4640 4 set_mon_vals no callback set 2024-04-26T04:21:02.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.265+0000 7f0330d6e640 1 -- 172.21.15.63:0/2617986603 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03240057d0 con 0x7f0314003ad0 2024-04-26T04:21:02.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.265+0000 7f0330d6e640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:21:02.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.265+0000 7f0330d6e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:21:02.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.265+0000 7f0330d6e640 10 monclient: dump: 2024-04-26T04:21:02.284 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:21:02.284 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:21:02.284 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:21:02.285 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:21:02.285 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:21:02.285 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:21:02.285 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:21:02.285 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:21:02.285 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:21:02.285 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:21:02.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.265+0000 7f0333fd4640 10 monclient: shutdown 2024-04-26T04:21:02.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.265+0000 7f0333fd4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:21:02.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.265+0000 7f0333fd4640 1 -- 172.21.15.63:0/2617986603 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0314003ad0 msgr2=0x7f0314003eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:21:02.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.265+0000 7f0333fd4640 1 --2- 172.21.15.63:0/2617986603 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0314003ad0 0x7f0314003eb0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f0324001a70 tx=0x7f032401c6a0 comp rx=0 tx=0).stop 2024-04-26T04:21:02.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.265+0000 7f0332571640 1 -- 172.21.15.63:0/2617986603 reap_dead start 2024-04-26T04:21:02.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.265+0000 7f0333fd4640 1 -- 172.21.15.63:0/2617986603 shutdown_connections 2024-04-26T04:21:02.287 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.265+0000 7f0333fd4640 1 -- 172.21.15.63:0/2617986603 >> 172.21.15.63:0/2617986603 conn(0x7f032c0a1e70 msgr2=0x7f032c0a2280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:21:02.287 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.265+0000 7f0333fd4640 1 -- 172.21.15.63:0/2617986603 shutdown_connections 2024-04-26T04:21:02.287 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.265+0000 7f0333fd4640 1 -- 172.21.15.63:0/2617986603 wait complete. 2024-04-26T04:21:02.287 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.269+0000 7f0333fd4640 10 monclient: build_initial_monmap 2024-04-26T04:21:02.287 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.269+0000 7f0333fd4640 10 monclient: monmap: 2024-04-26T04:21:02.287 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:21:02.287 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:21:02.287 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:21:02.287 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:21:02.287 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:21:02.287 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:21:02.288 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:21:02.288 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:21:02.288 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:21:02.288 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:21:02.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.269+0000 7f0333fd4640 1 Processor -- start 2024-04-26T04:21:02.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.269+0000 7f0333fd4640 1 -- start start 2024-04-26T04:21:02.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.269+0000 7f0333fd4640 10 monclient: init 2024-04-26T04:21:02.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.269+0000 7f0333fd4640 10 monclient: _reopen_session rank -1 2024-04-26T04:21:02.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.269+0000 7f0333fd4640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T04:21:02.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.269+0000 7f0333fd4640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0314003ad0 0x7f032c1cca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:21:02.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.269+0000 7f0333fd4640 10 monclient(hunting): picked mon.noname-a con 0x7f0314003ad0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:21:02.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.269+0000 7f0333fd4640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f032c1357a0 0x7f032c135b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:21:02.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.269+0000 7f0333fd4640 10 monclient(hunting): picked mon.noname-c con 0x7f032c1357a0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:21:02.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.269+0000 7f0333fd4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f032c12a4b0 0x7f032c12a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:21:02.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.269+0000 7f0333fd4640 10 monclient(hunting): picked mon.noname-b con 0x7f032c12a4b0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:21:02.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.269+0000 7f0333fd4640 10 monclient(hunting): start opening mon connection 2024-04-26T04:21:02.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.269+0000 7f0333fd4640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f032c13e300 con 0x7f0314003ad0 2024-04-26T04:21:02.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.269+0000 7f0333fd4640 10 monclient(hunting): start opening mon connection 2024-04-26T04:21:02.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.269+0000 7f0333fd4640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f032c13e180 con 0x7f032c12a4b0 2024-04-26T04:21:02.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.269+0000 7f0333fd4640 10 monclient(hunting): start opening mon connection 2024-04-26T04:21:02.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.269+0000 7f0333fd4640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f032c13e480 con 0x7f032c1357a0 2024-04-26T04:21:02.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.269+0000 7f0333fd4640 10 monclient(hunting): _renew_subs 2024-04-26T04:21:02.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.269+0000 7f0333fd4640 10 monclient(hunting): authenticate will time out at 1093.977302s 2024-04-26T04:21:02.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.269+0000 7f033156f640 1 -- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f032c1357a0 msgr2=0x7f032c135b80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:3301/0 2024-04-26T04:21:02.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.269+0000 7f033156f640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f032c1357a0 0x7f032c135b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:21:02.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.269+0000 7f0331d70640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0314003ad0 0x7f032c1cca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:21:02.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.269+0000 7f0331d70640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0314003ad0 0x7f032c1cca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:56784/0 (socket says 172.21.15.63:56784) 2024-04-26T04:21:02.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.269+0000 7f0331d70640 1 -- 172.21.15.63:0/3725255000 learned_addr learned my addr 172.21.15.63:0/3725255000 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:21:02.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.269+0000 7f0331d70640 10 monclient(hunting): get_auth_request con 0x7f0314003ad0 auth_method 0 2024-04-26T04:21:02.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.269+0000 7f0331d70640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:21:02.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.269+0000 7f0331d70640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:21:02.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.269+0000 7f0331d70640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:21:02.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.269+0000 7f0331d70640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:21:02.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.269+0000 7f0331d70640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:21:02.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.269+0000 7f0331d70640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:21:02.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.269+0000 7f0332571640 1 --2- 172.21.15.63:0/3725255000 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f032c12a4b0 0x7f032c12a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:21:02.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.269+0000 7f0332571640 10 monclient(hunting): get_auth_request con 0x7f032c12a4b0 auth_method 0 2024-04-26T04:21:02.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.269+0000 7f0332571640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:21:02.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.269+0000 7f0332571640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:21:02.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.269+0000 7f0332571640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:21:02.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.269+0000 7f0331d70640 10 monclient(hunting): handle_auth_done global_id 4872 payload 824 2024-04-26T04:21:02.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.269+0000 7f0331d70640 1 -- 172.21.15.63:0/3725255000 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f032c1357a0 msgr2=0x7f032c135b80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T04:21:02.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.269+0000 7f0331d70640 1 --2- 172.21.15.63:0/3725255000 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f032c1357a0 0x7f032c135b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:21:02.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.269+0000 7f0331d70640 1 -- 172.21.15.63:0/3725255000 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f032c12a4b0 msgr2=0x7f032c12a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:21:02.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.269+0000 7f0331d70640 1 --2- 172.21.15.63:0/3725255000 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f032c12a4b0 0x7f032c12a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:21:02.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.269+0000 7f0331d70640 10 monclient: _finish_hunting 0 2024-04-26T04:21:02.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.269+0000 7f0331d70640 1 monclient: found mon.noname-a 2024-04-26T04:21:02.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.269+0000 7f0331d70640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:21:02.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.269+0000 7f0331d70640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:21:02.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.269+0000 7f0331d70640 1 -- 172.21.15.63:0/3725255000 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f032c09cc50 con 0x7f0314003ad0 2024-04-26T04:21:02.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f0331d70640 1 -- 172.21.15.63:0/3725255000 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0314003ad0 msgr2=0x7f032c1cca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:21:02.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f0331d70640 1 --2- 172.21.15.63:0/3725255000 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0314003ad0 0x7f032c1cca80 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f0324001120 tx=0x7f0314005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:21:02.292 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f032a7fc640 1 -- 172.21.15.63:0/3725255000 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03240bd070 con 0x7f0314003ad0 2024-04-26T04:21:02.292 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f032a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:21:02.292 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f032a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:21:02.292 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f032a7fc640 10 monclient: dump: 2024-04-26T04:21:02.292 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:21:02.292 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:21:02.292 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:21:02.292 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:21:02.292 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:21:02.292 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:21:02.292 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:21:02.292 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:21:02.292 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:21:02.292 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:21:02.292 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f032a7fc640 10 monclient: _finish_auth 0 2024-04-26T04:21:02.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f032a7fc640 10 monclient: _check_auth_tickets 2024-04-26T04:21:02.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f032a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:21:02.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f032a7fc640 1 -- 172.21.15.63:0/3725255000 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03240b2070 con 0x7f0314003ad0 2024-04-26T04:21:02.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f0333fd4640 5 monclient: authenticate success, global_id 4872 2024-04-26T04:21:02.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f0333fd4640 10 monclient: _renew_subs 2024-04-26T04:21:02.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f0333fd4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:21:02.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f0333fd4640 1 -- 172.21.15.63:0/3725255000 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f032c12dc70 con 0x7f0314003ad0 2024-04-26T04:21:02.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f032a7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T04:21:02.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f032a7fc640 1 -- 172.21.15.63:0/3725255000 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03240017a0 con 0x7f0314003ad0 2024-04-26T04:21:02.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f032a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:21:02.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f032a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:21:02.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f032a7fc640 10 monclient: dump: 2024-04-26T04:21:02.293 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:21:02.294 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:21:02.294 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:21:02.294 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:21:02.294 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:21:02.294 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:21:02.294 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:21:02.294 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:21:02.295 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:21:02.295 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:21:02.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f0333fd4640 10 monclient: _renew_subs 2024-04-26T04:21:02.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f0333fd4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:21:02.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f0333fd4640 1 -- 172.21.15.63:0/3725255000 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f032c1ccfc0 con 0x7f0314003ad0 2024-04-26T04:21:02.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f032bfff640 4 set_mon_vals no callback set 2024-04-26T04:21:02.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f0331d70640 1 -- 172.21.15.63:0/3725255000 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0314003ad0 msgr2=0x7f032c1cca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T04:21:02.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f0331d70640 1 -- 172.21.15.63:0/3725255000 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0314003ad0 msgr2=0x7f032c1cca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T04:21:02.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f0331d70640 1 --2- 172.21.15.63:0/3725255000 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0314003ad0 0x7f032c1cca80 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f0324001120 tx=0x7f0314005840 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:21:02.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f0331d70640 1 --2- 172.21.15.63:0/3725255000 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0314003ad0 0x7f032c1cca80 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f0324001120 tx=0x7f0314005840 comp rx=0 tx=0).stop 2024-04-26T04:21:02.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f032a7fc640 10 monclient: ms_handle_reset current mon [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:21:02.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f032a7fc640 10 monclient: _reopen_session rank -1 2024-04-26T04:21:02.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f032a7fc640 1 -- 172.21.15.63:0/3725255000 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0314003ad0 msgr2=0x7f032c1cca80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:21:02.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f032a7fc640 1 --2- 172.21.15.63:0/3725255000 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0314003ad0 0x7f032c1cca80 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:21:02.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f032a7fc640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T04:21:02.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f032a7fc640 1 --2- 172.21.15.63:0/3725255000 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f032c1357a0 0x7f0318005880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:21:02.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f032a7fc640 10 monclient(hunting): picked mon.c con 0x7f032c1357a0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:21:02.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f032a7fc640 1 --2- 172.21.15.63:0/3725255000 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0318006250 0x7f03180086e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:21:02.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f032a7fc640 10 monclient(hunting): picked mon.a con 0x7f0318006250 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:21:02.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f032a7fc640 1 --2- 172.21.15.63:0/3725255000 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f032c12a4b0 0x7f031800b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:21:02.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f032a7fc640 10 monclient(hunting): picked mon.b con 0x7f032c12a4b0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:21:02.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f032a7fc640 10 monclient(hunting): start opening mon connection 2024-04-26T04:21:02.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f032a7fc640 1 -- 172.21.15.63:0/3725255000 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f031800bc10 con 0x7f0318006250 2024-04-26T04:21:02.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f032a7fc640 10 monclient(hunting): start opening mon connection 2024-04-26T04:21:02.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f032a7fc640 1 -- 172.21.15.63:0/3725255000 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f031800bd90 con 0x7f032c12a4b0 2024-04-26T04:21:02.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f032a7fc640 10 monclient(hunting): start opening mon connection 2024-04-26T04:21:02.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f0331d70640 1 -- 172.21.15.63:0/3725255000 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f032c1357a0 msgr2=0x7f0318005880 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:3301/0 2024-04-26T04:21:02.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f032a7fc640 1 -- 172.21.15.63:0/3725255000 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f031800bf40 con 0x7f032c1357a0 2024-04-26T04:21:02.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f0331d70640 1 --2- 172.21.15.63:0/3725255000 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f032c1357a0 0x7f0318005880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:21:02.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f032a7fc640 10 monclient(hunting): _renew_subs 2024-04-26T04:21:02.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f0333fd4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:21:02.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f0333fd4640 1 --2- 172.21.15.63:0/3725255000 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f032c1cca80 0x7f032c135b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:21:02.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f0333fd4640 10 monclient(hunting): start opening mon connection 2024-04-26T04:21:02.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f0333fd4640 1 -- 172.21.15.63:0/3725255000 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f032c12cb10 con 0x7f032c1cca80 2024-04-26T04:21:02.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f033156f640 1 --2- 172.21.15.63:0/3725255000 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0318006250 0x7f03180086e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:21:02.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f033156f640 10 monclient(hunting): get_auth_request con 0x7f0318006250 auth_method 0 2024-04-26T04:21:02.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f033156f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:21:02.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f033156f640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:21:02.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f033156f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:21:02.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f033156f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:21:02.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f033156f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:21:02.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f033156f640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T04:21:02.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f033156f640 10 monclient(hunting): handle_auth_done global_id 4872 payload 293 2024-04-26T04:21:02.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f0332571640 1 --2- 172.21.15.63:0/3725255000 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f032c12a4b0 0x7f031800b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:21:02.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f033156f640 1 -- 172.21.15.63:0/3725255000 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f032c1357a0 msgr2=0x7f0318005880 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T04:21:02.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f033156f640 1 --2- 172.21.15.63:0/3725255000 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f032c1357a0 0x7f0318005880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:21:02.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f033156f640 1 -- 172.21.15.63:0/3725255000 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f032c12a4b0 msgr2=0x7f031800b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:21:02.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f033156f640 1 --2- 172.21.15.63:0/3725255000 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f032c12a4b0 0x7f031800b200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:21:02.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f033156f640 10 monclient: _finish_hunting 0 2024-04-26T04:21:02.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f033156f640 1 monclient: found mon.a 2024-04-26T04:21:02.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f033156f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:21:02.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f033156f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:21:02.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f033156f640 1 -- 172.21.15.63:0/3725255000 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) -- 0x7f031800c270 con 0x7f0318006250 2024-04-26T04:21:02.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f033156f640 10 monclient: _finish_auth 0 2024-04-26T04:21:02.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f033156f640 10 monclient: _check_auth_tickets 2024-04-26T04:21:02.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f033156f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:21:02.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f0331d70640 1 --2- 172.21.15.63:0/3725255000 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f032c1cca80 0x7f032c135b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:21:02.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f0331d70640 10 monclient: get_auth_request con 0x7f032c1cca80 auth_method 0 2024-04-26T04:21:02.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f0331d70640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:21:02.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f0331d70640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:21:02.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f0331d70640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:21:02.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f0331d70640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:21:02.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f0331d70640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:21:02.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f0331d70640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:21:02.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f0331d70640 10 monclient(hunting): handle_auth_done global_id 4807 payload 824 2024-04-26T04:21:02.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f0331d70640 1 -- 172.21.15.63:0/3725255000 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f032c062e10 con 0x7f032c1cca80 2024-04-26T04:21:02.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f033156f640 1 -- 172.21.15.63:0/3725255000 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0318006250 msgr2=0x7f03180086e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:21:02.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f033156f640 1 --2- 172.21.15.63:0/3725255000 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0318006250 0x7f03180086e0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f0320003d10 tx=0x7f0324001670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:21:02.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f032a7fc640 1 -- 172.21.15.63:0/3725255000 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0320009a30 con 0x7f0318006250 2024-04-26T04:21:02.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f032a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:21:02.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f032a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:21:02.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f032a7fc640 10 monclient: dump: 2024-04-26T04:21:02.301 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:21:02.301 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:21:02.301 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:21:02.301 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:21:02.301 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:21:02.301 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:21:02.301 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:21:02.301 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:21:02.301 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:21:02.301 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:21:02.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f032a7fc640 1 -- 172.21.15.63:0/3725255000 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f032000d600 con 0x7f0318006250 2024-04-26T04:21:02.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.273+0000 7f032a7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T04:21:02.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.277+0000 7f032b7fe640 4 set_mon_vals no callback set 2024-04-26T04:21:02.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.277+0000 7f0331d70640 1 -- 172.21.15.63:0/3725255000 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f032c1cca80 msgr2=0x7f032c135b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:21:02.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.277+0000 7f0331d70640 1 --2- 172.21.15.63:0/3725255000 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f032c1cca80 0x7f032c135b80 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f032c12c210 tx=0x7f0324007c30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:21:02.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.277+0000 7f032a7fc640 1 -- 172.21.15.63:0/3725255000 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f032000d7b0 con 0x7f0318006250 2024-04-26T04:21:02.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.277+0000 7f032a7fc640 1 --2- 172.21.15.63:0/3725255000 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f0314003ad0 0x7f03180407b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:21:02.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.277+0000 7f032a7fc640 1 -- 172.21.15.63:0/3725255000 <== mon.0 v2:172.21.15.63:3300/0 4 ==== osd_map(115..115 src has 1..115) ==== 4346+0+0 (secure 0 0 0) 0x7f0320046e30 con 0x7f0318006250 2024-04-26T04:21:02.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.277+0000 7f032a7fc640 1 -- 172.21.15.63:0/3725255000 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03240c7060 con 0x7f032c1cca80 2024-04-26T04:21:02.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.277+0000 7f032a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:21:02.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.277+0000 7f032a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T04:21:02.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.277+0000 7f032a7fc640 10 monclient: dump: 2024-04-26T04:21:02.302 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:21:02.303 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:21:02.303 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:21:02.303 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:21:02.303 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:21:02.303 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:21:02.303 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:21:02.303 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:21:02.303 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:21:02.303 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:21:02.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.277+0000 7f0332571640 1 --2- 172.21.15.63:0/3725255000 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f0314003ad0 0x7f03180407b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:21:02.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.277+0000 7f0332571640 10 monclient: get_auth_request con 0x7f0314003ad0 auth_method 0 2024-04-26T04:21:02.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.277+0000 7f032a7fc640 1 -- 172.21.15.63:0/3725255000 <== mon.1 v2:172.21.15.186:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0324007330 con 0x7f032c1cca80 2024-04-26T04:21:02.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.277+0000 7f032a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:21:02.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.277+0000 7f032a7fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:21:02.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.277+0000 7f032a7fc640 1 -- 172.21.15.63:0/3725255000 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f032c1cca80 msgr2=0x7f032c135b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:21:02.304 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.277+0000 7f032a7fc640 1 --2- 172.21.15.63:0/3725255000 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f032c1cca80 0x7f032c135b80 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f032c12c210 tx=0x7f0324007c30 comp rx=0 tx=0).stop 2024-04-26T04:21:02.304 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.277+0000 7f032a7fc640 1 -- 172.21.15.63:0/3725255000 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f032c1cca80 msgr2=0x7f032c135b80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:21:02.304 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.277+0000 7f032a7fc640 1 --2- 172.21.15.63:0/3725255000 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f032c1cca80 0x7f032c135b80 secure :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f032c12c210 tx=0x7f0324007c30 comp rx=0 tx=0).stop 2024-04-26T04:21:02.304 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.281+0000 7f0332571640 1 --2- 172.21.15.63:0/3725255000 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f0314003ad0 0x7f03180407b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f032c0a3070 tx=0x7f031c0016a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:21:02.305 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.281+0000 7f0333fd4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T04:21:02.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.281+0000 7f0333fd4640 1 --2- 172.21.15.63:0/3725255000 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f032c12a890 0x7f032c12ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:21:02.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.281+0000 7f0333fd4640 10 monclient(hunting): start opening mon connection 2024-04-26T04:21:02.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.281+0000 7f0333fd4640 1 -- 172.21.15.63:0/3725255000 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f032c0a0c70 con 0x7f032c12a890 2024-04-26T04:21:02.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.285+0000 7f0331d70640 1 --2- 172.21.15.63:0/3725255000 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f032c12a890 0x7f032c12ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:21:02.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.285+0000 7f0331d70640 10 monclient: get_auth_request con 0x7f032c12a890 auth_method 0 2024-04-26T04:21:02.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.285+0000 7f0331d70640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:21:02.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.285+0000 7f0331d70640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:21:02.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.285+0000 7f0331d70640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:21:02.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.285+0000 7f0331d70640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:21:02.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.285+0000 7f0331d70640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:21:02.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.285+0000 7f0331d70640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:21:02.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.285+0000 7f0331d70640 10 monclient(hunting): handle_auth_done global_id 4810 payload 824 2024-04-26T04:21:02.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.285+0000 7f0331d70640 1 -- 172.21.15.63:0/3725255000 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f032c1360c0 con 0x7f032c12a890 2024-04-26T04:21:02.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.285+0000 7f0331d70640 1 -- 172.21.15.63:0/3725255000 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f032c12a890 msgr2=0x7f032c12ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:21:02.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.285+0000 7f0331d70640 1 --2- 172.21.15.63:0/3725255000 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f032c12a890 0x7f032c12ac70 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f0324000c10 tx=0x7f03240ca000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:21:02.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.285+0000 7f032a7fc640 1 -- 172.21.15.63:0/3725255000 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03240d8980 con 0x7f032c12a890 2024-04-26T04:21:02.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.285+0000 7f032a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:21:02.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.285+0000 7f032a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T04:21:02.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.285+0000 7f032a7fc640 10 monclient: dump: 2024-04-26T04:21:02.307 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:21:02.307 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:21:02.307 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:21:02.308 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:21:02.308 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:21:02.308 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:21:02.308 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:21:02.308 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:21:02.308 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:21:02.308 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:21:02.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.285+0000 7f032a7fc640 1 -- 172.21.15.63:0/3725255000 <== mon.1 v2:172.21.15.186:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4880 (secure 0 0 0) 0x7f03240d3400 con 0x7f032c12a890 2024-04-26T04:21:02.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.285+0000 7f032a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T04:21:02.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.285+0000 7f032a7fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:21:02.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.285+0000 7f032a7fc640 1 -- 172.21.15.63:0/3725255000 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f032c12a890 msgr2=0x7f032c12ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:21:02.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.285+0000 7f032a7fc640 1 --2- 172.21.15.63:0/3725255000 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f032c12a890 0x7f032c12ac70 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f0324000c10 tx=0x7f03240ca000 comp rx=0 tx=0).stop 2024-04-26T04:21:02.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.285+0000 7f032a7fc640 1 -- 172.21.15.63:0/3725255000 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f032c12a890 msgr2=0x7f032c12ac70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:21:02.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.285+0000 7f032a7fc640 1 --2- 172.21.15.63:0/3725255000 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f032c12a890 0x7f032c12ac70 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:21:02.309 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-26T04:21:02.309 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b", 2024-04-26T04:21:02.309 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 1, 2024-04-26T04:21:02.309 INFO:teuthology.orchestra.run.smithi063.stdout: "state": "peon", 2024-04-26T04:21:02.310 INFO:teuthology.orchestra.run.smithi063.stdout: "election_epoch": 18, 2024-04-26T04:21:02.310 INFO:teuthology.orchestra.run.smithi063.stdout: "uptime": 153400, 2024-04-26T04:21:02.310 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum": [ 2024-04-26T04:21:02.310 INFO:teuthology.orchestra.run.smithi063.stdout: 0, 2024-04-26T04:21:02.310 INFO:teuthology.orchestra.run.smithi063.stdout: 1 2024-04-26T04:21:02.310 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:21:02.310 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_age": 0, 2024-04-26T04:21:02.310 INFO:teuthology.orchestra.run.smithi063.stdout: "features": { 2024-04-26T04:21:02.310 INFO:teuthology.orchestra.run.smithi063.stdout: "required_con": "2450521708007866372", 2024-04-26T04:21:02.310 INFO:teuthology.orchestra.run.smithi063.stdout: "required_mon": [ 2024-04-26T04:21:02.310 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:21:02.310 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:21:02.311 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:21:02.311 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:21:02.311 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:21:02.311 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:21:02.311 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:21:02.311 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:21:02.311 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:21:02.311 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:21:02.311 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:21:02.311 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:21:02.311 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_con": "4540701547738038271", 2024-04-26T04:21:02.311 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_mon": [ 2024-04-26T04:21:02.311 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:21:02.311 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:21:02.311 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:21:02.312 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:21:02.312 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:21:02.312 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:21:02.312 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:21:02.312 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:21:02.312 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:21:02.312 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:21:02.312 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:21:02.312 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:21:02.312 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:21:02.312 INFO:teuthology.orchestra.run.smithi063.stdout: "outside_quorum": [], 2024-04-26T04:21:02.312 INFO:teuthology.orchestra.run.smithi063.stdout: "extra_probe_peers": [], 2024-04-26T04:21:02.312 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_provider": [], 2024-04-26T04:21:02.312 INFO:teuthology.orchestra.run.smithi063.stdout: "monmap": { 2024-04-26T04:21:02.313 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 1, 2024-04-26T04:21:02.313 INFO:teuthology.orchestra.run.smithi063.stdout: "fsid": "cd1c6a0c-4e4b-4014-89c0-2e535479ef39", 2024-04-26T04:21:02.313 INFO:teuthology.orchestra.run.smithi063.stdout: "modified": "2024-04-26T04:18:03.702339Z", 2024-04-26T04:21:02.313 INFO:teuthology.orchestra.run.smithi063.stdout: "created": "2024-04-26T04:18:03.702339Z", 2024-04-26T04:21:02.313 INFO:teuthology.orchestra.run.smithi063.stdout: "min_mon_release": 19, 2024-04-26T04:21:02.313 INFO:teuthology.orchestra.run.smithi063.stdout: "min_mon_release_name": "squid", 2024-04-26T04:21:02.313 INFO:teuthology.orchestra.run.smithi063.stdout: "election_strategy": 1, 2024-04-26T04:21:02.313 INFO:teuthology.orchestra.run.smithi063.stdout: "disallowed_leaders: ": "", 2024-04-26T04:21:02.313 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": false, 2024-04-26T04:21:02.313 INFO:teuthology.orchestra.run.smithi063.stdout: "tiebreaker_mon": "", 2024-04-26T04:21:02.313 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_ranks: ": "", 2024-04-26T04:21:02.313 INFO:teuthology.orchestra.run.smithi063.stdout: "features": { 2024-04-26T04:21:02.313 INFO:teuthology.orchestra.run.smithi063.stdout: "persistent": [ 2024-04-26T04:21:02.313 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:21:02.313 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:21:02.314 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:21:02.314 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:21:02.314 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:21:02.314 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:21:02.314 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:21:02.314 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:21:02.314 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:21:02.314 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:21:02.314 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:21:02.314 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:21:02.314 INFO:teuthology.orchestra.run.smithi063.stdout: "optional": [] 2024-04-26T04:21:02.314 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:21:02.314 INFO:teuthology.orchestra.run.smithi063.stdout: "mons": [ 2024-04-26T04:21:02.315 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:21:02.315 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 0, 2024-04-26T04:21:02.315 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a", 2024-04-26T04:21:02.315 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:21:02.315 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:21:02.315 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:21:02.315 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:21:02.315 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:3300", 2024-04-26T04:21:02.315 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:21:02.315 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:21:02.315 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:21:02.315 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:21:02.315 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6789", 2024-04-26T04:21:02.315 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:21:02.315 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:21:02.316 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:21:02.316 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:21:02.316 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6789/0", 2024-04-26T04:21:02.316 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6789/0", 2024-04-26T04:21:02.316 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:21:02.316 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:21:02.316 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:21:02.316 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:21:02.316 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:21:02.316 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 1, 2024-04-26T04:21:02.316 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b", 2024-04-26T04:21:02.316 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:21:02.316 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:21:02.316 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:21:02.316 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:21:02.317 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:3300", 2024-04-26T04:21:02.317 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:21:02.317 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:21:02.317 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:21:02.317 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:21:02.317 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:6789", 2024-04-26T04:21:02.317 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:21:02.317 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:21:02.317 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:21:02.317 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:21:02.317 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:6789/0", 2024-04-26T04:21:02.317 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.186:6789/0", 2024-04-26T04:21:02.317 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:21:02.317 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:21:02.318 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:21:02.318 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:21:02.318 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:21:02.318 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 2, 2024-04-26T04:21:02.318 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "c", 2024-04-26T04:21:02.318 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:21:02.318 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:21:02.318 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:21:02.318 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:21:02.318 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:3301", 2024-04-26T04:21:02.318 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:21:02.318 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:21:02.319 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:21:02.319 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:21:02.319 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6790", 2024-04-26T04:21:02.319 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:21:02.319 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:21:02.319 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:21:02.319 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:21:02.319 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6790/0", 2024-04-26T04:21:02.319 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6790/0", 2024-04-26T04:21:02.319 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:21:02.319 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:21:02.320 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:21:02.320 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:21:02.320 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:21:02.320 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:21:02.320 INFO:teuthology.orchestra.run.smithi063.stdout: "feature_map": { 2024-04-26T04:21:02.320 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": [ 2024-04-26T04:21:02.320 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:21:02.320 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:21:02.320 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:21:02.320 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 1 2024-04-26T04:21:02.320 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:21:02.320 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:21:02.320 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": [ 2024-04-26T04:21:02.320 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:21:02.320 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:21:02.321 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:21:02.321 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 2 2024-04-26T04:21:02.321 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:21:02.321 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:21:02.321 INFO:teuthology.orchestra.run.smithi063.stdout: "client": [ 2024-04-26T04:21:02.321 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:21:02.321 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:21:02.321 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:21:02.321 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 6 2024-04-26T04:21:02.321 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:21:02.321 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:21:02.321 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": [ 2024-04-26T04:21:02.321 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:21:02.321 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:21:02.321 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:21:02.322 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 1 2024-04-26T04:21:02.322 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:21:02.322 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:21:02.322 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:21:02.322 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": false 2024-04-26T04:21:02.322 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-26T04:21:02.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.289+0000 7f0333fd4640 1 -- 172.21.15.63:0/3725255000 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f0314003ad0 msgr2=0x7f03180407b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:21:02.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.289+0000 7f0333fd4640 1 --2- 172.21.15.63:0/3725255000 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f0314003ad0 0x7f03180407b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f032c0a3070 tx=0x7f031c0016a0 comp rx=0 tx=0).stop 2024-04-26T04:21:02.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.289+0000 7f0333fd4640 10 monclient: shutdown 2024-04-26T04:21:02.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.289+0000 7f0333fd4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:21:02.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.289+0000 7f0333fd4640 1 -- 172.21.15.63:0/3725255000 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0318006250 msgr2=0x7f03180086e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:21:02.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.289+0000 7f0333fd4640 1 --2- 172.21.15.63:0/3725255000 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0318006250 0x7f03180086e0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f0320003d10 tx=0x7f0324001670 comp rx=0 tx=0).stop 2024-04-26T04:21:02.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.289+0000 7f0332571640 1 -- 172.21.15.63:0/3725255000 reap_dead start 2024-04-26T04:21:02.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.289+0000 7f0332571640 1 -- 172.21.15.63:0/3725255000 reap_dead start 2024-04-26T04:21:02.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.289+0000 7f0333fd4640 1 -- 172.21.15.63:0/3725255000 shutdown_connections 2024-04-26T04:21:02.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.289+0000 7f0333fd4640 1 -- 172.21.15.63:0/3725255000 >> 172.21.15.63:0/3725255000 conn(0x7f032c0a1e70 msgr2=0x7f032c136530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:21:02.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.289+0000 7f0333fd4640 1 -- 172.21.15.63:0/3725255000 shutdown_connections 2024-04-26T04:21:02.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:21:02.289+0000 7f0333fd4640 1 -- 172.21.15.63:0/3725255000 wait complete. 2024-04-26T04:21:02.324 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-26T04:21:02.324 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-04-26T04:21:04.265 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:21:04.265 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29143]: starting. 2024-04-26T04:21:04.265 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29143]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-29108 2024-04-26T04:21:04.265 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29143]: created object 0... 2024-04-26T04:21:04.265 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29143]: created object 25... 2024-04-26T04:21:04.265 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29143]: created object 49... 2024-04-26T04:21:04.265 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29143]: finishing. 2024-04-26T04:21:04.265 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29143]: shutting down. 2024-04-26T04:21:04.324 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:21:04.324 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[29145]: starting. 2024-04-26T04:21:04.324 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[29145]: listing objects. 2024-04-26T04:21:04.324 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[29145]: listed object 0... 2024-04-26T04:21:04.324 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[29145]: listed object 25... 2024-04-26T04:21:04.324 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[29145]: saw 50 objects 2024-04-26T04:21:04.324 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[29145]: shutting down. 2024-04-26T04:21:05.868 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:21:05.868 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[29144]: starting. 2024-04-26T04:21:05.868 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[29144]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-29108 2024-04-26T04:21:05.869 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[29144]: shutting down. 2024-04-26T04:21:05.871 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:21:05.871 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:21:05.871 INFO:teuthology.orchestra.run.smithi186.stdout:******* SUCCESS ********** 2024-04-26T04:21:05.875 DEBUG:teuthology.orchestra.run.smithi186:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-26T04:21:08.090 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29196]: starting. 2024-04-26T04:21:08.091 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29196]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-29195 2024-04-26T04:21:08.091 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29196]: created object 0... 2024-04-26T04:21:08.091 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29196]: created object 25... 2024-04-26T04:21:08.091 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29196]: created object 49... 2024-04-26T04:21:08.092 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29196]: finishing. 2024-04-26T04:21:08.092 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29196]: shutting down. 2024-04-26T04:21:08.896 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[29197]: starting. 2024-04-26T04:21:08.897 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[29197]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-29195 2024-04-26T04:21:08.897 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[29197]: shutting down. 2024-04-26T04:21:11.160 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:21:11.160 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29230]: starting. 2024-04-26T04:21:11.161 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29230]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-29195 2024-04-26T04:21:11.161 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29230]: created object 0... 2024-04-26T04:21:11.161 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29230]: created object 25... 2024-04-26T04:21:11.161 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29230]: created object 49... 2024-04-26T04:21:11.161 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29230]: finishing. 2024-04-26T04:21:11.161 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29230]: shutting down. 2024-04-26T04:21:11.217 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:21:11.217 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[29232]: starting. 2024-04-26T04:21:11.217 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[29232]: listing objects. 2024-04-26T04:21:11.217 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[29232]: listed object 0... 2024-04-26T04:21:11.217 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[29232]: listed object 25... 2024-04-26T04:21:11.217 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[29232]: saw 50 objects 2024-04-26T04:21:11.217 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[29232]: shutting down. 2024-04-26T04:21:11.937 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:21:11.937 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[29231]: starting. 2024-04-26T04:21:11.937 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[29231]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-29195 2024-04-26T04:21:11.937 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[29231]: shutting down. 2024-04-26T04:21:11.940 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:21:11.940 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:21:11.940 INFO:teuthology.orchestra.run.smithi186.stdout:******* SUCCESS ********** 2024-04-26T04:21:11.944 DEBUG:teuthology.orchestra.run.smithi186:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-26T04:21:14.155 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29283]: starting. 2024-04-26T04:21:14.155 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29283]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-29282 2024-04-26T04:21:14.155 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29283]: created object 0... 2024-04-26T04:21:14.155 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29283]: created object 25... 2024-04-26T04:21:14.155 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29283]: created object 49... 2024-04-26T04:21:14.155 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29283]: finishing. 2024-04-26T04:21:14.156 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29283]: shutting down. 2024-04-26T04:21:14.946 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[29284]: starting. 2024-04-26T04:21:14.947 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[29284]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-29282 2024-04-26T04:21:14.947 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[29284]: shutting down. 2024-04-26T04:21:17.199 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:21:17.200 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29317]: starting. 2024-04-26T04:21:17.200 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29317]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-29282 2024-04-26T04:21:17.200 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29317]: created object 0... 2024-04-26T04:21:17.200 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29317]: created object 25... 2024-04-26T04:21:17.200 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29317]: created object 49... 2024-04-26T04:21:17.200 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29317]: finishing. 2024-04-26T04:21:17.200 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29317]: shutting down. 2024-04-26T04:21:17.262 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:21:17.262 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[29319]: starting. 2024-04-26T04:21:17.263 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[29319]: listing objects. 2024-04-26T04:21:17.263 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[29319]: listed object 0... 2024-04-26T04:21:17.263 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[29319]: listed object 25... 2024-04-26T04:21:17.263 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[29319]: saw 50 objects 2024-04-26T04:21:17.263 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[29319]: shutting down. 2024-04-26T04:21:18.002 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:21:18.002 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[29318]: starting. 2024-04-26T04:21:18.002 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[29318]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-29282 2024-04-26T04:21:18.002 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[29318]: shutting down. 2024-04-26T04:21:18.005 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:21:18.005 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:21:18.005 INFO:teuthology.orchestra.run.smithi186.stdout:******* SUCCESS ********** 2024-04-26T04:21:18.008 DEBUG:teuthology.orchestra.run.smithi186:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-26T04:21:20.134 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29370]: starting. 2024-04-26T04:21:20.134 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29370]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-29369 2024-04-26T04:21:20.135 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29370]: created object 0... 2024-04-26T04:21:20.135 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29370]: created object 25... 2024-04-26T04:21:20.135 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29370]: created object 49... 2024-04-26T04:21:20.135 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29370]: finishing. 2024-04-26T04:21:20.135 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29370]: shutting down. 2024-04-26T04:21:20.918 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[29371]: starting. 2024-04-26T04:21:20.918 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[29371]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-29369 2024-04-26T04:21:20.918 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[29371]: shutting down. 2024-04-26T04:21:23.372 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:21:23.372 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29404]: starting. 2024-04-26T04:21:23.372 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29404]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-29369 2024-04-26T04:21:23.372 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29404]: created object 0... 2024-04-26T04:21:23.372 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29404]: created object 25... 2024-04-26T04:21:23.372 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29404]: created object 49... 2024-04-26T04:21:23.372 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29404]: finishing. 2024-04-26T04:21:23.372 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29404]: shutting down. 2024-04-26T04:21:23.429 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:21:23.430 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[29406]: starting. 2024-04-26T04:21:23.430 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[29406]: listing objects. 2024-04-26T04:21:23.430 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[29406]: listed object 0... 2024-04-26T04:21:23.430 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[29406]: listed object 25... 2024-04-26T04:21:23.430 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[29406]: saw 50 objects 2024-04-26T04:21:23.430 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[29406]: shutting down. 2024-04-26T04:21:24.164 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:21:24.165 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[29405]: starting. 2024-04-26T04:21:24.165 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[29405]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-29369 2024-04-26T04:21:24.165 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[29405]: shutting down. 2024-04-26T04:21:24.168 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:21:24.168 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:21:24.168 INFO:teuthology.orchestra.run.smithi186.stdout:******* SUCCESS ********** 2024-04-26T04:21:24.171 DEBUG:teuthology.orchestra.run.smithi186:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-26T04:21:26.376 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29457]: starting. 2024-04-26T04:21:26.376 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29457]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-29456 2024-04-26T04:21:26.376 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29457]: created object 0... 2024-04-26T04:21:26.376 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29457]: created object 25... 2024-04-26T04:21:26.376 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29457]: created object 49... 2024-04-26T04:21:26.376 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29457]: finishing. 2024-04-26T04:21:26.376 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29457]: shutting down. 2024-04-26T04:21:27.173 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[29458]: starting. 2024-04-26T04:21:27.173 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[29458]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-29456 2024-04-26T04:21:27.173 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[29458]: shutting down. 2024-04-26T04:21:29.132 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:21:29.132 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29491]: starting. 2024-04-26T04:21:29.132 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29491]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-29456 2024-04-26T04:21:29.133 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29491]: created object 0... 2024-04-26T04:21:29.133 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29491]: created object 25... 2024-04-26T04:21:29.133 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29491]: created object 49... 2024-04-26T04:21:29.133 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29491]: finishing. 2024-04-26T04:21:29.133 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29491]: shutting down. 2024-04-26T04:21:29.188 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:21:29.188 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[29493]: starting. 2024-04-26T04:21:29.188 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[29493]: listing objects. 2024-04-26T04:21:29.188 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[29493]: listed object 0... 2024-04-26T04:21:29.188 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[29493]: listed object 25... 2024-04-26T04:21:29.188 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[29493]: saw 50 objects 2024-04-26T04:21:29.188 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[29493]: shutting down. 2024-04-26T04:21:29.921 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:21:29.921 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[29492]: starting. 2024-04-26T04:21:29.921 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[29492]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-29456 2024-04-26T04:21:29.921 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[29492]: shutting down. 2024-04-26T04:21:29.925 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:21:29.925 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:21:29.925 INFO:teuthology.orchestra.run.smithi186.stdout:******* SUCCESS ********** 2024-04-26T04:21:29.929 DEBUG:teuthology.orchestra.run.smithi186:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-26T04:21:32.420 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29544]: starting. 2024-04-26T04:21:32.420 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29544]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-29543 2024-04-26T04:21:32.420 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29544]: created object 0... 2024-04-26T04:21:32.420 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29544]: created object 25... 2024-04-26T04:21:32.420 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29544]: created object 49... 2024-04-26T04:21:32.420 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29544]: finishing. 2024-04-26T04:21:32.421 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29544]: shutting down. 2024-04-26T04:21:33.194 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[29545]: starting. 2024-04-26T04:21:33.194 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[29545]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-29543 2024-04-26T04:21:33.194 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[29545]: shutting down. 2024-04-26T04:21:35.443 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:21:35.444 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29578]: starting. 2024-04-26T04:21:35.444 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29578]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-29543 2024-04-26T04:21:35.444 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29578]: created object 0... 2024-04-26T04:21:35.444 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29578]: created object 25... 2024-04-26T04:21:35.444 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29578]: created object 49... 2024-04-26T04:21:35.444 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29578]: finishing. 2024-04-26T04:21:35.444 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29578]: shutting down. 2024-04-26T04:21:35.504 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:21:35.504 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[29580]: starting. 2024-04-26T04:21:35.504 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[29580]: listing objects. 2024-04-26T04:21:35.504 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[29580]: listed object 0... 2024-04-26T04:21:35.504 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[29580]: listed object 25... 2024-04-26T04:21:35.504 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[29580]: saw 50 objects 2024-04-26T04:21:35.504 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[29580]: shutting down. 2024-04-26T04:21:36.225 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:21:36.225 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[29579]: starting. 2024-04-26T04:21:36.226 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[29579]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-29543 2024-04-26T04:21:36.226 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[29579]: shutting down. 2024-04-26T04:21:36.229 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:21:36.229 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:21:36.229 INFO:teuthology.orchestra.run.smithi186.stdout:******* SUCCESS ********** 2024-04-26T04:21:36.232 DEBUG:teuthology.orchestra.run.smithi186:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-26T04:21:38.471 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29631]: starting. 2024-04-26T04:21:38.472 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29631]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-29630 2024-04-26T04:21:38.472 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29631]: created object 0... 2024-04-26T04:21:38.472 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29631]: created object 25... 2024-04-26T04:21:38.472 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29631]: created object 49... 2024-04-26T04:21:38.472 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29631]: finishing. 2024-04-26T04:21:38.472 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29631]: shutting down. 2024-04-26T04:21:38.921 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[29632]: starting. 2024-04-26T04:21:38.921 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[29632]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-29630 2024-04-26T04:21:38.921 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[29632]: shutting down. 2024-04-26T04:21:41.158 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:21:41.159 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29665]: starting. 2024-04-26T04:21:41.159 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29665]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-29630 2024-04-26T04:21:41.159 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29665]: created object 0... 2024-04-26T04:21:41.160 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29665]: created object 25... 2024-04-26T04:21:41.160 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29665]: created object 49... 2024-04-26T04:21:41.160 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29665]: finishing. 2024-04-26T04:21:41.160 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29665]: shutting down. 2024-04-26T04:21:41.218 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:21:41.218 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[29667]: starting. 2024-04-26T04:21:41.219 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[29667]: listing objects. 2024-04-26T04:21:41.219 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[29667]: listed object 0... 2024-04-26T04:21:41.219 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[29667]: listed object 25... 2024-04-26T04:21:41.219 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[29667]: saw 50 objects 2024-04-26T04:21:41.219 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[29667]: shutting down. 2024-04-26T04:21:41.938 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:21:41.938 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[29666]: starting. 2024-04-26T04:21:41.938 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[29666]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-29630 2024-04-26T04:21:41.938 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[29666]: shutting down. 2024-04-26T04:21:41.942 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:21:41.942 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:21:41.943 INFO:teuthology.orchestra.run.smithi186.stdout:******* SUCCESS ********** 2024-04-26T04:21:41.944 DEBUG:teuthology.orchestra.run.smithi186:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-26T04:21:44.190 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29718]: starting. 2024-04-26T04:21:44.190 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29718]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-29717 2024-04-26T04:21:44.190 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29718]: created object 0... 2024-04-26T04:21:44.190 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29718]: created object 25... 2024-04-26T04:21:44.190 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29718]: created object 49... 2024-04-26T04:21:44.190 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29718]: finishing. 2024-04-26T04:21:44.191 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29718]: shutting down. 2024-04-26T04:21:46.001 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[29719]: starting. 2024-04-26T04:21:46.001 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[29719]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-29717 2024-04-26T04:21:46.001 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[29719]: shutting down. 2024-04-26T04:21:48.219 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:21:48.219 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29752]: starting. 2024-04-26T04:21:48.219 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29752]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-29717 2024-04-26T04:21:48.219 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29752]: created object 0... 2024-04-26T04:21:48.220 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29752]: created object 25... 2024-04-26T04:21:48.220 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29752]: created object 49... 2024-04-26T04:21:48.220 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29752]: finishing. 2024-04-26T04:21:48.220 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29752]: shutting down. 2024-04-26T04:21:48.276 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:21:48.276 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[29754]: starting. 2024-04-26T04:21:48.276 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[29754]: listing objects. 2024-04-26T04:21:48.276 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[29754]: listed object 0... 2024-04-26T04:21:48.276 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[29754]: listed object 25... 2024-04-26T04:21:48.276 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[29754]: saw 50 objects 2024-04-26T04:21:48.276 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[29754]: shutting down. 2024-04-26T04:21:48.924 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:21:48.924 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[29753]: starting. 2024-04-26T04:21:48.924 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[29753]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-29717 2024-04-26T04:21:48.924 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[29753]: shutting down. 2024-04-26T04:21:48.927 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:21:48.927 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:21:48.927 INFO:teuthology.orchestra.run.smithi186.stdout:******* SUCCESS ********** 2024-04-26T04:21:48.931 DEBUG:teuthology.orchestra.run.smithi186:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-26T04:21:51.155 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29805]: starting. 2024-04-26T04:21:51.155 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29805]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-29804 2024-04-26T04:21:51.155 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29805]: created object 0... 2024-04-26T04:21:51.155 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29805]: created object 25... 2024-04-26T04:21:51.155 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29805]: created object 49... 2024-04-26T04:21:51.155 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29805]: finishing. 2024-04-26T04:21:51.155 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29805]: shutting down. 2024-04-26T04:21:51.951 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[29806]: starting. 2024-04-26T04:21:51.952 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[29806]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-29804 2024-04-26T04:21:51.952 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[29806]: shutting down. 2024-04-26T04:21:54.198 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:21:54.198 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29839]: starting. 2024-04-26T04:21:54.198 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29839]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-29804 2024-04-26T04:21:54.199 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29839]: created object 0... 2024-04-26T04:21:54.199 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29839]: created object 25... 2024-04-26T04:21:54.199 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29839]: created object 49... 2024-04-26T04:21:54.199 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29839]: finishing. 2024-04-26T04:21:54.199 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29839]: shutting down. 2024-04-26T04:21:54.262 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:21:54.262 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[29841]: starting. 2024-04-26T04:21:54.262 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[29841]: listing objects. 2024-04-26T04:21:54.262 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[29841]: listed object 0... 2024-04-26T04:21:54.262 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[29841]: listed object 25... 2024-04-26T04:21:54.262 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[29841]: saw 50 objects 2024-04-26T04:21:54.262 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[29841]: shutting down. 2024-04-26T04:21:54.973 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:21:54.973 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[29840]: starting. 2024-04-26T04:21:54.973 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[29840]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-29804 2024-04-26T04:21:54.973 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[29840]: shutting down. 2024-04-26T04:21:54.975 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:21:54.975 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:21:54.975 INFO:teuthology.orchestra.run.smithi186.stdout:******* SUCCESS ********** 2024-04-26T04:21:54.979 DEBUG:teuthology.orchestra.run.smithi186:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-26T04:21:57.411 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29892]: starting. 2024-04-26T04:21:57.411 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29892]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-29891 2024-04-26T04:21:57.411 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29892]: created object 0... 2024-04-26T04:21:57.412 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29892]: created object 25... 2024-04-26T04:21:57.412 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29892]: created object 49... 2024-04-26T04:21:57.412 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29892]: finishing. 2024-04-26T04:21:57.412 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29892]: shutting down. 2024-04-26T04:21:58.206 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[29893]: starting. 2024-04-26T04:21:58.207 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[29893]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-29891 2024-04-26T04:21:58.207 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[29893]: shutting down. 2024-04-26T04:22:00.147 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:22:00.147 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29926]: starting. 2024-04-26T04:22:00.147 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29926]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-29891 2024-04-26T04:22:00.148 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29926]: created object 0... 2024-04-26T04:22:00.148 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29926]: created object 25... 2024-04-26T04:22:00.148 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29926]: created object 49... 2024-04-26T04:22:00.148 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29926]: finishing. 2024-04-26T04:22:00.148 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[29926]: shutting down. 2024-04-26T04:22:00.203 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:22:00.203 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[29928]: starting. 2024-04-26T04:22:00.203 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[29928]: listing objects. 2024-04-26T04:22:00.203 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[29928]: listed object 0... 2024-04-26T04:22:00.203 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[29928]: listed object 25... 2024-04-26T04:22:00.203 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[29928]: saw 50 objects 2024-04-26T04:22:00.203 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[29928]: shutting down. 2024-04-26T04:22:00.943 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:22:00.943 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[29927]: starting. 2024-04-26T04:22:00.943 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[29927]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-29891 2024-04-26T04:22:00.943 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[29927]: shutting down. 2024-04-26T04:22:00.946 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:22:00.946 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:22:00.946 INFO:teuthology.orchestra.run.smithi186.stdout:******* SUCCESS ********** 2024-04-26T04:22:00.949 DEBUG:teuthology.orchestra.run.smithi186:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-26T04:22:03.160 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29979]: starting. 2024-04-26T04:22:03.160 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29979]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-29978 2024-04-26T04:22:03.160 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29979]: created object 0... 2024-04-26T04:22:03.160 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29979]: created object 25... 2024-04-26T04:22:03.160 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29979]: created object 49... 2024-04-26T04:22:03.160 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29979]: finishing. 2024-04-26T04:22:03.160 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[29979]: shutting down. 2024-04-26T04:22:03.967 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[29980]: starting. 2024-04-26T04:22:03.967 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[29980]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-29978 2024-04-26T04:22:03.967 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[29980]: shutting down. 2024-04-26T04:22:06.198 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:22:06.199 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30013]: starting. 2024-04-26T04:22:06.199 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30013]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-29978 2024-04-26T04:22:06.199 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30013]: created object 0... 2024-04-26T04:22:06.199 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30013]: created object 25... 2024-04-26T04:22:06.199 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30013]: created object 49... 2024-04-26T04:22:06.199 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30013]: finishing. 2024-04-26T04:22:06.199 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30013]: shutting down. 2024-04-26T04:22:06.261 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:22:06.262 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[30015]: starting. 2024-04-26T04:22:06.262 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[30015]: listing objects. 2024-04-26T04:22:06.262 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[30015]: listed object 0... 2024-04-26T04:22:06.262 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[30015]: listed object 25... 2024-04-26T04:22:06.262 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[30015]: saw 50 objects 2024-04-26T04:22:06.262 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[30015]: shutting down. 2024-04-26T04:22:06.991 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:22:06.991 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[30014]: starting. 2024-04-26T04:22:06.991 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[30014]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-29978 2024-04-26T04:22:06.991 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[30014]: shutting down. 2024-04-26T04:22:06.994 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:22:06.994 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:22:06.994 INFO:teuthology.orchestra.run.smithi186.stdout:******* SUCCESS ********** 2024-04-26T04:22:06.997 DEBUG:teuthology.orchestra.run.smithi186:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-26T04:22:09.131 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[30066]: starting. 2024-04-26T04:22:09.131 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[30066]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-30065 2024-04-26T04:22:09.131 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[30066]: created object 0... 2024-04-26T04:22:09.131 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[30066]: created object 25... 2024-04-26T04:22:09.131 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[30066]: created object 49... 2024-04-26T04:22:09.132 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[30066]: finishing. 2024-04-26T04:22:09.132 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[30066]: shutting down. 2024-04-26T04:22:09.934 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[30067]: starting. 2024-04-26T04:22:09.934 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[30067]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-30065 2024-04-26T04:22:09.934 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[30067]: shutting down. 2024-04-26T04:22:12.162 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:22:12.166 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30100]: starting. 2024-04-26T04:22:12.166 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30100]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-30065 2024-04-26T04:22:12.167 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30100]: created object 0... 2024-04-26T04:22:12.167 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30100]: created object 25... 2024-04-26T04:22:12.167 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30100]: created object 49... 2024-04-26T04:22:12.167 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30100]: finishing. 2024-04-26T04:22:12.167 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30100]: shutting down. 2024-04-26T04:22:12.222 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:22:12.222 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[30102]: starting. 2024-04-26T04:22:12.223 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[30102]: listing objects. 2024-04-26T04:22:12.223 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[30102]: listed object 0... 2024-04-26T04:22:12.223 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[30102]: listed object 25... 2024-04-26T04:22:12.223 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[30102]: saw 50 objects 2024-04-26T04:22:12.223 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[30102]: shutting down. 2024-04-26T04:22:12.951 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:22:12.952 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[30101]: starting. 2024-04-26T04:22:12.952 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[30101]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-30065 2024-04-26T04:22:12.952 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[30101]: shutting down. 2024-04-26T04:22:12.954 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:22:12.954 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:22:12.954 INFO:teuthology.orchestra.run.smithi186.stdout:******* SUCCESS ********** 2024-04-26T04:22:12.957 DEBUG:teuthology.orchestra.run.smithi186:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-26T04:22:15.176 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[30153]: starting. 2024-04-26T04:22:15.176 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[30153]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-30152 2024-04-26T04:22:15.176 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[30153]: created object 0... 2024-04-26T04:22:15.176 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[30153]: created object 25... 2024-04-26T04:22:15.176 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[30153]: created object 49... 2024-04-26T04:22:15.176 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[30153]: finishing. 2024-04-26T04:22:15.177 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[30153]: shutting down. 2024-04-26T04:22:15.989 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[30154]: starting. 2024-04-26T04:22:15.989 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[30154]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-30152 2024-04-26T04:22:15.989 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[30154]: shutting down. 2024-04-26T04:22:18.213 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:22:18.213 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30187]: starting. 2024-04-26T04:22:18.213 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30187]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-30152 2024-04-26T04:22:18.213 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30187]: created object 0... 2024-04-26T04:22:18.213 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30187]: created object 25... 2024-04-26T04:22:18.213 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30187]: created object 49... 2024-04-26T04:22:18.213 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30187]: finishing. 2024-04-26T04:22:18.213 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30187]: shutting down. 2024-04-26T04:22:18.275 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:22:18.275 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[30189]: starting. 2024-04-26T04:22:18.275 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[30189]: listing objects. 2024-04-26T04:22:18.275 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[30189]: listed object 0... 2024-04-26T04:22:18.275 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[30189]: listed object 25... 2024-04-26T04:22:18.275 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[30189]: saw 50 objects 2024-04-26T04:22:18.275 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[30189]: shutting down. 2024-04-26T04:22:18.933 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:22:18.933 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[30188]: starting. 2024-04-26T04:22:18.933 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[30188]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-30152 2024-04-26T04:22:18.934 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[30188]: shutting down. 2024-04-26T04:22:18.936 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:22:18.936 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:22:18.936 INFO:teuthology.orchestra.run.smithi186.stdout:******* SUCCESS ********** 2024-04-26T04:22:18.939 DEBUG:teuthology.orchestra.run.smithi186:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-26T04:22:21.163 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[30240]: starting. 2024-04-26T04:22:21.163 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[30240]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-30239 2024-04-26T04:22:21.163 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[30240]: created object 0... 2024-04-26T04:22:21.163 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[30240]: created object 25... 2024-04-26T04:22:21.163 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[30240]: created object 49... 2024-04-26T04:22:21.164 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[30240]: finishing. 2024-04-26T04:22:21.164 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[30240]: shutting down. 2024-04-26T04:22:21.971 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[30241]: starting. 2024-04-26T04:22:21.972 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[30241]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-30239 2024-04-26T04:22:21.972 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[30241]: shutting down. 2024-04-26T04:22:24.198 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:22:24.199 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30274]: starting. 2024-04-26T04:22:24.199 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30274]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-30239 2024-04-26T04:22:24.199 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30274]: created object 0... 2024-04-26T04:22:24.199 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30274]: created object 25... 2024-04-26T04:22:24.199 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30274]: created object 49... 2024-04-26T04:22:24.199 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30274]: finishing. 2024-04-26T04:22:24.199 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30274]: shutting down. 2024-04-26T04:22:24.254 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:22:24.254 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[30276]: starting. 2024-04-26T04:22:24.254 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[30276]: listing objects. 2024-04-26T04:22:24.254 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[30276]: listed object 0... 2024-04-26T04:22:24.254 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[30276]: listed object 25... 2024-04-26T04:22:24.254 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[30276]: saw 50 objects 2024-04-26T04:22:24.255 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[30276]: shutting down. 2024-04-26T04:22:25.000 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:22:25.000 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[30275]: starting. 2024-04-26T04:22:25.000 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[30275]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-30239 2024-04-26T04:22:25.000 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[30275]: shutting down. 2024-04-26T04:22:25.004 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:22:25.004 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:22:25.004 INFO:teuthology.orchestra.run.smithi186.stdout:******* SUCCESS ********** 2024-04-26T04:22:25.008 DEBUG:teuthology.orchestra.run.smithi186:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-26T04:22:27.239 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[30327]: starting. 2024-04-26T04:22:27.240 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[30327]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-30326 2024-04-26T04:22:27.240 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[30327]: created object 0... 2024-04-26T04:22:27.240 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[30327]: created object 25... 2024-04-26T04:22:27.240 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[30327]: created object 49... 2024-04-26T04:22:27.240 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[30327]: finishing. 2024-04-26T04:22:27.240 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[30327]: shutting down. 2024-04-26T04:22:28.025 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[30328]: starting. 2024-04-26T04:22:28.025 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[30328]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-30326 2024-04-26T04:22:28.026 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[30328]: shutting down. 2024-04-26T04:22:30.170 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:22:30.174 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30361]: starting. 2024-04-26T04:22:30.175 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30361]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-30326 2024-04-26T04:22:30.175 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30361]: created object 0... 2024-04-26T04:22:30.175 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30361]: created object 25... 2024-04-26T04:22:30.175 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30361]: created object 49... 2024-04-26T04:22:30.175 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30361]: finishing. 2024-04-26T04:22:30.175 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30361]: shutting down. 2024-04-26T04:22:30.234 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:22:30.234 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[30363]: starting. 2024-04-26T04:22:30.234 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[30363]: listing objects. 2024-04-26T04:22:30.234 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[30363]: listed object 0... 2024-04-26T04:22:30.234 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[30363]: listed object 25... 2024-04-26T04:22:30.234 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[30363]: saw 50 objects 2024-04-26T04:22:30.235 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[30363]: shutting down. 2024-04-26T04:22:30.956 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:22:30.957 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[30362]: starting. 2024-04-26T04:22:30.957 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[30362]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-30326 2024-04-26T04:22:30.957 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[30362]: shutting down. 2024-04-26T04:22:30.958 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:22:30.959 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:22:30.959 INFO:teuthology.orchestra.run.smithi186.stdout:******* SUCCESS ********** 2024-04-26T04:22:30.961 DEBUG:teuthology.orchestra.run.smithi186:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-26T04:22:32.325 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-04-26T04:22:32.326 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-04-26T04:22:32.326 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-26T04:22:32.326 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-04-26T04:22:32.329 INFO:tasks.ceph.mon.c:Started 2024-04-26T04:22:32.329 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-26T04:22:32.329 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T04:22:32.463 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.460+0000 7f7cfadf5640 10 monclient: get_monmap_and_config 2024-04-26T04:22:32.463 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.460+0000 7f7cfadf5640 10 monclient: build_initial_monmap 2024-04-26T04:22:32.463 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.460+0000 7f7cfadf5640 10 monclient: monmap: 2024-04-26T04:22:32.463 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:22:32.464 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:32.464 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:22:32.463395+0000 2024-04-26T04:22:32.464 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:22:32.463395+0000 2024-04-26T04:22:32.464 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:22:32.464 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:32.464 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:22:32.464 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:22:32.464 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:22:32.464 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:22:32.464 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:22:32.464 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:32.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.460+0000 7f7cfadf5640 1 Processor -- start 2024-04-26T04:22:32.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.460+0000 7f7cfadf5640 1 -- start start 2024-04-26T04:22:32.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.460+0000 7f7cfadf5640 10 monclient: init 2024-04-26T04:22:32.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.460+0000 7f7cfadf5640 10 monclient: _reopen_session rank -1 2024-04-26T04:22:32.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.460+0000 7f7cfadf5640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-04-26T04:22:32.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.460+0000 7f7cfadf5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7cf40a2cd0 addr v1:172.21.15.186:6789/0 2024-04-26T04:22:32.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.460+0000 7f7cfadf5640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f7cf40a30b0 0x7f7cf413c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:32.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.460+0000 7f7cfadf5640 10 monclient(hunting): picked mon.noname-a con 0x7f7cf40a30b0 addr v2:172.21.15.63:3300/0 2024-04-26T04:22:32.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.460+0000 7f7cfadf5640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f7cf413c540 0x7f7cf413e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:32.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.460+0000 7f7cfadf5640 10 monclient(hunting): picked mon.noname-b con 0x7f7cf413c540 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:22:32.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.460+0000 7f7cfadf5640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7cf40a4ee0 con 0x7f7cf40a2cd0 2024-04-26T04:22:32.468 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.460+0000 7f7cfadf5640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:32.468 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.460+0000 7f7cfadf5640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7f7cf40a0fb0 con 0x7f7cf40a30b0 2024-04-26T04:22:32.468 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.460+0000 7f7cfadf5640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:32.468 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.460+0000 7f7cfadf5640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f7cf40a5380 con 0x7f7cf413c540 2024-04-26T04:22:32.468 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.460+0000 7f7cfadf5640 10 monclient(hunting): _renew_subs 2024-04-26T04:22:32.468 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.460+0000 7f7cfadf5640 10 monclient(hunting): authenticate will time out at 1184.170867s 2024-04-26T04:22:32.468 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.460+0000 7f7cf9392640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f7cf413c540 0x7f7cf413e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:32.468 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.460+0000 7f7cf3fff640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f7cf40a30b0 0x7f7cf413c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:32.468 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.460+0000 7f7cf9392640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f7cf413c540 0x7f7cf413e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:49936/0 (socket says 172.21.15.63:49936) 2024-04-26T04:22:32.468 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.460+0000 7f7cf9392640 1 -- 172.21.15.63:0/1487404656 learned_addr learned my addr 172.21.15.63:0/1487404656 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:22:32.468 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.460+0000 7f7cf3fff640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f7cf40a30b0 0x7f7cf413c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:48296/0 (socket says 172.21.15.63:48296) 2024-04-26T04:22:32.468 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.460+0000 7f7cf9392640 10 monclient(hunting): get_auth_request con 0x7f7cf413c540 auth_method 0 2024-04-26T04:22:32.468 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.460+0000 7f7cf9392640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:32.469 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.460+0000 7f7cf9392640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:32.469 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.460+0000 7f7cf9392640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:32.469 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.460+0000 7f7cf3fff640 10 monclient(hunting): get_auth_request con 0x7f7cf40a30b0 auth_method 0 2024-04-26T04:22:32.469 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.460+0000 7f7cf3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:32.469 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.460+0000 7f7cf3fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:32.469 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.460+0000 7f7cf3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:32.469 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.460+0000 7f7cf3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:22:32.469 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.460+0000 7f7cf3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:22:32.469 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.460+0000 7f7cf3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:22:32.469 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.460+0000 7f7cf9392640 1 -- 172.21.15.63:0/1487404656 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f7cf413c540 msgr2=0x7f7cf413e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T04:22:32.469 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.460+0000 7f7cf9392640 1 -- 172.21.15.63:0/1487404656 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f7cf413c540 msgr2=0x7f7cf413e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:22:32.469 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.460+0000 7f7cf9392640 1 --2- 172.21.15.63:0/1487404656 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f7cf413c540 0x7f7cf413e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:22:32.469 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.460+0000 7f7cf9392640 1 --2- 172.21.15.63:0/1487404656 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f7cf413c540 0x7f7cf413e930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:22:32.469 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.460+0000 7f7cf3fff640 10 monclient(hunting): handle_auth_done global_id 5166 payload 274 2024-04-26T04:22:32.469 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.460+0000 7f7cf3fff640 1 -- 172.21.15.63:0/1487404656 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f7cf413c540 msgr2=0x7f7cf413e930 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T04:22:32.470 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.460+0000 7f7cf3fff640 1 --2- 172.21.15.63:0/1487404656 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f7cf413c540 0x7f7cf413e930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:32.470 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.460+0000 7f7cf3fff640 1 -- 172.21.15.63:0/1487404656 >> v1:172.21.15.186:6789/0 conn(0x7f7cf40a2cd0 legacy=0x7f7cf40a47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:22:32.470 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.460+0000 7f7cf3fff640 10 monclient: _finish_hunting 0 2024-04-26T04:22:32.470 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.460+0000 7f7cf3fff640 1 monclient: found mon.noname-a 2024-04-26T04:22:32.470 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.460+0000 7f7cf3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:22:32.470 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.460+0000 7f7cf3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:22:32.470 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.460+0000 7f7cf3fff640 1 -- 172.21.15.63:0/1487404656 --> v2:172.21.15.63:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cf40a1290 con 0x7f7cf40a30b0 2024-04-26T04:22:32.474 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cf3fff640 1 -- 172.21.15.63:0/1487404656 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7cf40a30b0 msgr2=0x7f7cf413c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:22:32.474 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cf3fff640 1 --2- 172.21.15.63:0/1487404656 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7cf40a30b0 0x7f7cf413c000 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f7ce00097c0 tx=0x7f7ce001c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d17f8381cf2af0f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:22:32.474 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cf37fe640 1 -- 172.21.15.63:0/1487404656 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ce00b0070 con 0x7f7cf40a30b0 2024-04-26T04:22:32.474 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cf37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:32.474 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cf37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:22:32.474 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cf37fe640 10 monclient: dump: 2024-04-26T04:22:32.474 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:32.474 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:32.474 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:32.474 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:32.474 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:32.475 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:32.475 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:32.475 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:32.475 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:32.475 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:32.475 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cf37fe640 10 monclient: _finish_auth 0 2024-04-26T04:22:32.475 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cf37fe640 10 monclient: _check_auth_tickets 2024-04-26T04:22:32.475 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cf37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:22:32.475 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cfadf5640 5 monclient: authenticate success, global_id 5166 2024-04-26T04:22:32.475 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cf37fe640 1 -- 172.21.15.63:0/1487404656 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ce00ab600 con 0x7f7cf40a30b0 2024-04-26T04:22:32.475 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cf37fe640 10 monclient: handle_config config(0 keys) 2024-04-26T04:22:32.475 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cfadf5640 10 monclient: get_monmap_and_config success 2024-04-26T04:22:32.475 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cf37fe640 1 -- 172.21.15.63:0/1487404656 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ce00ab9d0 con 0x7f7cf40a30b0 2024-04-26T04:22:32.475 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cf37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:32.475 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cfadf5640 4 set_mon_vals no callback set 2024-04-26T04:22:32.476 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cf37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:22:32.476 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cf37fe640 10 monclient: dump: 2024-04-26T04:22:32.476 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:32.476 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:32.476 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:32.476 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:32.476 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:32.476 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:32.476 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:32.476 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:32.476 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:32.476 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:32.476 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cfadf5640 10 monclient: shutdown 2024-04-26T04:22:32.476 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cfadf5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:22:32.477 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cfadf5640 1 -- 172.21.15.63:0/1487404656 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7cf40a30b0 msgr2=0x7f7cf413c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:32.477 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cfadf5640 1 --2- 172.21.15.63:0/1487404656 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7cf40a30b0 0x7f7cf413c000 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f7ce00097c0 tx=0x7f7ce001c3b0 comp rx=0 tx=0).stop 2024-04-26T04:22:32.477 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cfadf5640 1 -- 172.21.15.63:0/1487404656 shutdown_connections 2024-04-26T04:22:32.477 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cfadf5640 1 --2- 172.21.15.63:0/1487404656 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f7cf413c540 0x7f7cf413e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:32.477 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cfadf5640 1 --2- 172.21.15.63:0/1487404656 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7cf40a30b0 0x7f7cf413c000 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:32.477 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cfadf5640 1 -- 172.21.15.63:0/1487404656 >> 172.21.15.63:0/1487404656 conn(0x7f7cf40a5f40 msgr2=0x7f7cf40a6320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:22:32.477 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cfadf5640 1 -- 172.21.15.63:0/1487404656 shutdown_connections 2024-04-26T04:22:32.477 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cfadf5640 1 -- 172.21.15.63:0/1487404656 wait complete. 2024-04-26T04:22:32.477 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cfadf5640 10 monclient: build_initial_monmap 2024-04-26T04:22:32.477 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cfadf5640 10 monclient: monmap: 2024-04-26T04:22:32.477 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:22:32.477 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:22:32.477 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:22:32.477 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:22:32.477 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:22:32.478 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:32.478 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:22:32.478 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:22:32.478 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:22:32.478 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:32.478 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cfadf5640 1 Processor -- start 2024-04-26T04:22:32.478 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cfadf5640 1 -- start start 2024-04-26T04:22:32.478 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cfadf5640 10 monclient: init 2024-04-26T04:22:32.478 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cfadf5640 10 monclient: _reopen_session rank -1 2024-04-26T04:22:32.478 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cfadf5640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T04:22:32.478 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cfadf5640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f7cf40a2cd0 0x7f7cf41cef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:32.478 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cfadf5640 10 monclient(hunting): picked mon.noname-c con 0x7f7cf40a2cd0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:22:32.478 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cfadf5640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7cf40a30b0 0x7f7cf41cf5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:32.479 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cfadf5640 10 monclient(hunting): picked mon.noname-a con 0x7f7cf40a30b0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:22:32.479 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cfadf5640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7cf413c540 0x7f7cf4070080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:32.479 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cfadf5640 10 monclient(hunting): picked mon.noname-b con 0x7f7cf413c540 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:22:32.479 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cfadf5640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:32.479 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cfadf5640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f7cf4141a20 con 0x7f7cf40a30b0 2024-04-26T04:22:32.479 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cfadf5640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:32.479 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cfadf5640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7cf41418a0 con 0x7f7cf413c540 2024-04-26T04:22:32.479 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cfadf5640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:32.479 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cfadf5640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f7cf4141ba0 con 0x7f7cf40a2cd0 2024-04-26T04:22:32.479 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cfadf5640 10 monclient(hunting): _renew_subs 2024-04-26T04:22:32.479 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cfadf5640 10 monclient(hunting): authenticate will time out at 1184.178689s 2024-04-26T04:22:32.479 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cf8b91640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f7cf40a2cd0 0x7f7cf41cef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:32.479 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cf3fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7cf40a30b0 0x7f7cf41cf5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:32.479 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cf8b91640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f7cf40a2cd0 0x7f7cf41cef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:49944/0 (socket says 172.21.15.63:49944) 2024-04-26T04:22:32.479 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cf8b91640 1 -- 172.21.15.63:0/3860403870 learned_addr learned my addr 172.21.15.63:0/3860403870 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:22:32.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cf8b91640 10 monclient(hunting): get_auth_request con 0x7f7cf40a2cd0 auth_method 0 2024-04-26T04:22:32.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cf8b91640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:32.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cf8b91640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:32.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cf8b91640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:32.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cf3fff640 10 monclient(hunting): get_auth_request con 0x7f7cf40a30b0 auth_method 0 2024-04-26T04:22:32.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cf3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:32.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cf3fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:32.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cf3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:32.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cf8b91640 1 -- 172.21.15.63:0/3860403870 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f7cf40a2cd0 msgr2=0x7f7cf41cef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T04:22:32.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cf8b91640 1 -- 172.21.15.63:0/3860403870 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f7cf40a2cd0 msgr2=0x7f7cf41cef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T04:22:32.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cf8b91640 1 --2- 172.21.15.63:0/3860403870 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f7cf40a2cd0 0x7f7cf41cef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:22:32.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cf3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:22:32.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cf3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:22:32.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cf3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:22:32.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cf8b91640 1 --2- 172.21.15.63:0/3860403870 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f7cf40a2cd0 0x7f7cf41cef30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:22:32.481 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cf9392640 1 --2- 172.21.15.63:0/3860403870 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7cf413c540 0x7f7cf4070080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:32.481 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cf9392640 10 monclient(hunting): get_auth_request con 0x7f7cf413c540 auth_method 0 2024-04-26T04:22:32.481 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cf9392640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:32.481 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cf9392640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:32.481 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cf9392640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:32.481 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cf3fff640 10 monclient(hunting): handle_auth_done global_id 5169 payload 824 2024-04-26T04:22:32.481 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cf3fff640 1 -- 172.21.15.63:0/3860403870 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f7cf40a2cd0 msgr2=0x7f7cf41cef30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T04:22:32.481 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cf3fff640 1 --2- 172.21.15.63:0/3860403870 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f7cf40a2cd0 0x7f7cf41cef30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:32.481 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cf3fff640 1 -- 172.21.15.63:0/3860403870 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7cf413c540 msgr2=0x7f7cf4070080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:32.481 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cf3fff640 1 --2- 172.21.15.63:0/3860403870 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7cf413c540 0x7f7cf4070080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:32.481 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cf3fff640 10 monclient: _finish_hunting 0 2024-04-26T04:22:32.481 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cf3fff640 1 monclient: found mon.noname-a 2024-04-26T04:22:32.481 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cf3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:22:32.481 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cf3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:22:32.481 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.468+0000 7f7cf3fff640 1 -- 172.21.15.63:0/3860403870 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cf4120030 con 0x7f7cf40a30b0 2024-04-26T04:22:32.483 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.472+0000 7f7cf3fff640 1 -- 172.21.15.63:0/3860403870 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7cf40a30b0 msgr2=0x7f7cf41cf5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:22:32.484 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.472+0000 7f7cf3fff640 1 --2- 172.21.15.63:0/3860403870 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7cf40a30b0 0x7f7cf41cf5d0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f7ce00ad000 tx=0x7f7ce0019150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:22:32.484 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.472+0000 7f7cf17fa640 1 -- 172.21.15.63:0/3860403870 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ce00b9020 con 0x7f7cf40a30b0 2024-04-26T04:22:32.484 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.472+0000 7f7cf17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:32.484 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.472+0000 7f7cf17fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:22:32.484 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.472+0000 7f7cf17fa640 10 monclient: dump: 2024-04-26T04:22:32.484 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:32.484 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:32.484 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:32.484 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:32.484 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:32.484 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:32.484 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:32.484 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:32.485 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:32.485 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:32.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.472+0000 7f7cf17fa640 10 monclient: _finish_auth 0 2024-04-26T04:22:32.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.472+0000 7f7cf17fa640 10 monclient: _check_auth_tickets 2024-04-26T04:22:32.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.472+0000 7f7cf17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:22:32.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.472+0000 7f7cf17fa640 1 -- 172.21.15.63:0/3860403870 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ce00b5690 con 0x7f7cf40a30b0 2024-04-26T04:22:32.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.472+0000 7f7cf17fa640 10 monclient: handle_config config(0 keys) 2024-04-26T04:22:32.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.472+0000 7f7cf17fa640 1 -- 172.21.15.63:0/3860403870 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ce00b0040 con 0x7f7cf40a30b0 2024-04-26T04:22:32.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.472+0000 7f7cfadf5640 5 monclient: authenticate success, global_id 5169 2024-04-26T04:22:32.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.472+0000 7f7cfadf5640 10 monclient: _renew_subs 2024-04-26T04:22:32.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.472+0000 7f7cfadf5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:22:32.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.472+0000 7f7cfadf5640 1 -- 172.21.15.63:0/3860403870 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cf4120310 con 0x7f7cf40a30b0 2024-04-26T04:22:32.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.472+0000 7f7cf2ffd640 4 set_mon_vals no callback set 2024-04-26T04:22:32.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.472+0000 7f7cf17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:32.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.472+0000 7f7cf17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:22:32.486 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.472+0000 7f7cf17fa640 10 monclient: dump: 2024-04-26T04:22:32.486 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:32.486 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:32.486 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:32.486 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:32.486 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:32.486 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:32.486 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:32.486 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:32.486 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:32.486 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:32.486 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.472+0000 7f7cfadf5640 10 monclient: _renew_subs 2024-04-26T04:22:32.486 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.472+0000 7f7cfadf5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:22:32.486 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.472+0000 7f7cfadf5640 1 -- 172.21.15.63:0/3860403870 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cf40a7340 con 0x7f7cf40a30b0 2024-04-26T04:22:32.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.472+0000 7f7cfadf5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:22:32.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.472+0000 7f7cfadf5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:22:32.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.472+0000 7f7cfadf5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:22:32.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.472+0000 7f7cfadf5640 1 -- 172.21.15.63:0/3860403870 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7cf4062e10 con 0x7f7cf40a30b0 2024-04-26T04:22:32.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.472+0000 7f7cf17fa640 1 -- 172.21.15.63:0/3860403870 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f7ce00bc020 con 0x7f7cf40a30b0 2024-04-26T04:22:32.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.472+0000 7f7cf17fa640 1 --2- 172.21.15.63:0/3860403870 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f7cd8035f10 0x7f7cd80383d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:32.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.472+0000 7f7cf17fa640 1 -- 172.21.15.63:0/3860403870 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(206..206 src has 1..206) ==== 4738+0+0 (secure 0 0 0) 0x7f7ce00e2e00 con 0x7f7cf40a30b0 2024-04-26T04:22:32.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.472+0000 7f7cf8b91640 1 --2- 172.21.15.63:0/3860403870 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f7cd8035f10 0x7f7cd80383d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:32.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.472+0000 7f7cf8b91640 10 monclient: get_auth_request con 0x7f7cd8035f10 auth_method 0 2024-04-26T04:22:32.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.476+0000 7f7cf8b91640 1 --2- 172.21.15.63:0/3860403870 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f7cd8035f10 0x7f7cd80383d0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f7ce8007d20 tx=0x7f7ce8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:22:32.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.476+0000 7f7cf17fa640 1 -- 172.21.15.63:0/3860403870 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f7ce00e7070 con 0x7f7cf40a30b0 2024-04-26T04:22:32.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.476+0000 7f7cf17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:22:32.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.476+0000 7f7cf17fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:22:32.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.704+0000 7f7cfadf5640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T04:22:32.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.704+0000 7f7cfadf5640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T04:22:32.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.704+0000 7f7cfadf5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:22:32.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.704+0000 7f7cfadf5640 1 -- 172.21.15.63:0/3860403870 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f7cf40683d0 con 0x7f7cf40a30b0 2024-04-26T04:22:32.714 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.712+0000 7f7cf17fa640 1 -- 172.21.15.63:0/3860403870 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f7ce00c3030 con 0x7f7cf40a30b0 2024-04-26T04:22:32.714 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.712+0000 7f7cf17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T04:22:32.714 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.712+0000 7f7cf17fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:22:32.714 INFO:teuthology.orchestra.run.smithi063.stdout:{"election_epoch":20,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cd1c6a0c-4e4b-4014-89c0-2e535479ef39","modified":"2024-04-26T04:18:03.702339Z","created":"2024-04-26T04:18:03.702339Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3300","nonce":0},{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:3300","nonce":0},{"type":"v1","addr":"172.21.15.186:6789","nonce":0}]},"addr":"172.21.15.186:6789/0","public_addr":"172.21.15.186:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3301","nonce":0},{"type":"v1","addr":"172.21.15.63:6790","nonce":0}]},"addr":"172.21.15.63:6790/0","public_addr":"172.21.15.63:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T04:22:32.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.712+0000 7f7cfadf5640 1 -- 172.21.15.63:0/3860403870 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f7cd8035f10 msgr2=0x7f7cd80383d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:32.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.712+0000 7f7cfadf5640 1 --2- 172.21.15.63:0/3860403870 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f7cd8035f10 0x7f7cd80383d0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f7ce8007d20 tx=0x7f7ce8009040 comp rx=0 tx=0).stop 2024-04-26T04:22:32.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.712+0000 7f7cfadf5640 10 monclient: shutdown 2024-04-26T04:22:32.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.712+0000 7f7cfadf5640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:22:32.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.712+0000 7f7cfadf5640 1 -- 172.21.15.63:0/3860403870 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7cf40a30b0 msgr2=0x7f7cf41cf5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:32.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.712+0000 7f7cfadf5640 1 --2- 172.21.15.63:0/3860403870 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7cf40a30b0 0x7f7cf41cf5d0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f7ce00ad000 tx=0x7f7ce0019150 comp rx=0 tx=0).stop 2024-04-26T04:22:32.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.712+0000 7f7cfadf5640 1 -- 172.21.15.63:0/3860403870 shutdown_connections 2024-04-26T04:22:32.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.712+0000 7f7cfadf5640 1 --2- 172.21.15.63:0/3860403870 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f7cd8035f10 0x7f7cd80383d0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:32.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.712+0000 7f7cfadf5640 1 --2- 172.21.15.63:0/3860403870 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7cf413c540 0x7f7cf4070080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:32.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.712+0000 7f7cfadf5640 1 --2- 172.21.15.63:0/3860403870 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7cf40a30b0 0x7f7cf41cf5d0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:32.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.712+0000 7f7cfadf5640 1 --2- 172.21.15.63:0/3860403870 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f7cf40a2cd0 0x7f7cf41cef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:32.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.712+0000 7f7cfadf5640 1 -- 172.21.15.63:0/3860403870 >> 172.21.15.63:0/3860403870 conn(0x7f7cf40a5f40 msgr2=0x7f7cf40a1a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:22:32.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.712+0000 7f7cfadf5640 1 -- 172.21.15.63:0/3860403870 shutdown_connections 2024-04-26T04:22:32.718 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.712+0000 7f7cfadf5640 1 -- 172.21.15.63:0/3860403870 wait complete. 2024-04-26T04:22:32.733 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-26T04:22:32.733 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-26T04:22:32.734 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T04:22:32.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.852+0000 7f127048c640 10 monclient: get_monmap_and_config 2024-04-26T04:22:32.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.852+0000 7f127048c640 10 monclient: build_initial_monmap 2024-04-26T04:22:32.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.852+0000 7f127048c640 10 monclient: monmap: 2024-04-26T04:22:32.854 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:22:32.854 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:32.854 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:22:32.854025+0000 2024-04-26T04:22:32.854 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:22:32.854025+0000 2024-04-26T04:22:32.854 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:22:32.854 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:32.854 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:22:32.854 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:22:32.855 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:22:32.855 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:22:32.855 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:22:32.855 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:32.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.852+0000 7f127048c640 1 Processor -- start 2024-04-26T04:22:32.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.852+0000 7f127048c640 1 -- start start 2024-04-26T04:22:32.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.852+0000 7f127048c640 10 monclient: init 2024-04-26T04:22:32.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.852+0000 7f127048c640 10 monclient: _reopen_session rank -1 2024-04-26T04:22:32.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.852+0000 7f127048c640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-04-26T04:22:32.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.852+0000 7f127048c640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f12680a2cd0 0x7f12680a30b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:32.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.852+0000 7f127048c640 10 monclient(hunting): picked mon.noname-a con 0x7f12680a2cd0 addr v2:172.21.15.63:3300/0 2024-04-26T04:22:32.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.852+0000 7f127048c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f12680a47d0 addr v1:172.21.15.186:6789/0 2024-04-26T04:22:32.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.852+0000 7f127048c640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f12680a4c00 0x7f12681373f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:32.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.852+0000 7f127048c640 10 monclient(hunting): picked mon.noname-b con 0x7f12680a4c00 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:22:32.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.852+0000 7f127048c640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12680a0fb0 con 0x7f12680a47d0 2024-04-26T04:22:32.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.852+0000 7f127048c640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:32.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.852+0000 7f127048c640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7f12680a35f0 con 0x7f12680a2cd0 2024-04-26T04:22:32.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.852+0000 7f127048c640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:32.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.852+0000 7f127048c640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f12680a5380 con 0x7f12680a4c00 2024-04-26T04:22:32.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.852+0000 7f127048c640 10 monclient(hunting): _renew_subs 2024-04-26T04:22:32.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.852+0000 7f127048c640 10 monclient(hunting): authenticate will time out at 1184.561834s 2024-04-26T04:22:32.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.852+0000 7f126e228640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f12680a2cd0 0x7f12680a30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:32.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.852+0000 7f126ea29640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f12680a4c00 0x7f12681373f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:32.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.852+0000 7f126ea29640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f12680a4c00 0x7f12681373f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:49952/0 (socket says 172.21.15.63:49952) 2024-04-26T04:22:32.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.852+0000 7f126ea29640 1 -- 172.21.15.63:0/581414966 learned_addr learned my addr 172.21.15.63:0/581414966 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:22:32.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.852+0000 7f126ea29640 10 monclient(hunting): get_auth_request con 0x7f12680a4c00 auth_method 0 2024-04-26T04:22:32.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.852+0000 7f126e228640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f12680a2cd0 0x7f12680a30b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:48318/0 (socket says 172.21.15.63:48318) 2024-04-26T04:22:32.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.852+0000 7f126ea29640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:32.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.852+0000 7f126ea29640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:32.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.852+0000 7f126ea29640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:32.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.852+0000 7f126e228640 10 monclient(hunting): get_auth_request con 0x7f12680a2cd0 auth_method 0 2024-04-26T04:22:32.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.852+0000 7f126e228640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:32.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.852+0000 7f126e228640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:32.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.852+0000 7f126e228640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:32.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.852+0000 7f126e228640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:22:32.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.852+0000 7f126e228640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:22:32.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.852+0000 7f126ea29640 1 -- 172.21.15.63:0/581414966 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f12680a4c00 msgr2=0x7f12681373f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T04:22:32.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.852+0000 7f126ea29640 1 -- 172.21.15.63:0/581414966 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f12680a4c00 msgr2=0x7f12681373f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:22:32.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.852+0000 7f126ea29640 1 --2- 172.21.15.63:0/581414966 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f12680a4c00 0x7f12681373f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:22:32.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.852+0000 7f126e228640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:22:32.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.852+0000 7f126ea29640 1 --2- 172.21.15.63:0/581414966 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f12680a4c00 0x7f12681373f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:22:32.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.852+0000 7f126e228640 10 monclient(hunting): handle_auth_done global_id 5172 payload 274 2024-04-26T04:22:32.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.852+0000 7f126e228640 1 -- 172.21.15.63:0/581414966 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f12680a4c00 msgr2=0x7f12681373f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T04:22:32.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.852+0000 7f126e228640 1 --2- 172.21.15.63:0/581414966 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f12680a4c00 0x7f12681373f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:32.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.852+0000 7f126e228640 1 -- 172.21.15.63:0/581414966 >> v1:172.21.15.186:6789/0 conn(0x7f12680a47d0 legacy=0x7f1268136ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:22:32.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.852+0000 7f126e228640 10 monclient: _finish_hunting 0 2024-04-26T04:22:32.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.852+0000 7f126e228640 1 monclient: found mon.noname-a 2024-04-26T04:22:32.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.852+0000 7f126e228640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:22:32.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.852+0000 7f126e228640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:22:32.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.852+0000 7f126e228640 1 -- 172.21.15.63:0/581414966 --> v2:172.21.15.63:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12680a1290 con 0x7f12680a2cd0 2024-04-26T04:22:32.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.856+0000 7f126e228640 1 -- 172.21.15.63:0/581414966 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f12680a2cd0 msgr2=0x7f12680a30b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:22:32.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.856+0000 7f126e228640 1 --2- 172.21.15.63:0/581414966 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f12680a2cd0 0x7f12680a30b0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f12640097c0 tx=0x7f126401c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d33f23cc5fb054e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:22:32.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.856+0000 7f126d226640 1 -- 172.21.15.63:0/581414966 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12640b0070 con 0x7f12680a2cd0 2024-04-26T04:22:32.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.856+0000 7f126d226640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:32.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.856+0000 7f126d226640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:22:32.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.856+0000 7f126d226640 10 monclient: dump: 2024-04-26T04:22:32.862 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:32.862 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:32.862 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:32.862 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:32.863 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:32.863 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:32.863 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:32.863 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:32.863 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:32.863 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:32.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.856+0000 7f126d226640 10 monclient: _finish_auth 0 2024-04-26T04:22:32.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.856+0000 7f126d226640 10 monclient: _check_auth_tickets 2024-04-26T04:22:32.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.856+0000 7f126d226640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:22:32.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.856+0000 7f126d226640 1 -- 172.21.15.63:0/581414966 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12640ab400 con 0x7f12680a2cd0 2024-04-26T04:22:32.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.856+0000 7f126d226640 10 monclient: handle_config config(0 keys) 2024-04-26T04:22:32.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.856+0000 7f126d226640 1 -- 172.21.15.63:0/581414966 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12640ab7d0 con 0x7f12680a2cd0 2024-04-26T04:22:32.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.856+0000 7f127048c640 5 monclient: authenticate success, global_id 5172 2024-04-26T04:22:32.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.856+0000 7f127048c640 10 monclient: get_monmap_and_config success 2024-04-26T04:22:32.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.856+0000 7f127048c640 4 set_mon_vals no callback set 2024-04-26T04:22:32.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.856+0000 7f126d226640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:32.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.856+0000 7f126d226640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:22:32.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.856+0000 7f126d226640 10 monclient: dump: 2024-04-26T04:22:32.864 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:32.864 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:32.864 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:32.864 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:32.864 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:32.864 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:32.864 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:32.864 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:32.864 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:32.864 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:32.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.856+0000 7f127048c640 10 monclient: shutdown 2024-04-26T04:22:32.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.856+0000 7f127048c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:22:32.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.856+0000 7f127048c640 1 -- 172.21.15.63:0/581414966 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f12680a2cd0 msgr2=0x7f12680a30b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:32.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.856+0000 7f127048c640 1 --2- 172.21.15.63:0/581414966 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f12680a2cd0 0x7f12680a30b0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f12640097c0 tx=0x7f126401c3b0 comp rx=0 tx=0).stop 2024-04-26T04:22:32.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.856+0000 7f127048c640 1 -- 172.21.15.63:0/581414966 shutdown_connections 2024-04-26T04:22:32.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.856+0000 7f127048c640 1 --2- 172.21.15.63:0/581414966 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f12680a4c00 0x7f12681373f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:32.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.856+0000 7f127048c640 1 --2- 172.21.15.63:0/581414966 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f12680a2cd0 0x7f12680a30b0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:32.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.856+0000 7f127048c640 1 -- 172.21.15.63:0/581414966 >> 172.21.15.63:0/581414966 conn(0x7f12680a5f40 msgr2=0x7f12680a6320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:22:32.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.856+0000 7f127048c640 1 -- 172.21.15.63:0/581414966 shutdown_connections 2024-04-26T04:22:32.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.856+0000 7f127048c640 1 -- 172.21.15.63:0/581414966 wait complete. 2024-04-26T04:22:32.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.860+0000 7f127048c640 10 monclient: build_initial_monmap 2024-04-26T04:22:32.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.860+0000 7f127048c640 10 monclient: monmap: 2024-04-26T04:22:32.865 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:22:32.865 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:22:32.865 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:22:32.865 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:22:32.866 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:22:32.866 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:32.866 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:22:32.866 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:22:32.866 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:22:32.866 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:32.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.860+0000 7f127048c640 1 Processor -- start 2024-04-26T04:22:32.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.860+0000 7f127048c640 1 -- start start 2024-04-26T04:22:32.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.860+0000 7f127048c640 10 monclient: init 2024-04-26T04:22:32.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.860+0000 7f127048c640 10 monclient: _reopen_session rank -1 2024-04-26T04:22:32.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.860+0000 7f127048c640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T04:22:32.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.860+0000 7f127048c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f12680a2cd0 0x7f12681cefb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:32.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.860+0000 7f127048c640 10 monclient(hunting): picked mon.noname-a con 0x7f12680a2cd0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:22:32.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.860+0000 7f127048c640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f12680a47d0 0x7f12681cf5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:32.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.860+0000 7f127048c640 10 monclient(hunting): picked mon.noname-c con 0x7f12680a47d0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:22:32.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.860+0000 7f127048c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12680a4c00 0x7f12681cfb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:32.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.860+0000 7f127048c640 10 monclient(hunting): picked mon.noname-b con 0x7f12680a4c00 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:22:32.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.860+0000 7f127048c640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:32.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.860+0000 7f127048c640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f1268141a50 con 0x7f12680a2cd0 2024-04-26T04:22:32.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.860+0000 7f127048c640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:32.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.860+0000 7f127048c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1268137b40 con 0x7f12680a4c00 2024-04-26T04:22:32.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.860+0000 7f127048c640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:32.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.860+0000 7f127048c640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f1268141bd0 con 0x7f12680a47d0 2024-04-26T04:22:32.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.860+0000 7f127048c640 10 monclient(hunting): _renew_subs 2024-04-26T04:22:32.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.860+0000 7f127048c640 10 monclient(hunting): authenticate will time out at 1184.569246s 2024-04-26T04:22:32.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.860+0000 7f126e228640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f12680a2cd0 0x7f12681cefb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:32.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.860+0000 7f126da27640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f12680a47d0 0x7f12681cf5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:32.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.860+0000 7f126e228640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f12680a2cd0 0x7f12681cefb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:48320/0 (socket says 172.21.15.63:48320) 2024-04-26T04:22:32.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.860+0000 7f126e228640 1 -- 172.21.15.63:0/3073060184 learned_addr learned my addr 172.21.15.63:0/3073060184 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:22:32.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.860+0000 7f126e228640 10 monclient(hunting): get_auth_request con 0x7f12680a2cd0 auth_method 0 2024-04-26T04:22:32.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.860+0000 7f126e228640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:32.873 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.860+0000 7f126e228640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:32.873 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.860+0000 7f126e228640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:32.873 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.860+0000 7f126da27640 10 monclient(hunting): get_auth_request con 0x7f12680a47d0 auth_method 0 2024-04-26T04:22:32.873 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.860+0000 7f126da27640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:32.873 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.860+0000 7f126da27640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:32.873 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.860+0000 7f126da27640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:32.873 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.860+0000 7f126e228640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:22:32.873 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.860+0000 7f126e228640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:22:32.873 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.860+0000 7f126e228640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:22:32.873 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.860+0000 7f126da27640 1 -- 172.21.15.63:0/3073060184 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f12680a47d0 msgr2=0x7f12681cf5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-26T04:22:32.873 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.860+0000 7f126da27640 1 -- 172.21.15.63:0/3073060184 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f12680a47d0 msgr2=0x7f12681cf5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T04:22:32.873 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.860+0000 7f126da27640 1 --2- 172.21.15.63:0/3073060184 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f12680a47d0 0x7f12681cf5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:22:32.873 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.860+0000 7f126da27640 1 --2- 172.21.15.63:0/3073060184 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f12680a47d0 0x7f12681cf5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:22:32.873 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.860+0000 7f126e228640 10 monclient(hunting): handle_auth_done global_id 5175 payload 824 2024-04-26T04:22:32.874 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.860+0000 7f126e228640 1 -- 172.21.15.63:0/3073060184 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f12680a47d0 msgr2=0x7f12681cf5c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T04:22:32.874 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.860+0000 7f126e228640 1 --2- 172.21.15.63:0/3073060184 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f12680a47d0 0x7f12681cf5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:32.874 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.860+0000 7f126e228640 1 -- 172.21.15.63:0/3073060184 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12680a4c00 msgr2=0x7f12681cfb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:32.874 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.860+0000 7f126e228640 1 --2- 172.21.15.63:0/3073060184 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12680a4c00 0x7f12681cfb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:32.874 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.860+0000 7f126e228640 10 monclient: _finish_hunting 0 2024-04-26T04:22:32.874 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.860+0000 7f126e228640 1 monclient: found mon.noname-a 2024-04-26T04:22:32.874 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.860+0000 7f126e228640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:22:32.874 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.860+0000 7f126e228640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:22:32.874 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.860+0000 7f126e228640 1 -- 172.21.15.63:0/3073060184 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1268120030 con 0x7f12680a2cd0 2024-04-26T04:22:32.874 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.860+0000 7f126e228640 1 -- 172.21.15.63:0/3073060184 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f12680a2cd0 msgr2=0x7f12681cefb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:22:32.874 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.860+0000 7f126e228640 1 --2- 172.21.15.63:0/3073060184 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f12680a2cd0 0x7f12681cefb0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f126401aa60 tx=0x7f126401abf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:22:32.874 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.864+0000 7f125effd640 1 -- 172.21.15.63:0/3073060184 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12640b0050 con 0x7f12680a2cd0 2024-04-26T04:22:32.874 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.864+0000 7f125effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:32.874 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.864+0000 7f125effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:22:32.874 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.864+0000 7f125effd640 10 monclient: dump: 2024-04-26T04:22:32.875 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:32.875 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:32.875 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:32.875 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:32.875 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:32.875 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:32.875 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:32.875 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:32.875 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:32.875 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:32.875 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.864+0000 7f125effd640 10 monclient: _finish_auth 0 2024-04-26T04:22:32.875 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.864+0000 7f125effd640 10 monclient: _check_auth_tickets 2024-04-26T04:22:32.875 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.864+0000 7f125effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:22:32.875 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.864+0000 7f125effd640 1 -- 172.21.15.63:0/3073060184 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12640bc070 con 0x7f12680a2cd0 2024-04-26T04:22:32.876 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.864+0000 7f127048c640 5 monclient: authenticate success, global_id 5175 2024-04-26T04:22:32.876 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.864+0000 7f125effd640 10 monclient: handle_config config(0 keys) 2024-04-26T04:22:32.876 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.864+0000 7f125effd640 1 -- 172.21.15.63:0/3073060184 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12640b5a70 con 0x7f12680a2cd0 2024-04-26T04:22:32.876 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.864+0000 7f127048c640 10 monclient: _renew_subs 2024-04-26T04:22:32.876 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.864+0000 7f127048c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:22:32.876 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.864+0000 7f127048c640 1 -- 172.21.15.63:0/3073060184 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1268120310 con 0x7f12680a2cd0 2024-04-26T04:22:32.876 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.864+0000 7f125effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:32.876 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.864+0000 7f125effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:22:32.876 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.864+0000 7f125effd640 10 monclient: dump: 2024-04-26T04:22:32.876 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:32.876 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:32.876 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:32.876 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:32.876 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:32.876 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:32.877 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:32.877 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:32.877 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:32.877 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:32.877 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.864+0000 7f127048c640 10 monclient: _renew_subs 2024-04-26T04:22:32.877 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.864+0000 7f127048c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:22:32.877 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.864+0000 7f127048c640 1 -- 172.21.15.63:0/3073060184 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f126813e390 con 0x7f12680a2cd0 2024-04-26T04:22:32.877 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.864+0000 7f126ca25640 4 set_mon_vals no callback set 2024-04-26T04:22:32.877 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.864+0000 7f127048c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:22:32.877 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.864+0000 7f127048c640 1 --2- 172.21.15.63:0/3073060184 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f126813bb90 0x7f126813bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:32.877 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.864+0000 7f127048c640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:32.877 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.864+0000 7f127048c640 1 -- 172.21.15.63:0/3073060184 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f1268072cf0 con 0x7f126813bb90 2024-04-26T04:22:32.877 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.864+0000 7f126da27640 1 --2- 172.21.15.63:0/3073060184 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f126813bb90 0x7f126813bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:32.877 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.864+0000 7f126da27640 10 monclient: get_auth_request con 0x7f126813bb90 auth_method 0 2024-04-26T04:22:32.877 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.864+0000 7f126da27640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:32.878 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.864+0000 7f126da27640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:32.878 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.864+0000 7f126da27640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:32.878 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.864+0000 7f125effd640 1 -- 172.21.15.63:0/3073060184 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f12640b5c10 con 0x7f12680a2cd0 2024-04-26T04:22:32.878 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.864+0000 7f126da27640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:22:32.878 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.864+0000 7f126da27640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:22:32.878 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.864+0000 7f126da27640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:22:32.878 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.864+0000 7f126da27640 10 monclient(hunting): handle_auth_done global_id 5178 payload 824 2024-04-26T04:22:32.878 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.864+0000 7f125effd640 1 --2- 172.21.15.63:0/3073060184 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f124c035f10 0x7f124c0383d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:32.878 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.864+0000 7f126da27640 1 -- 172.21.15.63:0/3073060184 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1268062e10 con 0x7f126813bb90 2024-04-26T04:22:32.878 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.864+0000 7f125effd640 1 -- 172.21.15.63:0/3073060184 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(206..206 src has 1..206) ==== 4738+0+0 (secure 0 0 0) 0x7f12640e3530 con 0x7f12680a2cd0 2024-04-26T04:22:32.878 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.864+0000 7f126ea29640 1 --2- 172.21.15.63:0/3073060184 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f124c035f10 0x7f124c0383d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:32.878 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.864+0000 7f126ea29640 10 monclient: get_auth_request con 0x7f124c035f10 auth_method 0 2024-04-26T04:22:32.878 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.864+0000 7f126da27640 1 -- 172.21.15.63:0/3073060184 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f126813bb90 msgr2=0x7f126813bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:22:32.878 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.864+0000 7f126da27640 1 --2- 172.21.15.63:0/3073060184 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f126813bb90 0x7f126813bf70 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f126813dea0 tx=0x7f125800e960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:22:32.878 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.864+0000 7f125effd640 1 -- 172.21.15.63:0/3073060184 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f125801a070 con 0x7f126813bb90 2024-04-26T04:22:32.879 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.864+0000 7f125effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:32.879 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.864+0000 7f125effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:22:32.879 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.864+0000 7f125effd640 10 monclient: dump: 2024-04-26T04:22:32.879 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:32.879 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:32.879 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:32.879 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:32.879 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:32.879 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:32.879 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:32.879 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:32.879 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:32.879 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:32.879 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.868+0000 7f125effd640 1 -- 172.21.15.63:0/3073060184 <== mon.0 v2:172.21.15.63:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1258007450 con 0x7f126813bb90 2024-04-26T04:22:32.880 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.868+0000 7f125effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:22:32.880 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.868+0000 7f125effd640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:22:32.880 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.868+0000 7f125effd640 1 -- 172.21.15.63:0/3073060184 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f126813bb90 msgr2=0x7f126813bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:32.880 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.868+0000 7f125effd640 1 --2- 172.21.15.63:0/3073060184 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f126813bb90 0x7f126813bf70 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f126813dea0 tx=0x7f125800e960 comp rx=0 tx=0).stop 2024-04-26T04:22:32.880 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.868+0000 7f125effd640 1 -- 172.21.15.63:0/3073060184 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f126813bb90 msgr2=0x7f126813bf70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:22:32.880 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.868+0000 7f125effd640 1 --2- 172.21.15.63:0/3073060184 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f126813bb90 0x7f126813bf70 secure :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f126813dea0 tx=0x7f125800e960 comp rx=0 tx=0).stop 2024-04-26T04:22:32.880 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.868+0000 7f126ea29640 1 --2- 172.21.15.63:0/3073060184 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f124c035f10 0x7f124c0383d0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f1260009400 tx=0x7f1260007040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:22:32.881 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.872+0000 7f127048c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T04:22:32.881 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.872+0000 7f127048c640 1 --2- 172.21.15.63:0/3073060184 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f126813c4b0 0x7f1268137dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:32.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.872+0000 7f127048c640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:32.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.872+0000 7f127048c640 1 -- 172.21.15.63:0/3073060184 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f126813edc0 con 0x7f126813c4b0 2024-04-26T04:22:32.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.872+0000 7f126da27640 1 --2- 172.21.15.63:0/3073060184 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f126813c4b0 0x7f1268137dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:32.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.872+0000 7f126da27640 10 monclient: get_auth_request con 0x7f126813c4b0 auth_method 0 2024-04-26T04:22:32.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.872+0000 7f126da27640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:32.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.872+0000 7f126da27640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:32.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.872+0000 7f126da27640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:32.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.872+0000 7f126da27640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:22:32.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.872+0000 7f126da27640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:22:32.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.872+0000 7f126da27640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:22:32.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.872+0000 7f126da27640 10 monclient(hunting): handle_auth_done global_id 5181 payload 824 2024-04-26T04:22:32.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.872+0000 7f126da27640 1 -- 172.21.15.63:0/3073060184 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f126811fa90 con 0x7f126813c4b0 2024-04-26T04:22:32.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.876+0000 7f126da27640 1 -- 172.21.15.63:0/3073060184 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f126813c4b0 msgr2=0x7f1268137dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:22:32.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.876+0000 7f126da27640 1 --2- 172.21.15.63:0/3073060184 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f126813c4b0 0x7f1268137dd0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f125800efd0 tx=0x7f1258003770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:22:32.883 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.876+0000 7f125effd640 1 -- 172.21.15.63:0/3073060184 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f125801a050 con 0x7f126813c4b0 2024-04-26T04:22:32.883 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.876+0000 7f125effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:32.883 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.876+0000 7f125effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:22:32.883 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.876+0000 7f125effd640 10 monclient: dump: 2024-04-26T04:22:32.883 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:32.883 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:32.883 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:32.883 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:32.883 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:32.883 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:32.883 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:32.883 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:32.883 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:32.883 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:32.884 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.876+0000 7f125effd640 1 -- 172.21.15.63:0/3073060184 <== mon.0 v2:172.21.15.63:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4723 (secure 0 0 0) 0x7f12580069e0 con 0x7f126813c4b0 2024-04-26T04:22:32.884 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.876+0000 7f125effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T04:22:32.884 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.876+0000 7f125effd640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:22:32.884 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.876+0000 7f125effd640 1 -- 172.21.15.63:0/3073060184 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f126813c4b0 msgr2=0x7f1268137dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:32.884 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.876+0000 7f125effd640 1 --2- 172.21.15.63:0/3073060184 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f126813c4b0 0x7f1268137dd0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f125800efd0 tx=0x7f1258003770 comp rx=0 tx=0).stop 2024-04-26T04:22:32.884 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.876+0000 7f125effd640 1 -- 172.21.15.63:0/3073060184 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f126813c4b0 msgr2=0x7f1268137dd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:22:32.884 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.876+0000 7f125effd640 1 --2- 172.21.15.63:0/3073060184 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f126813c4b0 0x7f1268137dd0 secure :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f125800efd0 tx=0x7f1258003770 comp rx=0 tx=0).stop 2024-04-26T04:22:32.884 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-26T04:22:32.884 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a", 2024-04-26T04:22:32.884 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 0, 2024-04-26T04:22:32.885 INFO:teuthology.orchestra.run.smithi063.stdout: "state": "leader", 2024-04-26T04:22:32.885 INFO:teuthology.orchestra.run.smithi063.stdout: "election_epoch": 20, 2024-04-26T04:22:32.885 INFO:teuthology.orchestra.run.smithi063.stdout: "uptime": 244029, 2024-04-26T04:22:32.885 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum": [ 2024-04-26T04:22:32.885 INFO:teuthology.orchestra.run.smithi063.stdout: 0, 2024-04-26T04:22:32.885 INFO:teuthology.orchestra.run.smithi063.stdout: 1, 2024-04-26T04:22:32.885 INFO:teuthology.orchestra.run.smithi063.stdout: 2 2024-04-26T04:22:32.885 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:22:32.885 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_age": 0, 2024-04-26T04:22:32.885 INFO:teuthology.orchestra.run.smithi063.stdout: "features": { 2024-04-26T04:22:32.885 INFO:teuthology.orchestra.run.smithi063.stdout: "required_con": "2450521708007866372", 2024-04-26T04:22:32.885 INFO:teuthology.orchestra.run.smithi063.stdout: "required_mon": [ 2024-04-26T04:22:32.885 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:22:32.885 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:22:32.885 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:22:32.886 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:22:32.886 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:22:32.886 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:22:32.886 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:22:32.886 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:22:32.886 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:22:32.886 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:22:32.886 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:22:32.886 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:22:32.886 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_con": "4540701547738038271", 2024-04-26T04:22:32.886 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_mon": [ 2024-04-26T04:22:32.886 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:22:32.886 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:22:32.886 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:22:32.887 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:22:32.887 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:22:32.887 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:22:32.887 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:22:32.887 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:22:32.887 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:22:32.887 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:22:32.887 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:22:32.887 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:22:32.887 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:32.887 INFO:teuthology.orchestra.run.smithi063.stdout: "outside_quorum": [], 2024-04-26T04:22:32.887 INFO:teuthology.orchestra.run.smithi063.stdout: "extra_probe_peers": [], 2024-04-26T04:22:32.887 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_provider": [], 2024-04-26T04:22:32.888 INFO:teuthology.orchestra.run.smithi063.stdout: "monmap": { 2024-04-26T04:22:32.888 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 1, 2024-04-26T04:22:32.888 INFO:teuthology.orchestra.run.smithi063.stdout: "fsid": "cd1c6a0c-4e4b-4014-89c0-2e535479ef39", 2024-04-26T04:22:32.888 INFO:teuthology.orchestra.run.smithi063.stdout: "modified": "2024-04-26T04:18:03.702339Z", 2024-04-26T04:22:32.888 INFO:teuthology.orchestra.run.smithi063.stdout: "created": "2024-04-26T04:18:03.702339Z", 2024-04-26T04:22:32.888 INFO:teuthology.orchestra.run.smithi063.stdout: "min_mon_release": 19, 2024-04-26T04:22:32.888 INFO:teuthology.orchestra.run.smithi063.stdout: "min_mon_release_name": "squid", 2024-04-26T04:22:32.888 INFO:teuthology.orchestra.run.smithi063.stdout: "election_strategy": 1, 2024-04-26T04:22:32.888 INFO:teuthology.orchestra.run.smithi063.stdout: "disallowed_leaders: ": "", 2024-04-26T04:22:32.888 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": false, 2024-04-26T04:22:32.888 INFO:teuthology.orchestra.run.smithi063.stdout: "tiebreaker_mon": "", 2024-04-26T04:22:32.888 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_ranks: ": "", 2024-04-26T04:22:32.888 INFO:teuthology.orchestra.run.smithi063.stdout: "features": { 2024-04-26T04:22:32.888 INFO:teuthology.orchestra.run.smithi063.stdout: "persistent": [ 2024-04-26T04:22:32.888 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:22:32.889 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:22:32.889 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:22:32.889 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:22:32.889 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:22:32.889 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:22:32.889 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:22:32.889 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:22:32.889 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:22:32.889 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:22:32.889 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:22:32.889 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:22:32.889 INFO:teuthology.orchestra.run.smithi063.stdout: "optional": [] 2024-04-26T04:22:32.889 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:32.889 INFO:teuthology.orchestra.run.smithi063.stdout: "mons": [ 2024-04-26T04:22:32.890 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:32.890 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 0, 2024-04-26T04:22:32.890 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a", 2024-04-26T04:22:32.890 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:22:32.890 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:22:32.890 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:32.890 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:22:32.890 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:3300", 2024-04-26T04:22:32.890 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:22:32.890 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:32.890 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:32.890 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:22:32.890 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6789", 2024-04-26T04:22:32.890 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:22:32.891 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:22:32.891 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:22:32.891 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:32.891 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6789/0", 2024-04-26T04:22:32.891 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6789/0", 2024-04-26T04:22:32.891 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:22:32.891 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:22:32.891 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:22:32.891 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:32.891 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:32.891 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 1, 2024-04-26T04:22:32.891 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b", 2024-04-26T04:22:32.891 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:22:32.891 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:22:32.892 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:32.892 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:22:32.892 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:3300", 2024-04-26T04:22:32.892 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:22:32.892 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:32.892 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:32.892 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:22:32.892 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:6789", 2024-04-26T04:22:32.892 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:22:32.892 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:22:32.892 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:22:32.892 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:32.892 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:6789/0", 2024-04-26T04:22:32.892 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.186:6789/0", 2024-04-26T04:22:32.892 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:22:32.893 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:22:32.893 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:22:32.893 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:32.893 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:32.893 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 2, 2024-04-26T04:22:32.893 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "c", 2024-04-26T04:22:32.893 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:22:32.893 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:22:32.893 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:32.893 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:22:32.893 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:3301", 2024-04-26T04:22:32.893 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:22:32.893 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:32.893 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:32.894 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:22:32.894 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6790", 2024-04-26T04:22:32.894 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:22:32.894 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:22:32.894 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:22:32.894 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:32.894 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6790/0", 2024-04-26T04:22:32.894 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6790/0", 2024-04-26T04:22:32.894 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:22:32.894 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:22:32.894 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:22:32.894 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:22:32.894 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:22:32.894 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:32.895 INFO:teuthology.orchestra.run.smithi063.stdout: "feature_map": { 2024-04-26T04:22:32.895 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": [ 2024-04-26T04:22:32.895 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:32.895 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:22:32.895 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:22:32.895 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 1 2024-04-26T04:22:32.895 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:22:32.895 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:22:32.895 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": [ 2024-04-26T04:22:32.895 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:32.895 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:22:32.895 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:22:32.895 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 3 2024-04-26T04:22:32.895 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:22:32.896 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:22:32.896 INFO:teuthology.orchestra.run.smithi063.stdout: "client": [ 2024-04-26T04:22:32.896 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:32.896 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:22:32.896 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:22:32.896 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 2 2024-04-26T04:22:32.896 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:22:32.896 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:22:32.896 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:32.896 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": false 2024-04-26T04:22:32.896 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-26T04:22:32.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.876+0000 7f127048c640 1 -- 172.21.15.63:0/3073060184 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f124c035f10 msgr2=0x7f124c0383d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:32.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.876+0000 7f127048c640 1 --2- 172.21.15.63:0/3073060184 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f124c035f10 0x7f124c0383d0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f1260009400 tx=0x7f1260007040 comp rx=0 tx=0).stop 2024-04-26T04:22:32.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.876+0000 7f127048c640 10 monclient: shutdown 2024-04-26T04:22:32.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.876+0000 7f127048c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:22:32.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.876+0000 7f127048c640 1 -- 172.21.15.63:0/3073060184 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f12680a2cd0 msgr2=0x7f12681cefb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:32.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.876+0000 7f127048c640 1 --2- 172.21.15.63:0/3073060184 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f12680a2cd0 0x7f12681cefb0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f126401aa60 tx=0x7f126401abf0 comp rx=0 tx=0).stop 2024-04-26T04:22:32.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.876+0000 7f126ea29640 1 -- 172.21.15.63:0/3073060184 reap_dead start 2024-04-26T04:22:32.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.876+0000 7f126ea29640 1 -- 172.21.15.63:0/3073060184 reap_dead start 2024-04-26T04:22:32.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.876+0000 7f127048c640 1 -- 172.21.15.63:0/3073060184 shutdown_connections 2024-04-26T04:22:32.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.876+0000 7f127048c640 1 -- 172.21.15.63:0/3073060184 >> 172.21.15.63:0/3073060184 conn(0x7f12680a5f40 msgr2=0x7f12680a1a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:22:32.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.876+0000 7f127048c640 1 -- 172.21.15.63:0/3073060184 shutdown_connections 2024-04-26T04:22:32.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:32.876+0000 7f127048c640 1 -- 172.21.15.63:0/3073060184 wait complete. 2024-04-26T04:22:32.898 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T04:22:32.899 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-26T04:22:32.899 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T04:22:33.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.016+0000 7f119a6e9640 10 monclient: get_monmap_and_config 2024-04-26T04:22:33.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.016+0000 7f119a6e9640 10 monclient: build_initial_monmap 2024-04-26T04:22:33.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.016+0000 7f119a6e9640 10 monclient: monmap: 2024-04-26T04:22:33.019 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:22:33.019 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:33.019 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:22:33.019504+0000 2024-04-26T04:22:33.019 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:22:33.019504+0000 2024-04-26T04:22:33.020 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:22:33.020 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:33.020 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:22:33.020 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:22:33.020 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:22:33.020 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:22:33.020 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:22:33.020 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:33.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.016+0000 7f119a6e9640 1 Processor -- start 2024-04-26T04:22:33.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.016+0000 7f119a6e9640 1 -- start start 2024-04-26T04:22:33.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.016+0000 7f119a6e9640 10 monclient: init 2024-04-26T04:22:33.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.016+0000 7f119a6e9640 10 monclient: _reopen_session rank -1 2024-04-26T04:22:33.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.016+0000 7f119a6e9640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2024-04-26T04:22:33.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.016+0000 7f119a6e9640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f119409f140 0x7f119409aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:33.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.016+0000 7f119a6e9640 10 monclient(hunting): picked mon.noname-b con 0x7f119409f140 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:22:33.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.016+0000 7f119a6e9640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f119409c730 0x7f119409af80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:33.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.016+0000 7f119a6e9640 10 monclient(hunting): picked mon.noname-c con 0x7f119409c730 addr v2:172.21.15.186:3300/0 2024-04-26T04:22:33.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.016+0000 7f119a6e9640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f119409b4c0 0x7f119413a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:33.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.016+0000 7f119a6e9640 10 monclient(hunting): picked mon.noname-a con 0x7f119409b4c0 addr v2:172.21.15.63:3300/0 2024-04-26T04:22:33.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.016+0000 7f119a6e9640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:33.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.016+0000 7f119a6e9640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7f119409da90 con 0x7f119409b4c0 2024-04-26T04:22:33.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.016+0000 7f119a6e9640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:33.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.016+0000 7f119a6e9640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f1194072e80 con 0x7f119409c730 2024-04-26T04:22:33.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.016+0000 7f119a6e9640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:33.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.016+0000 7f119a6e9640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f11940a2710 con 0x7f119409f140 2024-04-26T04:22:33.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.016+0000 7f119a6e9640 10 monclient(hunting): _renew_subs 2024-04-26T04:22:33.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.016+0000 7f119a6e9640 10 monclient(hunting): authenticate will time out at 1184.726844s 2024-04-26T04:22:33.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.016+0000 7f1198c86640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f119409b4c0 0x7f119413a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:33.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.016+0000 7f1193fff640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f119409f140 0x7f119409aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:33.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.016+0000 7f1198c86640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f119409b4c0 0x7f119413a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:48338/0 (socket says 172.21.15.63:48338) 2024-04-26T04:22:33.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.016+0000 7f1193fff640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f119409f140 0x7f119409aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:49956/0 (socket says 172.21.15.63:49956) 2024-04-26T04:22:33.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.016+0000 7f1198c86640 1 -- 172.21.15.63:0/1742734867 learned_addr learned my addr 172.21.15.63:0/1742734867 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:22:33.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.016+0000 7f1198c86640 10 monclient(hunting): get_auth_request con 0x7f119409b4c0 auth_method 0 2024-04-26T04:22:33.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.016+0000 7f1198c86640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:33.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.016+0000 7f1198c86640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:33.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.016+0000 7f1198c86640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:33.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.016+0000 7f1193fff640 10 monclient(hunting): get_auth_request con 0x7f119409f140 auth_method 0 2024-04-26T04:22:33.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.016+0000 7f1193fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:33.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.016+0000 7f1193fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:33.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.016+0000 7f1193fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:33.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.016+0000 7f1198c86640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:22:33.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.016+0000 7f1198c86640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:22:33.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.016+0000 7f1198c86640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:22:33.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.016+0000 7f1193fff640 1 -- 172.21.15.63:0/1742734867 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f119409f140 msgr2=0x7f119409aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T04:22:33.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.016+0000 7f118bfff640 1 --2- 172.21.15.63:0/1742734867 >> v2:172.21.15.186:3300/0 conn(0x7f119409c730 0x7f119409af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:33.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.016+0000 7f1193fff640 1 -- 172.21.15.63:0/1742734867 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f119409f140 msgr2=0x7f119409aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:22:33.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.016+0000 7f1193fff640 1 --2- 172.21.15.63:0/1742734867 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f119409f140 0x7f119409aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:22:33.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.016+0000 7f118bfff640 10 monclient(hunting): get_auth_request con 0x7f119409c730 auth_method 0 2024-04-26T04:22:33.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.016+0000 7f118bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:33.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.016+0000 7f118bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:33.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.016+0000 7f118bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:33.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.016+0000 7f1193fff640 1 --2- 172.21.15.63:0/1742734867 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f119409f140 0x7f119409aa40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:22:33.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.016+0000 7f1198c86640 10 monclient(hunting): handle_auth_done global_id 5184 payload 274 2024-04-26T04:22:33.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.016+0000 7f1198c86640 1 -- 172.21.15.63:0/1742734867 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f119409f140 msgr2=0x7f119409aa40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T04:22:33.027 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.016+0000 7f1198c86640 1 --2- 172.21.15.63:0/1742734867 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f119409f140 0x7f119409aa40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:33.027 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.016+0000 7f1198c86640 1 -- 172.21.15.63:0/1742734867 >> v2:172.21.15.186:3300/0 conn(0x7f119409c730 msgr2=0x7f119409af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:22:33.027 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.016+0000 7f1198c86640 1 --2- 172.21.15.63:0/1742734867 >> v2:172.21.15.186:3300/0 conn(0x7f119409c730 0x7f119409af80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:33.027 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.016+0000 7f1198c86640 10 monclient: _finish_hunting 0 2024-04-26T04:22:33.027 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.016+0000 7f1198c86640 1 monclient: found mon.noname-a 2024-04-26T04:22:33.027 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.016+0000 7f1198c86640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:22:33.027 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.016+0000 7f1198c86640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:22:33.027 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.016+0000 7f1198c86640 1 -- 172.21.15.63:0/1742734867 --> v2:172.21.15.63:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f119409ce50 con 0x7f119409b4c0 2024-04-26T04:22:33.027 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.016+0000 7f118bfff640 1 --2- 172.21.15.63:0/1742734867 >> v2:172.21.15.186:3300/0 conn(0x7f119409c730 0x7f119409af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T04:22:33.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.020+0000 7f1198c86640 1 -- 172.21.15.63:0/1742734867 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f119409b4c0 msgr2=0x7f119413a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:22:33.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.020+0000 7f1198c86640 1 --2- 172.21.15.63:0/1742734867 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f119409b4c0 0x7f119413a7d0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f11940a3aa0 tx=0x7f118c01c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=735367dd0443db6c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:22:33.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.020+0000 7f11937fe640 1 -- 172.21.15.63:0/1742734867 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f118c0a5540 con 0x7f119409b4c0 2024-04-26T04:22:33.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.020+0000 7f11937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:33.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.020+0000 7f11937fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:22:33.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.020+0000 7f11937fe640 10 monclient: dump: 2024-04-26T04:22:33.033 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:33.033 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:33.033 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:33.034 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:33.034 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:33.034 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:33.034 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:33.034 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:33.034 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:33.034 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:33.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.020+0000 7f11937fe640 10 monclient: _finish_auth 0 2024-04-26T04:22:33.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.020+0000 7f11937fe640 10 monclient: _check_auth_tickets 2024-04-26T04:22:33.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.020+0000 7f11937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:22:33.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.020+0000 7f11937fe640 1 -- 172.21.15.63:0/1742734867 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f118c0a56e0 con 0x7f119409b4c0 2024-04-26T04:22:33.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.020+0000 7f11937fe640 10 monclient: handle_config config(0 keys) 2024-04-26T04:22:33.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.020+0000 7f11937fe640 1 -- 172.21.15.63:0/1742734867 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f118c0a5ab0 con 0x7f119409b4c0 2024-04-26T04:22:33.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.020+0000 7f119a6e9640 5 monclient: authenticate success, global_id 5184 2024-04-26T04:22:33.035 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.020+0000 7f119a6e9640 10 monclient: get_monmap_and_config success 2024-04-26T04:22:33.035 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.020+0000 7f119a6e9640 4 set_mon_vals no callback set 2024-04-26T04:22:33.035 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.020+0000 7f11937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:33.035 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.020+0000 7f11937fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:22:33.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.020+0000 7f11937fe640 10 monclient: dump: 2024-04-26T04:22:33.036 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:33.036 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:33.036 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:33.036 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:33.036 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:33.036 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:33.036 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:33.036 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:33.036 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:33.036 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:33.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.024+0000 7f119a6e9640 10 monclient: shutdown 2024-04-26T04:22:33.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.024+0000 7f119a6e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:22:33.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.024+0000 7f119a6e9640 1 -- 172.21.15.63:0/1742734867 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f119409b4c0 msgr2=0x7f119413a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:33.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.024+0000 7f119a6e9640 1 --2- 172.21.15.63:0/1742734867 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f119409b4c0 0x7f119413a7d0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f11940a3aa0 tx=0x7f118c01c8b0 comp rx=0 tx=0).stop 2024-04-26T04:22:33.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.024+0000 7f119a6e9640 1 -- 172.21.15.63:0/1742734867 shutdown_connections 2024-04-26T04:22:33.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.024+0000 7f119a6e9640 1 --2- 172.21.15.63:0/1742734867 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f119409b4c0 0x7f119413a7d0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:33.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.024+0000 7f119a6e9640 1 --2- 172.21.15.63:0/1742734867 >> v2:172.21.15.186:3300/0 conn(0x7f119409c730 0x7f119409af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:33.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.024+0000 7f119a6e9640 1 --2- 172.21.15.63:0/1742734867 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f119409f140 0x7f119409aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:33.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.024+0000 7f119a6e9640 1 -- 172.21.15.63:0/1742734867 >> 172.21.15.63:0/1742734867 conn(0x7f11940a6cd0 msgr2=0x7f11940a2b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:22:33.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.024+0000 7f119a6e9640 1 -- 172.21.15.63:0/1742734867 shutdown_connections 2024-04-26T04:22:33.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.024+0000 7f119a6e9640 1 -- 172.21.15.63:0/1742734867 wait complete. 2024-04-26T04:22:33.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.024+0000 7f119a6e9640 10 monclient: build_initial_monmap 2024-04-26T04:22:33.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.024+0000 7f119a6e9640 10 monclient: monmap: 2024-04-26T04:22:33.038 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:22:33.038 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:22:33.038 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:22:33.038 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:22:33.039 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:22:33.039 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:33.039 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:22:33.039 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:22:33.039 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:22:33.039 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:33.039 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.024+0000 7f119a6e9640 1 Processor -- start 2024-04-26T04:22:33.039 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.024+0000 7f119a6e9640 1 -- start start 2024-04-26T04:22:33.039 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.024+0000 7f119a6e9640 10 monclient: init 2024-04-26T04:22:33.039 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.024+0000 7f119a6e9640 10 monclient: _reopen_session rank -1 2024-04-26T04:22:33.039 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.024+0000 7f119a6e9640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T04:22:33.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.024+0000 7f119a6e9640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f119409c730 0x7f11941cad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:33.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.024+0000 7f119a6e9640 10 monclient(hunting): picked mon.noname-c con 0x7f119409c730 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:22:33.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.024+0000 7f119a6e9640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f119409f140 0x7f11941cb260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:33.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.024+0000 7f119a6e9640 10 monclient(hunting): picked mon.noname-a con 0x7f119409f140 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:22:33.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.024+0000 7f119a6e9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1194121210 0x7f11941216c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:33.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.024+0000 7f119a6e9640 10 monclient(hunting): picked mon.noname-b con 0x7f1194121210 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:22:33.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.024+0000 7f119a6e9640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:33.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.024+0000 7f119a6e9640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f11940a1dc0 con 0x7f119409f140 2024-04-26T04:22:33.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.024+0000 7f119a6e9640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:33.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.024+0000 7f119a6e9640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f11940a1c40 con 0x7f1194121210 2024-04-26T04:22:33.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.024+0000 7f119a6e9640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:33.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.024+0000 7f119a6e9640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f119406d930 con 0x7f119409c730 2024-04-26T04:22:33.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.024+0000 7f119a6e9640 10 monclient(hunting): _renew_subs 2024-04-26T04:22:33.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.024+0000 7f119a6e9640 10 monclient(hunting): authenticate will time out at 1184.733940s 2024-04-26T04:22:33.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.024+0000 7f118bfff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f119409f140 0x7f11941cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:33.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.024+0000 7f1193fff640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f119409c730 0x7f11941cad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:33.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.024+0000 7f118bfff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f119409f140 0x7f11941cb260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:48354/0 (socket says 172.21.15.63:48354) 2024-04-26T04:22:33.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.024+0000 7f118bfff640 1 -- 172.21.15.63:0/4084571384 learned_addr learned my addr 172.21.15.63:0/4084571384 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:22:33.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.024+0000 7f118bfff640 10 monclient(hunting): get_auth_request con 0x7f119409f140 auth_method 0 2024-04-26T04:22:33.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.024+0000 7f118bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:33.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.024+0000 7f118bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:33.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.024+0000 7f118bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:33.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.024+0000 7f1193fff640 10 monclient(hunting): get_auth_request con 0x7f119409c730 auth_method 0 2024-04-26T04:22:33.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.024+0000 7f1193fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:33.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.024+0000 7f1193fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:33.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.024+0000 7f1193fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:33.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.024+0000 7f118bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:22:33.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.024+0000 7f118bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:22:33.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.024+0000 7f118bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:22:33.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.024+0000 7f1193fff640 1 -- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f119409c730 msgr2=0x7f11941cad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T04:22:33.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.024+0000 7f1193fff640 1 -- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f119409c730 msgr2=0x7f11941cad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T04:22:33.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.024+0000 7f1193fff640 1 --2- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f119409c730 0x7f11941cad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:22:33.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.024+0000 7f1193fff640 1 --2- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f119409c730 0x7f11941cad20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:22:33.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.024+0000 7f1198c86640 1 --2- 172.21.15.63:0/4084571384 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1194121210 0x7f11941216c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:33.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.024+0000 7f1198c86640 10 monclient(hunting): get_auth_request con 0x7f1194121210 auth_method 0 2024-04-26T04:22:33.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.024+0000 7f1198c86640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:33.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.024+0000 7f1198c86640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:33.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.024+0000 7f1198c86640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:33.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.024+0000 7f118bfff640 10 monclient(hunting): handle_auth_done global_id 5187 payload 824 2024-04-26T04:22:33.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.024+0000 7f118bfff640 1 -- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f119409c730 msgr2=0x7f11941cad20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T04:22:33.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.024+0000 7f118bfff640 1 --2- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f119409c730 0x7f11941cad20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:33.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.024+0000 7f118bfff640 1 -- 172.21.15.63:0/4084571384 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1194121210 msgr2=0x7f11941216c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:33.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.024+0000 7f118bfff640 1 --2- 172.21.15.63:0/4084571384 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1194121210 0x7f11941216c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:33.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.024+0000 7f118bfff640 10 monclient: _finish_hunting 0 2024-04-26T04:22:33.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.024+0000 7f118bfff640 1 monclient: found mon.noname-a 2024-04-26T04:22:33.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.024+0000 7f118bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:22:33.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.024+0000 7f118bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:22:33.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.024+0000 7f118bfff640 1 -- 172.21.15.63:0/4084571384 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1194124b50 con 0x7f119409f140 2024-04-26T04:22:33.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.028+0000 7f118bfff640 1 -- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f119409f140 msgr2=0x7f11941cb260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:22:33.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.028+0000 7f118bfff640 1 --2- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f119409f140 0x7f11941cb260 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f117c00e840 tx=0x7f117c00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:22:33.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.028+0000 7f11917fa640 1 -- 172.21.15.63:0/4084571384 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f117c017070 con 0x7f119409f140 2024-04-26T04:22:33.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.028+0000 7f11917fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:33.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.028+0000 7f11917fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:22:33.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.028+0000 7f11917fa640 10 monclient: dump: 2024-04-26T04:22:33.044 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:33.044 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:33.044 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:33.044 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:33.044 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:33.044 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:33.044 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:33.044 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:33.044 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:33.044 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:33.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.028+0000 7f11917fa640 10 monclient: _finish_auth 0 2024-04-26T04:22:33.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.028+0000 7f11917fa640 10 monclient: _check_auth_tickets 2024-04-26T04:22:33.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.028+0000 7f11917fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:22:33.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.028+0000 7f11917fa640 1 -- 172.21.15.63:0/4084571384 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f117c00cae0 con 0x7f119409f140 2024-04-26T04:22:33.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.028+0000 7f119a6e9640 5 monclient: authenticate success, global_id 5187 2024-04-26T04:22:33.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.028+0000 7f119a6e9640 10 monclient: _renew_subs 2024-04-26T04:22:33.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.028+0000 7f119a6e9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:22:33.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.028+0000 7f119a6e9640 1 -- 172.21.15.63:0/4084571384 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1194124e30 con 0x7f119409f140 2024-04-26T04:22:33.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.028+0000 7f11917fa640 10 monclient: handle_config config(0 keys) 2024-04-26T04:22:33.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.028+0000 7f11917fa640 1 -- 172.21.15.63:0/4084571384 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f117c00f040 con 0x7f119409f140 2024-04-26T04:22:33.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.028+0000 7f11917fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:33.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.028+0000 7f11917fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:22:33.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.028+0000 7f11917fa640 10 monclient: dump: 2024-04-26T04:22:33.045 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:33.045 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:33.045 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:33.045 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:33.045 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:33.045 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:33.045 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:33.046 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:33.046 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:33.046 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:33.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.028+0000 7f1192ffd640 4 set_mon_vals no callback set 2024-04-26T04:22:33.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.028+0000 7f119a6e9640 10 monclient: _renew_subs 2024-04-26T04:22:33.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.028+0000 7f119a6e9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:22:33.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.028+0000 7f119a6e9640 1 -- 172.21.15.63:0/4084571384 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11940721b0 con 0x7f119409f140 2024-04-26T04:22:33.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.028+0000 7f119a6e9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:22:33.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.028+0000 7f119a6e9640 1 --2- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f11940a6cd0 0x7f1194121c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:33.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.028+0000 7f119a6e9640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:33.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.028+0000 7f119a6e9640 1 -- 172.21.15.63:0/4084571384 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f1194123760 con 0x7f11940a6cd0 2024-04-26T04:22:33.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.028+0000 7f11917fa640 1 -- 172.21.15.63:0/4084571384 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f117c01a070 con 0x7f119409f140 2024-04-26T04:22:33.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.028+0000 7f1193fff640 1 --2- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f11940a6cd0 0x7f1194121c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:33.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.028+0000 7f1193fff640 10 monclient: get_auth_request con 0x7f11940a6cd0 auth_method 0 2024-04-26T04:22:33.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.028+0000 7f1193fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:33.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.028+0000 7f1193fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:33.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.028+0000 7f1193fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:33.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.028+0000 7f1193fff640 1 -- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f11940a6cd0 msgr2=0x7f1194121c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T04:22:33.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.028+0000 7f1193fff640 1 -- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f11940a6cd0 msgr2=0x7f1194121c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T04:22:33.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.028+0000 7f1193fff640 1 --2- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f11940a6cd0 0x7f1194121c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:22:33.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.028+0000 7f1193fff640 1 --2- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f11940a6cd0 0x7f1194121c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:22:33.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.028+0000 7f11917fa640 1 --2- 172.21.15.63:0/4084571384 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f1174035ec0 0x7f1174038380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:33.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.028+0000 7f11917fa640 1 -- 172.21.15.63:0/4084571384 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(207..207 src has 1..207) ==== 4738+0+0 (secure 0 0 0) 0x7f117c048ac0 con 0x7f119409f140 2024-04-26T04:22:33.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.028+0000 7f1198c86640 1 --2- 172.21.15.63:0/4084571384 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f1174035ec0 0x7f1174038380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:33.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.028+0000 7f1198c86640 10 monclient: get_auth_request con 0x7f1174035ec0 auth_method 0 2024-04-26T04:22:33.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.032+0000 7f1198c86640 1 --2- 172.21.15.63:0/4084571384 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f1174035ec0 0x7f1174038380 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f119409d610 tx=0x7f118c0a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:22:33.171 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[30414]: starting. 2024-04-26T04:22:33.171 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[30414]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-30413 2024-04-26T04:22:33.171 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[30414]: created object 0... 2024-04-26T04:22:33.171 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[30414]: created object 25... 2024-04-26T04:22:33.171 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[30414]: created object 49... 2024-04-26T04:22:33.171 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[30414]: finishing. 2024-04-26T04:22:33.171 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[30414]: shutting down. 2024-04-26T04:22:33.234 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.232+0000 7f1193fff640 1 --2- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f11940a6cd0 0x7f1194121c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:33.234 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.232+0000 7f1193fff640 10 monclient: get_auth_request con 0x7f11940a6cd0 auth_method 0 2024-04-26T04:22:33.234 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.232+0000 7f1193fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:33.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.232+0000 7f1193fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:33.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.232+0000 7f1193fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:22:33.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.232+0000 7f1193fff640 1 -- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f11940a6cd0 msgr2=0x7f1194121c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-26T04:22:33.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.232+0000 7f1193fff640 1 -- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f11940a6cd0 msgr2=0x7f1194121c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T04:22:33.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.232+0000 7f1193fff640 1 --2- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f11940a6cd0 0x7f1194121c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:22:33.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.232+0000 7f1193fff640 1 --2- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f11940a6cd0 0x7f1194121c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T04:22:33.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.636+0000 7f1193fff640 1 --2- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f11940a6cd0 0x7f1194121c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:33.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.636+0000 7f1193fff640 10 monclient: get_auth_request con 0x7f11940a6cd0 auth_method 0 2024-04-26T04:22:33.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.636+0000 7f1193fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:33.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.636+0000 7f1193fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:33.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.636+0000 7f1193fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:22:33.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.636+0000 7f1193fff640 1 -- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f11940a6cd0 msgr2=0x7f1194121c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-26T04:22:33.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.636+0000 7f1193fff640 1 -- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f11940a6cd0 msgr2=0x7f1194121c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T04:22:33.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.636+0000 7f1193fff640 1 --2- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f11940a6cd0 0x7f1194121c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:22:33.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:33.636+0000 7f1193fff640 1 --2- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f11940a6cd0 0x7f1194121c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T04:22:34.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:34.440+0000 7f1193fff640 1 --2- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f11940a6cd0 0x7f1194121c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:34.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:34.440+0000 7f1193fff640 10 monclient: get_auth_request con 0x7f11940a6cd0 auth_method 0 2024-04-26T04:22:34.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:34.440+0000 7f1193fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:34.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:34.440+0000 7f1193fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:34.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:34.440+0000 7f1193fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:22:34.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:34.440+0000 7f1193fff640 1 -- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f11940a6cd0 msgr2=0x7f1194121c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-26T04:22:34.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:34.440+0000 7f1193fff640 1 -- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f11940a6cd0 msgr2=0x7f1194121c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T04:22:34.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:34.440+0000 7f1193fff640 1 --2- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f11940a6cd0 0x7f1194121c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:22:34.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:34.440+0000 7f1193fff640 1 --2- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f11940a6cd0 0x7f1194121c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T04:22:35.010 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[30415]: starting. 2024-04-26T04:22:35.010 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[30415]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-30413 2024-04-26T04:22:35.010 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[30415]: shutting down. 2024-04-26T04:22:36.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:36.024+0000 7f118b7fe640 10 monclient: tick 2024-04-26T04:22:36.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:36.024+0000 7f118b7fe640 10 monclient: _check_auth_tickets 2024-04-26T04:22:36.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:36.024+0000 7f118b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:22:36.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:36.024+0000 7f118b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:22:36.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:36.044+0000 7f1193fff640 1 --2- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f11940a6cd0 0x7f1194121c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:36.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:36.044+0000 7f1193fff640 10 monclient: get_auth_request con 0x7f11940a6cd0 auth_method 0 2024-04-26T04:22:36.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:36.044+0000 7f1193fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:36.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:36.044+0000 7f1193fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:36.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:36.044+0000 7f1193fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:22:36.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:36.044+0000 7f1193fff640 1 -- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f11940a6cd0 msgr2=0x7f1194121c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-26T04:22:36.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:36.044+0000 7f1193fff640 1 -- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f11940a6cd0 msgr2=0x7f1194121c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T04:22:36.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:36.044+0000 7f1193fff640 1 --2- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f11940a6cd0 0x7f1194121c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:22:36.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:36.044+0000 7f1193fff640 1 --2- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f11940a6cd0 0x7f1194121c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-26T04:22:37.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:37.024+0000 7f118b7fe640 10 monclient: tick 2024-04-26T04:22:37.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:37.024+0000 7f118b7fe640 10 monclient: _check_auth_tickets 2024-04-26T04:22:37.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:37.024+0000 7f118b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:22:37.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:37.024+0000 7f118b7fe640 5 monclient: _check_tell_commands timeout tell command 1 2024-04-26T04:22:37.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:37.024+0000 7f118b7fe640 1 -- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f11940a6cd0 msgr2=0x7f1194121c00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T04:22:37.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:37.024+0000 7f118b7fe640 1 --2- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f11940a6cd0 0x7f1194121c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:37.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:37.024+0000 7f118b7fe640 1 --2- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f116c0035a0 0x7f116c005a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:37.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:37.024+0000 7f118b7fe640 1 -- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f11940a6cd0 msgr2=0x7f1194121c00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:22:37.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:37.024+0000 7f118b7fe640 1 --2- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f11940a6cd0 0x7f1194121c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:37.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:37.024+0000 7f118b7fe640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:37.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:37.024+0000 7f118b7fe640 1 -- 172.21.15.63:0/4084571384 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f1194123760 con 0x7f116c0035a0 2024-04-26T04:22:37.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:37.024+0000 7f1193fff640 1 --2- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f116c0035a0 0x7f116c005a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:37.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:37.024+0000 7f1193fff640 10 monclient: get_auth_request con 0x7f116c0035a0 auth_method 0 2024-04-26T04:22:37.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:37.024+0000 7f1193fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:37.031 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:37.024+0000 7f1193fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:37.031 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:37.024+0000 7f1193fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:37.031 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:37.024+0000 7f1193fff640 1 -- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f116c0035a0 msgr2=0x7f116c005a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-26T04:22:37.031 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:37.024+0000 7f1193fff640 1 -- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f116c0035a0 msgr2=0x7f116c005a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T04:22:37.031 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:37.024+0000 7f1193fff640 1 --2- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f116c0035a0 0x7f116c005a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:22:37.031 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:37.028+0000 7f1193fff640 1 --2- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f116c0035a0 0x7f116c005a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:22:37.230 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:22:37.230 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30448]: starting. 2024-04-26T04:22:37.230 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30448]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-30413 2024-04-26T04:22:37.230 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30448]: created object 0... 2024-04-26T04:22:37.230 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30448]: created object 25... 2024-04-26T04:22:37.231 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30448]: created object 49... 2024-04-26T04:22:37.231 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30448]: finishing. 2024-04-26T04:22:37.231 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30448]: shutting down. 2024-04-26T04:22:37.234 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:37.232+0000 7f1193fff640 1 --2- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f116c0035a0 0x7f116c005a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:37.234 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:37.232+0000 7f1193fff640 10 monclient: get_auth_request con 0x7f116c0035a0 auth_method 0 2024-04-26T04:22:37.234 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:37.232+0000 7f1193fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:37.234 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:37.232+0000 7f1193fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:37.234 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:37.232+0000 7f1193fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:22:37.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:37.232+0000 7f1193fff640 1 -- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f116c0035a0 msgr2=0x7f116c005a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-26T04:22:37.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:37.232+0000 7f1193fff640 1 -- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f116c0035a0 msgr2=0x7f116c005a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T04:22:37.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:37.232+0000 7f1193fff640 1 --2- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f116c0035a0 0x7f116c005a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:22:37.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:37.232+0000 7f1193fff640 1 --2- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f116c0035a0 0x7f116c005a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T04:22:37.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:37.636+0000 7f1193fff640 1 --2- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f116c0035a0 0x7f116c005a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:37.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:37.636+0000 7f1193fff640 10 monclient: get_auth_request con 0x7f116c0035a0 auth_method 0 2024-04-26T04:22:37.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:37.636+0000 7f1193fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:37.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:37.636+0000 7f1193fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:37.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:37.636+0000 7f1193fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:22:37.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:37.636+0000 7f1193fff640 1 -- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f116c0035a0 msgr2=0x7f116c005a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-26T04:22:37.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:37.636+0000 7f1193fff640 1 -- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f116c0035a0 msgr2=0x7f116c005a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T04:22:37.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:37.636+0000 7f1193fff640 1 --2- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f116c0035a0 0x7f116c005a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:22:37.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:37.636+0000 7f1193fff640 1 --2- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f116c0035a0 0x7f116c005a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T04:22:37.701 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:22:37.701 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[30450]: starting. 2024-04-26T04:22:37.702 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[30450]: listing objects. 2024-04-26T04:22:37.702 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[30450]: listed object 0... 2024-04-26T04:22:37.702 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[30450]: listed object 25... 2024-04-26T04:22:37.702 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[30450]: saw 50 objects 2024-04-26T04:22:37.702 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[30450]: shutting down. 2024-04-26T04:22:38.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.024+0000 7f118b7fe640 10 monclient: tick 2024-04-26T04:22:38.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.024+0000 7f118b7fe640 10 monclient: _check_auth_tickets 2024-04-26T04:22:38.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.024+0000 7f118b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:22:38.126 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:22:38.126 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[30449]: starting. 2024-04-26T04:22:38.127 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[30449]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-30413 2024-04-26T04:22:38.127 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[30449]: shutting down. 2024-04-26T04:22:38.131 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:22:38.131 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:22:38.131 INFO:teuthology.orchestra.run.smithi186.stdout:******* SUCCESS ********** 2024-04-26T04:22:38.135 DEBUG:teuthology.orchestra.run.smithi186:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-26T04:22:38.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.440+0000 7f1193fff640 1 --2- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f116c0035a0 0x7f116c005a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:38.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.440+0000 7f1193fff640 10 monclient: get_auth_request con 0x7f116c0035a0 auth_method 0 2024-04-26T04:22:38.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.440+0000 7f1193fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:38.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.440+0000 7f1193fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:38.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.440+0000 7f1193fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:22:38.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.440+0000 7f1193fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:22:38.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.440+0000 7f1193fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:22:38.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.440+0000 7f1193fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:22:38.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.440+0000 7f1193fff640 10 monclient(hunting): handle_auth_done global_id 24113 payload 824 2024-04-26T04:22:38.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.440+0000 7f1193fff640 1 -- 172.21.15.63:0/4084571384 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f116c006110 con 0x7f116c0035a0 2024-04-26T04:22:38.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.440+0000 7f1193fff640 1 -- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f116c0035a0 msgr2=0x7f116c005a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:22:38.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.440+0000 7f1193fff640 1 --2- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f116c0035a0 0x7f116c005a30 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1184011990 tx=0x7f1184011e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:22:38.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.444+0000 7f11917fa640 1 -- 172.21.15.63:0/4084571384 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1184021070 con 0x7f116c0035a0 2024-04-26T04:22:38.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.444+0000 7f11917fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:38.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.444+0000 7f11917fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:22:38.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.444+0000 7f11917fa640 10 monclient: dump: 2024-04-26T04:22:38.446 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:38.446 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:38.446 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:38.446 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:38.446 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:38.446 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:38.446 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:38.446 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:38.447 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:38.447 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:38.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.444+0000 7f11917fa640 1 -- 172.21.15.63:0/4084571384 <== mon.2 v2:172.21.15.63:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1184004220 con 0x7f116c0035a0 2024-04-26T04:22:38.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.444+0000 7f11917fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:22:38.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.444+0000 7f11917fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:22:38.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.444+0000 7f11917fa640 1 -- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f116c0035a0 msgr2=0x7f116c005a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:38.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.444+0000 7f11917fa640 1 --2- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f116c0035a0 0x7f116c005a30 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1184011990 tx=0x7f1184011e50 comp rx=0 tx=0).stop 2024-04-26T04:22:38.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.444+0000 7f11917fa640 1 -- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f116c0035a0 msgr2=0x7f116c005a30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:22:38.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.444+0000 7f11917fa640 1 --2- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f116c0035a0 0x7f116c005a30 secure :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1184011990 tx=0x7f1184011e50 comp rx=0 tx=0).stop 2024-04-26T04:22:38.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.452+0000 7f119a6e9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T04:22:38.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.452+0000 7f119a6e9640 1 --2- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f1194122140 0x7f119411d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:38.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.452+0000 7f119a6e9640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:38.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.452+0000 7f119a6e9640 1 -- 172.21.15.63:0/4084571384 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f1194125070 con 0x7f1194122140 2024-04-26T04:22:38.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.452+0000 7f1193fff640 1 --2- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f1194122140 0x7f119411d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:38.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.452+0000 7f1193fff640 10 monclient: get_auth_request con 0x7f1194122140 auth_method 0 2024-04-26T04:22:38.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.452+0000 7f1193fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:38.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.452+0000 7f1193fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:38.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.452+0000 7f1193fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:38.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.452+0000 7f1193fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:22:38.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.452+0000 7f1193fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:22:38.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.452+0000 7f1193fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:22:38.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.452+0000 7f1193fff640 10 monclient(hunting): handle_auth_done global_id 24116 payload 824 2024-04-26T04:22:38.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.452+0000 7f1193fff640 1 -- 172.21.15.63:0/4084571384 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1194122550 con 0x7f1194122140 2024-04-26T04:22:38.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.452+0000 7f1193fff640 1 -- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f1194122140 msgr2=0x7f119411d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:22:38.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.456+0000 7f1193fff640 1 --2- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f1194122140 0x7f119411d630 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1184011990 tx=0x7f11840046d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:22:38.458 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.456+0000 7f11917fa640 1 -- 172.21.15.63:0/4084571384 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1184021070 con 0x7f1194122140 2024-04-26T04:22:38.458 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.456+0000 7f11917fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:38.458 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.456+0000 7f11917fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:22:38.458 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.456+0000 7f11917fa640 10 monclient: dump: 2024-04-26T04:22:38.458 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:38.458 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:38.458 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:38.458 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:38.458 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:38.458 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:38.459 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:38.459 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:38.459 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:38.459 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:38.459 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.456+0000 7f11917fa640 1 -- 172.21.15.63:0/4084571384 <== mon.2 v2:172.21.15.63:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4549 (secure 0 0 0) 0x7f1184025400 con 0x7f1194122140 2024-04-26T04:22:38.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.456+0000 7f11917fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T04:22:38.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.456+0000 7f11917fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:22:38.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.456+0000 7f11917fa640 1 -- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f1194122140 msgr2=0x7f119411d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:38.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.456+0000 7f11917fa640 1 --2- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f1194122140 0x7f119411d630 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1184011990 tx=0x7f11840046d0 comp rx=0 tx=0).stop 2024-04-26T04:22:38.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.456+0000 7f11917fa640 1 -- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f1194122140 msgr2=0x7f119411d630 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:22:38.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.456+0000 7f11917fa640 1 --2- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f1194122140 0x7f119411d630 secure :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1184011990 tx=0x7f11840046d0 comp rx=0 tx=0).stop 2024-04-26T04:22:38.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.456+0000 7f1198c86640 1 -- 172.21.15.63:0/4084571384 reap_dead start 2024-04-26T04:22:38.460 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-26T04:22:38.460 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "c", 2024-04-26T04:22:38.460 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 2, 2024-04-26T04:22:38.461 INFO:teuthology.orchestra.run.smithi063.stdout: "state": "peon", 2024-04-26T04:22:38.461 INFO:teuthology.orchestra.run.smithi063.stdout: "election_epoch": 20, 2024-04-26T04:22:38.461 INFO:teuthology.orchestra.run.smithi063.stdout: "uptime": 6055, 2024-04-26T04:22:38.461 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum": [ 2024-04-26T04:22:38.461 INFO:teuthology.orchestra.run.smithi063.stdout: 0, 2024-04-26T04:22:38.461 INFO:teuthology.orchestra.run.smithi063.stdout: 1, 2024-04-26T04:22:38.461 INFO:teuthology.orchestra.run.smithi063.stdout: 2 2024-04-26T04:22:38.461 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:22:38.461 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_age": 5, 2024-04-26T04:22:38.461 INFO:teuthology.orchestra.run.smithi063.stdout: "features": { 2024-04-26T04:22:38.461 INFO:teuthology.orchestra.run.smithi063.stdout: "required_con": "2450521708007866372", 2024-04-26T04:22:38.461 INFO:teuthology.orchestra.run.smithi063.stdout: "required_mon": [ 2024-04-26T04:22:38.461 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:22:38.461 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:22:38.462 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:22:38.462 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:22:38.462 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:22:38.462 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:22:38.462 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:22:38.462 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:22:38.462 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:22:38.462 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:22:38.462 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:22:38.462 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:22:38.462 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_con": "4540701547738038271", 2024-04-26T04:22:38.462 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_mon": [ 2024-04-26T04:22:38.462 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:22:38.463 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:22:38.463 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:22:38.463 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:22:38.463 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:22:38.463 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:22:38.463 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:22:38.463 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:22:38.463 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:22:38.463 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:22:38.463 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:22:38.463 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:22:38.463 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:38.463 INFO:teuthology.orchestra.run.smithi063.stdout: "outside_quorum": [], 2024-04-26T04:22:38.464 INFO:teuthology.orchestra.run.smithi063.stdout: "extra_probe_peers": [], 2024-04-26T04:22:38.464 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_provider": [], 2024-04-26T04:22:38.464 INFO:teuthology.orchestra.run.smithi063.stdout: "monmap": { 2024-04-26T04:22:38.464 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 1, 2024-04-26T04:22:38.464 INFO:teuthology.orchestra.run.smithi063.stdout: "fsid": "cd1c6a0c-4e4b-4014-89c0-2e535479ef39", 2024-04-26T04:22:38.464 INFO:teuthology.orchestra.run.smithi063.stdout: "modified": "2024-04-26T04:18:03.702339Z", 2024-04-26T04:22:38.464 INFO:teuthology.orchestra.run.smithi063.stdout: "created": "2024-04-26T04:18:03.702339Z", 2024-04-26T04:22:38.464 INFO:teuthology.orchestra.run.smithi063.stdout: "min_mon_release": 19, 2024-04-26T04:22:38.465 INFO:teuthology.orchestra.run.smithi063.stdout: "min_mon_release_name": "squid", 2024-04-26T04:22:38.465 INFO:teuthology.orchestra.run.smithi063.stdout: "election_strategy": 1, 2024-04-26T04:22:38.465 INFO:teuthology.orchestra.run.smithi063.stdout: "disallowed_leaders: ": "", 2024-04-26T04:22:38.465 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": false, 2024-04-26T04:22:38.465 INFO:teuthology.orchestra.run.smithi063.stdout: "tiebreaker_mon": "", 2024-04-26T04:22:38.465 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_ranks: ": "", 2024-04-26T04:22:38.465 INFO:teuthology.orchestra.run.smithi063.stdout: "features": { 2024-04-26T04:22:38.465 INFO:teuthology.orchestra.run.smithi063.stdout: "persistent": [ 2024-04-26T04:22:38.465 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:22:38.465 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:22:38.465 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:22:38.465 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:22:38.465 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:22:38.466 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:22:38.466 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:22:38.466 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:22:38.466 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:22:38.466 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:22:38.466 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:22:38.466 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:22:38.466 INFO:teuthology.orchestra.run.smithi063.stdout: "optional": [] 2024-04-26T04:22:38.466 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:38.466 INFO:teuthology.orchestra.run.smithi063.stdout: "mons": [ 2024-04-26T04:22:38.466 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:38.466 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 0, 2024-04-26T04:22:38.466 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a", 2024-04-26T04:22:38.467 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:22:38.467 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:22:38.467 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:38.467 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:22:38.467 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:3300", 2024-04-26T04:22:38.467 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:22:38.467 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:38.467 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:38.467 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:22:38.467 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6789", 2024-04-26T04:22:38.467 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:22:38.467 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:22:38.467 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:22:38.467 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:38.468 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6789/0", 2024-04-26T04:22:38.468 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6789/0", 2024-04-26T04:22:38.468 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:22:38.468 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:22:38.468 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:22:38.468 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:38.468 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:38.468 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 1, 2024-04-26T04:22:38.468 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b", 2024-04-26T04:22:38.468 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:22:38.468 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:22:38.468 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:38.468 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:22:38.468 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:3300", 2024-04-26T04:22:38.469 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:22:38.469 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:38.469 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:38.469 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:22:38.469 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:6789", 2024-04-26T04:22:38.469 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:22:38.469 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:22:38.469 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:22:38.469 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:38.469 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:6789/0", 2024-04-26T04:22:38.469 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.186:6789/0", 2024-04-26T04:22:38.469 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:22:38.469 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:22:38.469 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:22:38.470 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:38.470 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:38.470 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 2, 2024-04-26T04:22:38.470 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "c", 2024-04-26T04:22:38.470 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:22:38.470 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:22:38.470 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:38.470 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:22:38.470 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:3301", 2024-04-26T04:22:38.470 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:22:38.470 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:38.470 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:38.470 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:22:38.470 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6790", 2024-04-26T04:22:38.471 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:22:38.471 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:22:38.471 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:22:38.471 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:38.471 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6790/0", 2024-04-26T04:22:38.471 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6790/0", 2024-04-26T04:22:38.471 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:22:38.471 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:22:38.471 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:22:38.471 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:22:38.471 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:22:38.471 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:38.472 INFO:teuthology.orchestra.run.smithi063.stdout: "feature_map": { 2024-04-26T04:22:38.472 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": [ 2024-04-26T04:22:38.472 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:38.472 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:22:38.472 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:22:38.472 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 1 2024-04-26T04:22:38.472 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:22:38.472 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:22:38.472 INFO:teuthology.orchestra.run.smithi063.stdout: "client": [ 2024-04-26T04:22:38.472 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:38.472 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:22:38.472 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:22:38.472 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 1 2024-04-26T04:22:38.472 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:22:38.472 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:22:38.473 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:38.473 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": false 2024-04-26T04:22:38.473 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-26T04:22:38.473 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.456+0000 7f119a6e9640 1 -- 172.21.15.63:0/4084571384 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f1174035ec0 msgr2=0x7f1174038380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:38.473 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.456+0000 7f119a6e9640 1 --2- 172.21.15.63:0/4084571384 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f1174035ec0 0x7f1174038380 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f119409d610 tx=0x7f118c0a7070 comp rx=0 tx=0).stop 2024-04-26T04:22:38.474 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.456+0000 7f119a6e9640 10 monclient: shutdown 2024-04-26T04:22:38.474 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.456+0000 7f119a6e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:22:38.474 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.456+0000 7f119a6e9640 1 -- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f119409f140 msgr2=0x7f11941cb260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:38.474 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.456+0000 7f119a6e9640 1 --2- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f119409f140 0x7f11941cb260 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f117c00e840 tx=0x7f117c00ed00 comp rx=0 tx=0).stop 2024-04-26T04:22:38.474 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.456+0000 7f119a6e9640 1 -- 172.21.15.63:0/4084571384 shutdown_connections 2024-04-26T04:22:38.474 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.456+0000 7f119a6e9640 1 --2- 172.21.15.63:0/4084571384 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f1174035ec0 0x7f1174038380 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:38.474 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.456+0000 7f119a6e9640 1 --2- 172.21.15.63:0/4084571384 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f119409f140 0x7f11941cb260 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:38.474 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.456+0000 7f119a6e9640 1 -- 172.21.15.63:0/4084571384 >> 172.21.15.63:0/4084571384 conn(0x7f119409b4c0 msgr2=0x7f1194070b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:22:38.474 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.456+0000 7f119a6e9640 1 -- 172.21.15.63:0/4084571384 shutdown_connections 2024-04-26T04:22:38.474 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.456+0000 7f119a6e9640 1 -- 172.21.15.63:0/4084571384 wait complete. 2024-04-26T04:22:38.483 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T04:22:38.483 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-26T04:22:38.483 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T04:22:38.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.608+0000 7f546c4e1640 10 monclient: get_monmap_and_config 2024-04-26T04:22:38.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.608+0000 7f546c4e1640 10 monclient: build_initial_monmap 2024-04-26T04:22:38.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.608+0000 7f546c4e1640 10 monclient: monmap: 2024-04-26T04:22:38.611 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:22:38.611 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:38.611 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:22:38.611073+0000 2024-04-26T04:22:38.611 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:22:38.611073+0000 2024-04-26T04:22:38.611 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:22:38.612 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:38.612 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:22:38.612 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:22:38.612 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:22:38.612 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:22:38.612 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:22:38.612 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:38.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.608+0000 7f546c4e1640 1 Processor -- start 2024-04-26T04:22:38.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.608+0000 7f546c4e1640 1 -- start start 2024-04-26T04:22:38.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.608+0000 7f546c4e1640 10 monclient: init 2024-04-26T04:22:38.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.608+0000 7f546c4e1640 10 monclient: _reopen_session rank -1 2024-04-26T04:22:38.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.608+0000 7f546c4e1640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-04-26T04:22:38.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.608+0000 7f546c4e1640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f54640a2cd0 0x7f54640a30b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:38.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.608+0000 7f546c4e1640 10 monclient(hunting): picked mon.noname-c con 0x7f54640a2cd0 addr v2:172.21.15.186:3300/0 2024-04-26T04:22:38.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.608+0000 7f546c4e1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f54640a47d0 addr v1:172.21.15.63:6789/0 2024-04-26T04:22:38.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.608+0000 7f546c4e1640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f54640a4c00 0x7f54641373f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:38.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.608+0000 7f546c4e1640 10 monclient(hunting): picked mon.noname-b con 0x7f54640a4c00 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:22:38.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.608+0000 7f546c4e1640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54640a0fb0 con 0x7f54640a47d0 2024-04-26T04:22:38.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.608+0000 7f546c4e1640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:38.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.608+0000 7f546c4e1640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f54640a35f0 con 0x7f54640a2cd0 2024-04-26T04:22:38.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.608+0000 7f546c4e1640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:38.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.608+0000 7f546c4e1640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f54640a5380 con 0x7f54640a4c00 2024-04-26T04:22:38.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.608+0000 7f546c4e1640 10 monclient(hunting): _renew_subs 2024-04-26T04:22:38.616 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.608+0000 7f546c4e1640 10 monclient(hunting): authenticate will time out at 1190.319580s 2024-04-26T04:22:38.616 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.608+0000 7f546aa7e640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f54640a4c00 0x7f54641373f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:38.616 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.608+0000 7f5469a7c640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f54640a47d0 0x7f5464136ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:58962/0 (socket says 172.21.15.63:58962) 2024-04-26T04:22:38.616 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.608+0000 7f546aa7e640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f54640a4c00 0x7f54641373f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:50048/0 (socket says 172.21.15.63:50048) 2024-04-26T04:22:38.616 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.608+0000 7f5469a7c640 1 -- 172.21.15.63:0/4270338208 learned_addr learned my addr 172.21.15.63:0/4270338208 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-04-26T04:22:38.616 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.608+0000 7f546aa7e640 10 monclient(hunting): get_auth_request con 0x7f54640a4c00 auth_method 0 2024-04-26T04:22:38.616 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.608+0000 7f546aa7e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:38.616 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.608+0000 7f546aa7e640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:38.616 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.608+0000 7f546aa7e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:38.616 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.608+0000 7f546aa7e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:22:38.616 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.608+0000 7f546aa7e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:22:38.616 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.608+0000 7f546aa7e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:22:38.616 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.612+0000 7f546a27d640 1 --2- 172.21.15.63:0/4270338208 >> v2:172.21.15.186:3300/0 conn(0x7f54640a2cd0 0x7f54640a30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:38.616 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.612+0000 7f546a27d640 10 monclient(hunting): get_auth_request con 0x7f54640a2cd0 auth_method 0 2024-04-26T04:22:38.616 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.612+0000 7f546a27d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:38.617 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.612+0000 7f546a27d640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:38.617 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.612+0000 7f546a27d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:38.617 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.612+0000 7f546aa7e640 10 monclient(hunting): handle_auth_done global_id 24119 payload 274 2024-04-26T04:22:38.617 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.612+0000 7f546aa7e640 1 -- 172.21.15.63:0/4270338208 >> v2:172.21.15.186:3300/0 conn(0x7f54640a2cd0 msgr2=0x7f54640a30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:22:38.617 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.612+0000 7f546aa7e640 1 --2- 172.21.15.63:0/4270338208 >> v2:172.21.15.186:3300/0 conn(0x7f54640a2cd0 0x7f54640a30b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:38.617 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.612+0000 7f546aa7e640 1 -- 172.21.15.63:0/4270338208 >> v1:172.21.15.63:6789/0 conn(0x7f54640a47d0 legacy=0x7f5464136ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:22:38.617 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.612+0000 7f546aa7e640 10 monclient: _finish_hunting 0 2024-04-26T04:22:38.617 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.612+0000 7f546aa7e640 1 monclient: found mon.noname-b 2024-04-26T04:22:38.617 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.612+0000 7f546aa7e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:22:38.617 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.612+0000 7f546aa7e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.63:3301/0 2024-04-26T04:22:38.617 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.612+0000 7f546aa7e640 1 -- 172.21.15.63:0/4270338208 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54640a1290 con 0x7f54640a4c00 2024-04-26T04:22:38.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.616+0000 7f546aa7e640 1 -- 172.21.15.63:0/4270338208 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f54640a4c00 msgr2=0x7f54641373f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:22:38.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.616+0000 7f546aa7e640 1 --2- 172.21.15.63:0/4270338208 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f54640a4c00 0x7f54641373f0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f545c009d90 tx=0x7f545c01c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a175ddc7fca5a3f server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:22:38.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.616+0000 7f546927b640 1 -- 172.21.15.63:0/4270338208 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f545c0a6710 con 0x7f54640a4c00 2024-04-26T04:22:38.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.616+0000 7f546927b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:38.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.616+0000 7f546927b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T04:22:38.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.616+0000 7f546927b640 10 monclient: dump: 2024-04-26T04:22:38.621 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:38.621 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:38.621 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:38.621 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:38.621 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:38.621 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:38.621 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:38.621 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:38.621 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:38.621 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:38.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.616+0000 7f546927b640 10 monclient: _finish_auth 0 2024-04-26T04:22:38.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.616+0000 7f546927b640 10 monclient: _check_auth_tickets 2024-04-26T04:22:38.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.616+0000 7f546927b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:22:38.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.616+0000 7f546927b640 1 -- 172.21.15.63:0/4270338208 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f545c0a68b0 con 0x7f54640a4c00 2024-04-26T04:22:38.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.616+0000 7f546c4e1640 5 monclient: authenticate success, global_id 24119 2024-04-26T04:22:38.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.616+0000 7f546c4e1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T04:22:38.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.616+0000 7f546927b640 10 monclient: handle_config config(0 keys) 2024-04-26T04:22:38.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.616+0000 7f546c4e1640 10 monclient: get_monmap_and_config success 2024-04-26T04:22:38.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.616+0000 7f546c4e1640 4 set_mon_vals no callback set 2024-04-26T04:22:38.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.616+0000 7f546927b640 1 -- 172.21.15.63:0/4270338208 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f545c0a6c80 con 0x7f54640a4c00 2024-04-26T04:22:38.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.616+0000 7f546927b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:38.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.616+0000 7f546927b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:22:38.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.616+0000 7f546927b640 10 monclient: dump: 2024-04-26T04:22:38.622 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:38.622 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:38.622 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:38.623 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:38.623 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:38.623 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:38.623 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:38.623 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:38.623 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:38.623 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:38.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.616+0000 7f546c4e1640 10 monclient: shutdown 2024-04-26T04:22:38.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.616+0000 7f546c4e1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:22:38.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.616+0000 7f546c4e1640 1 -- 172.21.15.63:0/4270338208 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f54640a4c00 msgr2=0x7f54641373f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:38.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.616+0000 7f546c4e1640 1 --2- 172.21.15.63:0/4270338208 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f54640a4c00 0x7f54641373f0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f545c009d90 tx=0x7f545c01c880 comp rx=0 tx=0).stop 2024-04-26T04:22:38.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.616+0000 7f546c4e1640 1 -- 172.21.15.63:0/4270338208 shutdown_connections 2024-04-26T04:22:38.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.616+0000 7f546c4e1640 1 --2- 172.21.15.63:0/4270338208 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f54640a4c00 0x7f54641373f0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:38.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.616+0000 7f546c4e1640 1 --2- 172.21.15.63:0/4270338208 >> v2:172.21.15.186:3300/0 conn(0x7f54640a2cd0 0x7f54640a30b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:38.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.616+0000 7f546c4e1640 1 -- 172.21.15.63:0/4270338208 >> 172.21.15.63:0/4270338208 conn(0x7f54640a5f40 msgr2=0x7f54640a6320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:22:38.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.616+0000 7f546c4e1640 1 -- 172.21.15.63:0/4270338208 shutdown_connections 2024-04-26T04:22:38.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.616+0000 7f546c4e1640 1 -- 172.21.15.63:0/4270338208 wait complete. 2024-04-26T04:22:38.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.616+0000 7f546c4e1640 10 monclient: build_initial_monmap 2024-04-26T04:22:38.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.616+0000 7f546c4e1640 10 monclient: monmap: 2024-04-26T04:22:38.624 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:22:38.624 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:22:38.624 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:22:38.624 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:22:38.624 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:22:38.624 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:38.624 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:22:38.624 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:22:38.624 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:22:38.625 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:38.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.616+0000 7f546c4e1640 1 Processor -- start 2024-04-26T04:22:38.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.616+0000 7f546c4e1640 1 -- start start 2024-04-26T04:22:38.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.616+0000 7f546c4e1640 10 monclient: init 2024-04-26T04:22:38.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f546c4e1640 10 monclient: _reopen_session rank -1 2024-04-26T04:22:38.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f546c4e1640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T04:22:38.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f546c4e1640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f54640a2cd0 0x7f54641cef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:38.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f546c4e1640 10 monclient(hunting): picked mon.noname-c con 0x7f54640a2cd0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:22:38.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f546c4e1640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f54640a47d0 0x7f5464135d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:38.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f546c4e1640 10 monclient(hunting): picked mon.noname-a con 0x7f54640a47d0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:22:38.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f546c4e1640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54640a4c00 0x7f546413bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:38.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f546c4e1640 10 monclient(hunting): picked mon.noname-b con 0x7f54640a4c00 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:22:38.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f546c4e1640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:38.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f546c4e1640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f5464141a50 con 0x7f54640a47d0 2024-04-26T04:22:38.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f546c4e1640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:38.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f546c4e1640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5464137b40 con 0x7f54640a4c00 2024-04-26T04:22:38.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f546c4e1640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:38.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f546c4e1640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f5464141bd0 con 0x7f54640a2cd0 2024-04-26T04:22:38.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f546c4e1640 10 monclient(hunting): _renew_subs 2024-04-26T04:22:38.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f546c4e1640 10 monclient(hunting): authenticate will time out at 1190.328451s 2024-04-26T04:22:38.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f5469a7c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f54640a47d0 0x7f5464135d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:38.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f5469a7c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f54640a47d0 0x7f5464135d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:48380/0 (socket says 172.21.15.63:48380) 2024-04-26T04:22:38.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f5469a7c640 1 -- 172.21.15.63:0/2584541640 learned_addr learned my addr 172.21.15.63:0/2584541640 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:22:38.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f5469a7c640 10 monclient(hunting): get_auth_request con 0x7f54640a47d0 auth_method 0 2024-04-26T04:22:38.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f5469a7c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:38.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f5469a7c640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:38.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f5469a7c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:38.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f546a27d640 1 --2- 172.21.15.63:0/2584541640 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f54640a2cd0 0x7f54641cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:38.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f546a27d640 10 monclient(hunting): get_auth_request con 0x7f54640a2cd0 auth_method 0 2024-04-26T04:22:38.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f546a27d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:38.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f546a27d640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:38.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f546a27d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:38.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f5469a7c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:22:38.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f5469a7c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:22:38.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f5469a7c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:22:38.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f546a27d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:22:38.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f546a27d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:22:38.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f546a27d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:22:38.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f546aa7e640 1 --2- 172.21.15.63:0/2584541640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54640a4c00 0x7f546413bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:38.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f546aa7e640 10 monclient(hunting): get_auth_request con 0x7f54640a4c00 auth_method 0 2024-04-26T04:22:38.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f546aa7e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:38.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f546aa7e640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:38.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f546aa7e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:38.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f5469a7c640 10 monclient(hunting): handle_auth_done global_id 5214 payload 824 2024-04-26T04:22:38.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f5469a7c640 1 -- 172.21.15.63:0/2584541640 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f54640a2cd0 msgr2=0x7f54641cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:38.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f5469a7c640 1 --2- 172.21.15.63:0/2584541640 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f54640a2cd0 0x7f54641cef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:38.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f5469a7c640 1 -- 172.21.15.63:0/2584541640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54640a4c00 msgr2=0x7f546413bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:38.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f5469a7c640 1 --2- 172.21.15.63:0/2584541640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54640a4c00 0x7f546413bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:38.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f5469a7c640 10 monclient: _finish_hunting 0 2024-04-26T04:22:38.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f5469a7c640 1 monclient: found mon.noname-a 2024-04-26T04:22:38.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f5469a7c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:22:38.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f5469a7c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:22:38.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f5469a7c640 1 -- 172.21.15.63:0/2584541640 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5464120710 con 0x7f54640a47d0 2024-04-26T04:22:38.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f546a27d640 1 --2- 172.21.15.63:0/2584541640 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f54640a2cd0 0x7f54641cef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T04:22:38.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f546aa7e640 1 --2- 172.21.15.63:0/2584541640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54640a4c00 0x7f546413bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T04:22:38.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f5469a7c640 1 -- 172.21.15.63:0/2584541640 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f54640a47d0 msgr2=0x7f5464135d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:22:38.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f5469a7c640 1 --2- 172.21.15.63:0/2584541640 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f54640a47d0 0x7f5464135d80 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f545400c470 tx=0x7f545400c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:22:38.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f545affd640 1 -- 172.21.15.63:0/2584541640 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5454007070 con 0x7f54640a47d0 2024-04-26T04:22:38.634 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f545affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:38.634 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f545affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:22:38.634 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f545affd640 10 monclient: dump: 2024-04-26T04:22:38.634 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:38.634 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:38.634 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:38.634 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:38.634 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:38.634 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:38.634 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:38.634 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:38.634 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:38.634 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:38.634 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f545affd640 10 monclient: _finish_auth 0 2024-04-26T04:22:38.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f545affd640 10 monclient: _check_auth_tickets 2024-04-26T04:22:38.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f545affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:22:38.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f545affd640 1 -- 172.21.15.63:0/2584541640 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5454007210 con 0x7f54640a47d0 2024-04-26T04:22:38.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f545affd640 10 monclient: handle_config config(0 keys) 2024-04-26T04:22:38.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f545affd640 1 -- 172.21.15.63:0/2584541640 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5454007600 con 0x7f54640a47d0 2024-04-26T04:22:38.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f546c4e1640 5 monclient: authenticate success, global_id 5214 2024-04-26T04:22:38.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f546c4e1640 10 monclient: _renew_subs 2024-04-26T04:22:38.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f546c4e1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:22:38.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f546c4e1640 1 -- 172.21.15.63:0/2584541640 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f546413e260 con 0x7f54640a47d0 2024-04-26T04:22:38.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f5468a7a640 4 set_mon_vals no callback set 2024-04-26T04:22:38.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f545affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:38.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f545affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:22:38.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f545affd640 10 monclient: dump: 2024-04-26T04:22:38.635 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:38.635 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:38.636 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:38.636 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:38.636 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:38.636 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:38.636 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:38.636 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:38.636 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:38.636 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:38.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f546c4e1640 10 monclient: _renew_subs 2024-04-26T04:22:38.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f546c4e1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:22:38.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.620+0000 7f546c4e1640 1 -- 172.21.15.63:0/2584541640 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f546413d980 con 0x7f54640a47d0 2024-04-26T04:22:38.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.624+0000 7f546c4e1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:22:38.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.624+0000 7f546c4e1640 1 --2- 172.21.15.63:0/2584541640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f546413c270 0x7f546413c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:38.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.624+0000 7f546c4e1640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:38.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.624+0000 7f546c4e1640 1 -- 172.21.15.63:0/2584541640 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5464120090 con 0x7f546413c270 2024-04-26T04:22:38.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.624+0000 7f545affd640 1 -- 172.21.15.63:0/2584541640 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f545401b400 con 0x7f54640a47d0 2024-04-26T04:22:38.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.624+0000 7f546a27d640 1 --2- 172.21.15.63:0/2584541640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f546413c270 0x7f546413c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:38.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.624+0000 7f546a27d640 10 monclient: get_auth_request con 0x7f546413c270 auth_method 0 2024-04-26T04:22:38.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.624+0000 7f546a27d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:38.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.624+0000 7f546a27d640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:38.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.624+0000 7f546a27d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:38.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.624+0000 7f545affd640 1 --2- 172.21.15.63:0/2584541640 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f5448035ec0 0x7f5448038380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:38.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.624+0000 7f546a27d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:22:38.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.624+0000 7f546a27d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:22:38.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.624+0000 7f546a27d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:22:38.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.624+0000 7f545affd640 1 -- 172.21.15.63:0/2584541640 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(212..212 src has 1..212) ==== 4346+0+0 (secure 0 0 0) 0x7f5454048a10 con 0x7f54640a47d0 2024-04-26T04:22:38.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.624+0000 7f546aa7e640 1 --2- 172.21.15.63:0/2584541640 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f5448035ec0 0x7f5448038380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:38.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.624+0000 7f546aa7e640 10 monclient: get_auth_request con 0x7f5448035ec0 auth_method 0 2024-04-26T04:22:38.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.624+0000 7f546a27d640 10 monclient(hunting): handle_auth_done global_id 5233 payload 824 2024-04-26T04:22:38.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.624+0000 7f546a27d640 1 -- 172.21.15.63:0/2584541640 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f546413cb90 con 0x7f546413c270 2024-04-26T04:22:38.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.628+0000 7f546a27d640 1 -- 172.21.15.63:0/2584541640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f546413c270 msgr2=0x7f546413c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:22:38.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.628+0000 7f546a27d640 1 --2- 172.21.15.63:0/2584541640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f546413c270 0x7f546413c650 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f5460010bd0 tx=0x7f546000e5a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:22:38.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.628+0000 7f546aa7e640 1 --2- 172.21.15.63:0/2584541640 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f5448035ec0 0x7f5448038380 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f54640a09d0 tx=0x7f545c0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:22:38.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.632+0000 7f545affd640 1 -- 172.21.15.63:0/2584541640 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5460018070 con 0x7f546413c270 2024-04-26T04:22:38.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.632+0000 7f545affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:38.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.632+0000 7f545affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T04:22:38.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.632+0000 7f545affd640 10 monclient: dump: 2024-04-26T04:22:38.639 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:38.639 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:38.639 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:38.639 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:38.639 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:38.639 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:38.639 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:38.639 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:38.639 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:38.640 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:38.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.632+0000 7f545affd640 1 -- 172.21.15.63:0/2584541640 <== mon.1 v2:172.21.15.186:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f546001d450 con 0x7f546413c270 2024-04-26T04:22:38.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.632+0000 7f545affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:22:38.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.632+0000 7f545affd640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:22:38.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.632+0000 7f545affd640 1 -- 172.21.15.63:0/2584541640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f546413c270 msgr2=0x7f546413c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:38.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.632+0000 7f545affd640 1 --2- 172.21.15.63:0/2584541640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f546413c270 0x7f546413c650 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f5460010bd0 tx=0x7f546000e5a0 comp rx=0 tx=0).stop 2024-04-26T04:22:38.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.632+0000 7f545affd640 1 -- 172.21.15.63:0/2584541640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f546413c270 msgr2=0x7f546413c650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:22:38.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.632+0000 7f545affd640 1 --2- 172.21.15.63:0/2584541640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f546413c270 0x7f546413c650 secure :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f5460010bd0 tx=0x7f546000e5a0 comp rx=0 tx=0).stop 2024-04-26T04:22:38.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.640+0000 7f546c4e1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T04:22:38.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.640+0000 7f546c4e1640 1 --2- 172.21.15.63:0/2584541640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5464137db0 0x7f54641381c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:38.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.640+0000 7f546c4e1640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:38.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.640+0000 7f546c4e1640 1 -- 172.21.15.63:0/2584541640 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f54640a3f40 con 0x7f5464137db0 2024-04-26T04:22:38.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.640+0000 7f546a27d640 1 --2- 172.21.15.63:0/2584541640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5464137db0 0x7f54641381c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:38.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.640+0000 7f546a27d640 10 monclient: get_auth_request con 0x7f5464137db0 auth_method 0 2024-04-26T04:22:38.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.640+0000 7f546a27d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:38.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.640+0000 7f546a27d640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:38.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.640+0000 7f546a27d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:38.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.640+0000 7f546a27d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:22:38.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.640+0000 7f546a27d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:22:38.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.640+0000 7f546a27d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:22:38.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.640+0000 7f546a27d640 10 monclient(hunting): handle_auth_done global_id 5236 payload 824 2024-04-26T04:22:38.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.640+0000 7f546a27d640 1 -- 172.21.15.63:0/2584541640 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f54641cfdf0 con 0x7f5464137db0 2024-04-26T04:22:38.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.644+0000 7f546a27d640 1 -- 172.21.15.63:0/2584541640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5464137db0 msgr2=0x7f54641381c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:22:38.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.644+0000 7f546a27d640 1 --2- 172.21.15.63:0/2584541640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5464137db0 0x7f54641381c0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f546001a000 tx=0x7f54600046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:22:38.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.644+0000 7f545affd640 1 -- 172.21.15.63:0/2584541640 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5460004230 con 0x7f5464137db0 2024-04-26T04:22:38.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.644+0000 7f545affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:38.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.644+0000 7f545affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T04:22:38.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.644+0000 7f545affd640 10 monclient: dump: 2024-04-26T04:22:38.646 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:38.646 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:38.647 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:38.647 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:38.647 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:38.647 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:38.647 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:38.647 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:38.647 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:38.647 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:38.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.644+0000 7f545affd640 1 -- 172.21.15.63:0/2584541640 <== mon.1 v2:172.21.15.186:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4891 (secure 0 0 0) 0x7f5460004230 con 0x7f5464137db0 2024-04-26T04:22:38.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.644+0000 7f545affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T04:22:38.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.644+0000 7f545affd640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:22:38.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.644+0000 7f545affd640 1 -- 172.21.15.63:0/2584541640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5464137db0 msgr2=0x7f54641381c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:38.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.644+0000 7f545affd640 1 --2- 172.21.15.63:0/2584541640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5464137db0 0x7f54641381c0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f546001a000 tx=0x7f54600046d0 comp rx=0 tx=0).stop 2024-04-26T04:22:38.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.644+0000 7f545affd640 1 -- 172.21.15.63:0/2584541640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5464137db0 msgr2=0x7f54641381c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:22:38.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.644+0000 7f545affd640 1 --2- 172.21.15.63:0/2584541640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5464137db0 0x7f54641381c0 secure :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f546001a000 tx=0x7f54600046d0 comp rx=0 tx=0).stop 2024-04-26T04:22:38.648 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-26T04:22:38.648 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b", 2024-04-26T04:22:38.648 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 1, 2024-04-26T04:22:38.649 INFO:teuthology.orchestra.run.smithi063.stdout: "state": "peon", 2024-04-26T04:22:38.649 INFO:teuthology.orchestra.run.smithi063.stdout: "election_epoch": 20, 2024-04-26T04:22:38.649 INFO:teuthology.orchestra.run.smithi063.stdout: "uptime": 249754, 2024-04-26T04:22:38.649 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum": [ 2024-04-26T04:22:38.649 INFO:teuthology.orchestra.run.smithi063.stdout: 0, 2024-04-26T04:22:38.649 INFO:teuthology.orchestra.run.smithi063.stdout: 1, 2024-04-26T04:22:38.649 INFO:teuthology.orchestra.run.smithi063.stdout: 2 2024-04-26T04:22:38.649 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:22:38.649 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_age": 5, 2024-04-26T04:22:38.649 INFO:teuthology.orchestra.run.smithi063.stdout: "features": { 2024-04-26T04:22:38.649 INFO:teuthology.orchestra.run.smithi063.stdout: "required_con": "2450521708007866372", 2024-04-26T04:22:38.649 INFO:teuthology.orchestra.run.smithi063.stdout: "required_mon": [ 2024-04-26T04:22:38.649 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:22:38.649 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:22:38.649 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:22:38.650 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:22:38.650 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:22:38.650 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:22:38.650 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:22:38.650 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:22:38.650 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:22:38.650 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:22:38.650 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:22:38.650 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:22:38.650 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_con": "4540701547738038271", 2024-04-26T04:22:38.650 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_mon": [ 2024-04-26T04:22:38.650 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:22:38.650 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:22:38.650 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:22:38.651 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:22:38.651 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:22:38.651 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:22:38.651 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:22:38.651 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:22:38.651 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:22:38.651 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:22:38.651 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:22:38.651 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:22:38.651 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:38.651 INFO:teuthology.orchestra.run.smithi063.stdout: "outside_quorum": [], 2024-04-26T04:22:38.651 INFO:teuthology.orchestra.run.smithi063.stdout: "extra_probe_peers": [], 2024-04-26T04:22:38.651 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_provider": [], 2024-04-26T04:22:38.651 INFO:teuthology.orchestra.run.smithi063.stdout: "monmap": { 2024-04-26T04:22:38.651 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 1, 2024-04-26T04:22:38.652 INFO:teuthology.orchestra.run.smithi063.stdout: "fsid": "cd1c6a0c-4e4b-4014-89c0-2e535479ef39", 2024-04-26T04:22:38.652 INFO:teuthology.orchestra.run.smithi063.stdout: "modified": "2024-04-26T04:18:03.702339Z", 2024-04-26T04:22:38.652 INFO:teuthology.orchestra.run.smithi063.stdout: "created": "2024-04-26T04:18:03.702339Z", 2024-04-26T04:22:38.652 INFO:teuthology.orchestra.run.smithi063.stdout: "min_mon_release": 19, 2024-04-26T04:22:38.652 INFO:teuthology.orchestra.run.smithi063.stdout: "min_mon_release_name": "squid", 2024-04-26T04:22:38.652 INFO:teuthology.orchestra.run.smithi063.stdout: "election_strategy": 1, 2024-04-26T04:22:38.652 INFO:teuthology.orchestra.run.smithi063.stdout: "disallowed_leaders: ": "", 2024-04-26T04:22:38.652 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": false, 2024-04-26T04:22:38.652 INFO:teuthology.orchestra.run.smithi063.stdout: "tiebreaker_mon": "", 2024-04-26T04:22:38.652 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_ranks: ": "", 2024-04-26T04:22:38.652 INFO:teuthology.orchestra.run.smithi063.stdout: "features": { 2024-04-26T04:22:38.652 INFO:teuthology.orchestra.run.smithi063.stdout: "persistent": [ 2024-04-26T04:22:38.652 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:22:38.652 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:22:38.652 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:22:38.653 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:22:38.653 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:22:38.653 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:22:38.653 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:22:38.653 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:22:38.653 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:22:38.653 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:22:38.653 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:22:38.653 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:22:38.653 INFO:teuthology.orchestra.run.smithi063.stdout: "optional": [] 2024-04-26T04:22:38.653 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:38.653 INFO:teuthology.orchestra.run.smithi063.stdout: "mons": [ 2024-04-26T04:22:38.653 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:38.653 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 0, 2024-04-26T04:22:38.653 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a", 2024-04-26T04:22:38.654 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:22:38.654 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:22:38.654 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:38.654 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:22:38.654 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:3300", 2024-04-26T04:22:38.654 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:22:38.654 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:38.654 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:38.654 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:22:38.654 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6789", 2024-04-26T04:22:38.654 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:22:38.654 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:22:38.654 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:22:38.654 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:38.655 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6789/0", 2024-04-26T04:22:38.655 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6789/0", 2024-04-26T04:22:38.655 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:22:38.655 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:22:38.655 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:22:38.655 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:38.655 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:38.655 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 1, 2024-04-26T04:22:38.655 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b", 2024-04-26T04:22:38.655 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:22:38.655 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:22:38.655 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:38.655 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:22:38.655 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:3300", 2024-04-26T04:22:38.655 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:22:38.656 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:38.656 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:38.656 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:22:38.656 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:6789", 2024-04-26T04:22:38.656 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:22:38.656 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:22:38.656 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:22:38.656 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:38.656 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:6789/0", 2024-04-26T04:22:38.656 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.186:6789/0", 2024-04-26T04:22:38.656 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:22:38.656 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:22:38.656 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:22:38.656 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:38.656 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:38.657 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 2, 2024-04-26T04:22:38.657 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "c", 2024-04-26T04:22:38.657 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:22:38.657 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:22:38.657 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:38.657 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:22:38.657 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:3301", 2024-04-26T04:22:38.657 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:22:38.657 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:38.657 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:38.657 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:22:38.657 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6790", 2024-04-26T04:22:38.657 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:22:38.657 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:22:38.657 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:22:38.658 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:38.658 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6790/0", 2024-04-26T04:22:38.658 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6790/0", 2024-04-26T04:22:38.658 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:22:38.658 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:22:38.658 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:22:38.658 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:22:38.658 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:22:38.658 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:38.658 INFO:teuthology.orchestra.run.smithi063.stdout: "feature_map": { 2024-04-26T04:22:38.658 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": [ 2024-04-26T04:22:38.658 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:38.658 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:22:38.658 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:22:38.659 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 1 2024-04-26T04:22:38.659 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:22:38.659 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:22:38.659 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": [ 2024-04-26T04:22:38.659 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:38.659 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:22:38.659 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:22:38.659 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 3 2024-04-26T04:22:38.659 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:22:38.659 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:22:38.659 INFO:teuthology.orchestra.run.smithi063.stdout: "client": [ 2024-04-26T04:22:38.659 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:38.659 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:22:38.659 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:22:38.659 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 7 2024-04-26T04:22:38.660 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:22:38.660 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:22:38.660 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": [ 2024-04-26T04:22:38.660 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:38.660 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:22:38.660 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:22:38.660 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 1 2024-04-26T04:22:38.660 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:22:38.660 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:22:38.660 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:38.660 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": false 2024-04-26T04:22:38.660 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-26T04:22:38.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.644+0000 7f546c4e1640 1 -- 172.21.15.63:0/2584541640 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f5448035ec0 msgr2=0x7f5448038380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:38.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.644+0000 7f546c4e1640 1 --2- 172.21.15.63:0/2584541640 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f5448035ec0 0x7f5448038380 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f54640a09d0 tx=0x7f545c0a7000 comp rx=0 tx=0).stop 2024-04-26T04:22:38.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.644+0000 7f546c4e1640 10 monclient: shutdown 2024-04-26T04:22:38.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.644+0000 7f546c4e1640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:22:38.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.644+0000 7f546c4e1640 1 -- 172.21.15.63:0/2584541640 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f54640a47d0 msgr2=0x7f5464135d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:38.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.644+0000 7f546c4e1640 1 --2- 172.21.15.63:0/2584541640 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f54640a47d0 0x7f5464135d80 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f545400c470 tx=0x7f545400c930 comp rx=0 tx=0).stop 2024-04-26T04:22:38.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.644+0000 7f546aa7e640 1 -- 172.21.15.63:0/2584541640 reap_dead start 2024-04-26T04:22:38.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.644+0000 7f546aa7e640 1 -- 172.21.15.63:0/2584541640 reap_dead start 2024-04-26T04:22:38.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.644+0000 7f546c4e1640 1 -- 172.21.15.63:0/2584541640 shutdown_connections 2024-04-26T04:22:38.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.644+0000 7f546c4e1640 1 -- 172.21.15.63:0/2584541640 >> 172.21.15.63:0/2584541640 conn(0x7f54640a5f40 msgr2=0x7f5464072ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:22:38.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.644+0000 7f546c4e1640 1 -- 172.21.15.63:0/2584541640 shutdown_connections 2024-04-26T04:22:38.662 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.644+0000 7f546c4e1640 1 -- 172.21.15.63:0/2584541640 wait complete. 2024-04-26T04:22:38.664 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T04:22:38.664 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-04-26T04:22:38.664 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-04-26T04:22:38.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.772+0000 7fb6755e0640 10 monclient: get_monmap_and_config 2024-04-26T04:22:38.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.772+0000 7fb6755e0640 10 monclient: build_initial_monmap 2024-04-26T04:22:38.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.772+0000 7fb6755e0640 10 monclient: monmap: 2024-04-26T04:22:38.777 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:22:38.777 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:38.777 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:22:38.776544+0000 2024-04-26T04:22:38.777 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:22:38.776544+0000 2024-04-26T04:22:38.777 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:22:38.777 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:38.777 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:22:38.777 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:22:38.777 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:22:38.777 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:22:38.777 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:22:38.777 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:38.779 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.772+0000 7fb6755e0640 1 Processor -- start 2024-04-26T04:22:38.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.772+0000 7fb6755e0640 1 -- start start 2024-04-26T04:22:38.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.772+0000 7fb6755e0640 10 monclient: init 2024-04-26T04:22:38.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.776+0000 7fb6755e0640 10 monclient: _reopen_session rank -1 2024-04-26T04:22:38.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.776+0000 7fb6755e0640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-04-26T04:22:38.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.776+0000 7fb6755e0640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7fb6700a2cd0 0x7fb6700a30b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:38.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.776+0000 7fb6755e0640 10 monclient(hunting): picked mon.noname-a con 0x7fb6700a2cd0 addr v2:172.21.15.63:3300/0 2024-04-26T04:22:38.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.776+0000 7fb6755e0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb6700a47d0 addr v1:172.21.15.186:6789/0 2024-04-26T04:22:38.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.776+0000 7fb6755e0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb6700a4c00 addr v1:172.21.15.63:6789/0 2024-04-26T04:22:38.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.776+0000 7fb6755e0640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb670137b00 con 0x7fb6700a4c00 2024-04-26T04:22:38.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.776+0000 7fb6755e0640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6700a0fb0 con 0x7fb6700a47d0 2024-04-26T04:22:38.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.776+0000 7fb6755e0640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:38.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.776+0000 7fb6755e0640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7fb6700a35f0 con 0x7fb6700a2cd0 2024-04-26T04:22:38.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.776+0000 7fb6755e0640 10 monclient(hunting): _renew_subs 2024-04-26T04:22:38.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.776+0000 7fb6755e0640 10 monclient(hunting): authenticate will time out at 1190.484710s 2024-04-26T04:22:38.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.776+0000 7fb66effd640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7fb6700a2cd0 0x7fb6700a30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:38.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.776+0000 7fb66f7fe640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7fb6700a4c00 0x7fb6701373f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:58976/0 (socket says 172.21.15.63:58976) 2024-04-26T04:22:38.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.776+0000 7fb66f7fe640 1 -- 172.21.15.63:0/1717185506 learned_addr learned my addr 172.21.15.63:0/1717185506 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-04-26T04:22:38.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.776+0000 7fb66effd640 10 monclient(hunting): get_auth_request con 0x7fb6700a2cd0 auth_method 0 2024-04-26T04:22:38.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.776+0000 7fb66effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:38.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.776+0000 7fb66effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:38.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.776+0000 7fb66effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:38.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.776+0000 7fb66effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:22:38.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.776+0000 7fb66effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:22:38.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.776+0000 7fb66effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:22:38.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.776+0000 7fb66effd640 10 monclient(hunting): handle_auth_done global_id 5217 payload 274 2024-04-26T04:22:38.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.776+0000 7fb66effd640 1 -- 172.21.15.63:0/1717185506 >> v1:172.21.15.186:6789/0 conn(0x7fb6700a47d0 legacy=0x7fb670136ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:22:38.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.776+0000 7fb66effd640 1 -- 172.21.15.63:0/1717185506 >> v1:172.21.15.63:6789/0 conn(0x7fb6700a4c00 legacy=0x7fb6701373f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:22:38.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.776+0000 7fb66effd640 10 monclient: _finish_hunting 0 2024-04-26T04:22:38.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.776+0000 7fb66effd640 1 monclient: found mon.noname-a 2024-04-26T04:22:38.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.776+0000 7fb66effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:22:38.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.776+0000 7fb66effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:22:38.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.776+0000 7fb66effd640 1 -- 172.21.15.63:0/1717185506 --> v2:172.21.15.63:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6700a5380 con 0x7fb6700a2cd0 2024-04-26T04:22:38.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.780+0000 7fb66effd640 1 -- 172.21.15.63:0/1717185506 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb6700a2cd0 msgr2=0x7fb6700a30b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:22:38.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.780+0000 7fb66effd640 1 --2- 172.21.15.63:0/1717185506 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb6700a2cd0 0x7fb6700a30b0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fb6600097c0 tx=0x7fb66001c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af32e07cad975a63 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:22:38.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.780+0000 7fb66dffb640 1 -- 172.21.15.63:0/1717185506 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6600b1070 con 0x7fb6700a2cd0 2024-04-26T04:22:38.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.780+0000 7fb66dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:38.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.780+0000 7fb66dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:22:38.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.780+0000 7fb66dffb640 10 monclient: dump: 2024-04-26T04:22:38.784 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:38.784 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:38.784 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:38.784 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:38.784 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:38.784 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:38.784 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:38.784 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:38.784 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:38.784 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:38.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.780+0000 7fb66dffb640 10 monclient: _finish_auth 0 2024-04-26T04:22:38.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.780+0000 7fb66dffb640 10 monclient: _check_auth_tickets 2024-04-26T04:22:38.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.780+0000 7fb66dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:22:38.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.780+0000 7fb66dffb640 1 -- 172.21.15.63:0/1717185506 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6600ac640 con 0x7fb6700a2cd0 2024-04-26T04:22:38.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.780+0000 7fb66dffb640 10 monclient: handle_config config(0 keys) 2024-04-26T04:22:38.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.780+0000 7fb66dffb640 1 -- 172.21.15.63:0/1717185506 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6600aca10 con 0x7fb6700a2cd0 2024-04-26T04:22:38.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.780+0000 7fb6755e0640 5 monclient: authenticate success, global_id 5217 2024-04-26T04:22:38.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.780+0000 7fb6755e0640 10 monclient: get_monmap_and_config success 2024-04-26T04:22:38.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.780+0000 7fb6755e0640 4 set_mon_vals no callback set 2024-04-26T04:22:38.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.780+0000 7fb66dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:38.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.780+0000 7fb66dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:22:38.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.780+0000 7fb66dffb640 10 monclient: dump: 2024-04-26T04:22:38.785 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:38.785 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:38.785 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:38.785 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:38.785 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:38.785 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:38.785 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:38.786 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:38.786 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:38.786 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:38.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.780+0000 7fb6755e0640 10 monclient: shutdown 2024-04-26T04:22:38.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.780+0000 7fb6755e0640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:22:38.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.780+0000 7fb6755e0640 1 -- 172.21.15.63:0/1717185506 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb6700a2cd0 msgr2=0x7fb6700a30b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:38.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.780+0000 7fb6755e0640 1 --2- 172.21.15.63:0/1717185506 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb6700a2cd0 0x7fb6700a30b0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fb6600097c0 tx=0x7fb66001c400 comp rx=0 tx=0).stop 2024-04-26T04:22:38.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.780+0000 7fb6755e0640 1 -- 172.21.15.63:0/1717185506 shutdown_connections 2024-04-26T04:22:38.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.780+0000 7fb6755e0640 1 --2- 172.21.15.63:0/1717185506 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb6700a2cd0 0x7fb6700a30b0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:38.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.780+0000 7fb6755e0640 1 -- 172.21.15.63:0/1717185506 >> 172.21.15.63:0/1717185506 conn(0x7fb6700a5f40 msgr2=0x7fb6700a6320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:22:38.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.780+0000 7fb6755e0640 1 -- 172.21.15.63:0/1717185506 shutdown_connections 2024-04-26T04:22:38.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.780+0000 7fb6755e0640 1 -- 172.21.15.63:0/1717185506 wait complete. 2024-04-26T04:22:38.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.780+0000 7fb6755e0640 10 monclient: build_initial_monmap 2024-04-26T04:22:38.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.780+0000 7fb6755e0640 10 monclient: monmap: 2024-04-26T04:22:38.791 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:22:38.791 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:22:38.791 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:22:38.791 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:22:38.791 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:22:38.791 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:38.791 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:22:38.791 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:22:38.791 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:22:38.791 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:38.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.780+0000 7fb6755e0640 1 Processor -- start 2024-04-26T04:22:38.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.780+0000 7fb6755e0640 1 -- start start 2024-04-26T04:22:38.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.780+0000 7fb6755e0640 10 monclient: init 2024-04-26T04:22:38.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.780+0000 7fb6755e0640 10 monclient: _reopen_session rank -1 2024-04-26T04:22:38.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.780+0000 7fb6755e0640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T04:22:38.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.780+0000 7fb6755e0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6700a2cd0 0x7fb6701cf2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:38.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.780+0000 7fb6755e0640 10 monclient(hunting): picked mon.noname-b con 0x7fb6700a2cd0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:22:38.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.780+0000 7fb6755e0640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fb6700a47d0 0x7fb6701cf800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:38.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.780+0000 7fb6755e0640 10 monclient(hunting): picked mon.noname-c con 0x7fb6700a47d0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:22:38.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.780+0000 7fb6755e0640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb6700a4c00 0x7fb67013ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:38.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.780+0000 7fb6755e0640 10 monclient(hunting): picked mon.noname-a con 0x7fb6700a4c00 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:22:38.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.780+0000 7fb6755e0640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:38.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.780+0000 7fb6755e0640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fb670142be0 con 0x7fb6700a4c00 2024-04-26T04:22:38.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.780+0000 7fb6755e0640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:38.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.780+0000 7fb6755e0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb670142a60 con 0x7fb6700a2cd0 2024-04-26T04:22:38.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.780+0000 7fb6755e0640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:38.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.780+0000 7fb6755e0640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7fb670142d60 con 0x7fb6700a47d0 2024-04-26T04:22:38.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.780+0000 7fb6755e0640 10 monclient(hunting): _renew_subs 2024-04-26T04:22:38.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.780+0000 7fb6755e0640 10 monclient(hunting): authenticate will time out at 1190.491973s 2024-04-26T04:22:38.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.784+0000 7fb66f7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb6700a4c00 0x7fb67013ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:38.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.784+0000 7fb66e7fc640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fb6700a47d0 0x7fb6701cf800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:38.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.784+0000 7fb66f7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb6700a4c00 0x7fb67013ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:48396/0 (socket says 172.21.15.63:48396) 2024-04-26T04:22:38.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.784+0000 7fb66f7fe640 1 -- 172.21.15.63:0/2042894007 learned_addr learned my addr 172.21.15.63:0/2042894007 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:22:38.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.784+0000 7fb66e7fc640 10 monclient(hunting): get_auth_request con 0x7fb6700a47d0 auth_method 0 2024-04-26T04:22:38.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.784+0000 7fb66e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:38.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.784+0000 7fb66e7fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:38.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.784+0000 7fb66e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:38.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.784+0000 7fb66f7fe640 10 monclient(hunting): get_auth_request con 0x7fb6700a4c00 auth_method 0 2024-04-26T04:22:38.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.784+0000 7fb66f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:38.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.784+0000 7fb66f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:38.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.784+0000 7fb66f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:38.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.784+0000 7fb66e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:22:38.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.784+0000 7fb66e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:22:38.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.784+0000 7fb66e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:22:38.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.784+0000 7fb66f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:22:38.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.784+0000 7fb66f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:22:38.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.784+0000 7fb66f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:22:38.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.784+0000 7fb66effd640 1 --2- 172.21.15.63:0/2042894007 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6700a2cd0 0x7fb6701cf2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:38.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.784+0000 7fb66effd640 10 monclient(hunting): get_auth_request con 0x7fb6700a2cd0 auth_method 0 2024-04-26T04:22:38.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.784+0000 7fb66effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:38.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.784+0000 7fb66effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:38.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.784+0000 7fb66effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:38.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.784+0000 7fb66e7fc640 10 monclient(hunting): handle_auth_done global_id 24125 payload 824 2024-04-26T04:22:38.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.784+0000 7fb66e7fc640 1 -- 172.21.15.63:0/2042894007 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6700a2cd0 msgr2=0x7fb6701cf2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:38.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.784+0000 7fb66e7fc640 1 --2- 172.21.15.63:0/2042894007 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6700a2cd0 0x7fb6701cf2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:38.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.784+0000 7fb66e7fc640 1 -- 172.21.15.63:0/2042894007 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb6700a4c00 msgr2=0x7fb67013ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:38.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.784+0000 7fb66e7fc640 1 --2- 172.21.15.63:0/2042894007 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb6700a4c00 0x7fb67013ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:38.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.784+0000 7fb66e7fc640 10 monclient: _finish_hunting 0 2024-04-26T04:22:38.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.784+0000 7fb66e7fc640 1 monclient: found mon.noname-c 2024-04-26T04:22:38.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.784+0000 7fb66e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:22:38.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.784+0000 7fb66e7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.63:3301/0 2024-04-26T04:22:38.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.784+0000 7fb66e7fc640 1 -- 172.21.15.63:0/2042894007 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb67013f340 con 0x7fb6700a47d0 2024-04-26T04:22:38.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.784+0000 7fb66f7fe640 1 --2- 172.21.15.63:0/2042894007 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb6700a4c00 0x7fb67013ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T04:22:38.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.784+0000 7fb66e7fc640 1 -- 172.21.15.63:0/2042894007 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fb6700a47d0 msgr2=0x7fb6701cf800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:22:38.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.784+0000 7fb66e7fc640 1 --2- 172.21.15.63:0/2042894007 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fb6700a47d0 0x7fb6701cf800 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fb66400c3e0 tx=0x7fb66400c8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:22:38.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.784+0000 7fb6577fe640 1 -- 172.21.15.63:0/2042894007 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6640119a0 con 0x7fb6700a47d0 2024-04-26T04:22:38.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.784+0000 7fb6577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:38.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.784+0000 7fb6577fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:22:38.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.784+0000 7fb6577fe640 10 monclient: dump: 2024-04-26T04:22:38.795 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:38.796 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:38.796 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:38.796 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:38.796 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:38.796 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:38.796 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:38.796 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:38.796 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:38.796 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:38.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.784+0000 7fb6577fe640 10 monclient: _finish_auth 0 2024-04-26T04:22:38.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.784+0000 7fb6577fe640 10 monclient: _check_auth_tickets 2024-04-26T04:22:38.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.784+0000 7fb6577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:22:38.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.784+0000 7fb6577fe640 1 -- 172.21.15.63:0/2042894007 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb664017070 con 0x7fb6700a47d0 2024-04-26T04:22:38.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.784+0000 7fb6755e0640 5 monclient: authenticate success, global_id 24125 2024-04-26T04:22:38.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.784+0000 7fb6755e0640 10 monclient: _renew_subs 2024-04-26T04:22:38.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.784+0000 7fb6755e0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:22:38.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.784+0000 7fb6755e0640 1 -- 172.21.15.63:0/2042894007 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb67013e170 con 0x7fb6700a47d0 2024-04-26T04:22:38.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.784+0000 7fb6577fe640 10 monclient: handle_config config(0 keys) 2024-04-26T04:22:38.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.784+0000 7fb6577fe640 1 -- 172.21.15.63:0/2042894007 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb664011d80 con 0x7fb6700a47d0 2024-04-26T04:22:38.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.784+0000 7fb6577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:38.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.784+0000 7fb6577fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:22:38.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.784+0000 7fb6577fe640 10 monclient: dump: 2024-04-26T04:22:38.797 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:38.797 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:38.797 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:38.797 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:38.797 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:38.797 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:38.797 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:38.797 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:38.798 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:38.798 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:38.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.784+0000 7fb66cff9640 4 set_mon_vals no callback set 2024-04-26T04:22:38.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.784+0000 7fb6755e0640 10 monclient: _renew_subs 2024-04-26T04:22:38.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.784+0000 7fb6755e0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:22:38.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.784+0000 7fb6755e0640 1 -- 172.21.15.63:0/2042894007 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb67013f060 con 0x7fb6700a47d0 2024-04-26T04:22:38.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.784+0000 7fb6755e0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:22:38.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.784+0000 7fb6755e0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:22:38.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.784+0000 7fb6755e0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:22:38.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.784+0000 7fb6755e0640 1 -- 172.21.15.63:0/2042894007 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb670062e10 con 0x7fb6700a47d0 2024-04-26T04:22:38.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.788+0000 7fb6577fe640 1 -- 172.21.15.63:0/2042894007 <== mon.2 v2:172.21.15.63:3301/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fb66401c460 con 0x7fb6700a47d0 2024-04-26T04:22:38.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.788+0000 7fb6577fe640 1 --2- 172.21.15.63:0/2042894007 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fb65c035ec0 0x7fb65c038380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:38.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.788+0000 7fb6577fe640 1 -- 172.21.15.63:0/2042894007 <== mon.2 v2:172.21.15.63:3301/0 5 ==== osd_map(212..212 src has 1..212) ==== 4346+0+0 (secure 0 0 0) 0x7fb664048ac0 con 0x7fb6700a47d0 2024-04-26T04:22:38.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.788+0000 7fb66effd640 1 --2- 172.21.15.63:0/2042894007 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fb65c035ec0 0x7fb65c038380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:38.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.788+0000 7fb66effd640 10 monclient: get_auth_request con 0x7fb65c035ec0 auth_method 0 2024-04-26T04:22:38.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.788+0000 7fb66effd640 1 --2- 172.21.15.63:0/2042894007 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fb65c035ec0 0x7fb65c038380 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fb670068310 tx=0x7fb6600a6070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:22:38.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.792+0000 7fb6577fe640 1 -- 172.21.15.63:0/2042894007 <== mon.2 v2:172.21.15.63:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fb66401a760 con 0x7fb6700a47d0 2024-04-26T04:22:38.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.792+0000 7fb6577fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:22:38.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:38.792+0000 7fb6577fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:22:39.035 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:39.032+0000 7fb6755e0640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-04-26T04:22:39.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:39.032+0000 7fb6755e0640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-04-26T04:22:39.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:39.032+0000 7fb6755e0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:22:39.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:39.032+0000 7fb6755e0640 1 -- 172.21.15.63:0/2042894007 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fb67013def0 con 0x7fb6700a47d0 2024-04-26T04:22:39.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:39.036+0000 7fb6577fe640 1 -- 172.21.15.63:0/2042894007 <== mon.2 v2:172.21.15.63:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fb664012420 con 0x7fb6700a47d0 2024-04-26T04:22:39.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:39.036+0000 7fb6577fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-04-26T04:22:39.039 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:39.036+0000 7fb6577fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:22:39.039 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.036+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:22:39.039 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.036+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=9,config=2,health=12,kv=12,logm=65} crc {auth=631872589,config=2586702516,health=441301785,kv=3642618011,logm=3420742039}) 2024-04-26T04:22:39.040 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.036+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=9,config=2,health=11,kv=12,logm=66} crc {auth=631872589,config=2586702516,health=739229086,kv=3642618011,logm=736736515}) 2024-04-26T04:22:39.040 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.036+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:22:39.040 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.036+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=9,config=2,health=12,kv=12,logm=65} crc {auth=631872589,config=2586702516,health=441301785,kv=3642618011,logm=3420742039}) 2024-04-26T04:22:39.040 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.036+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=9,config=2,health=12,kv=12,logm=65} crc {auth=631872589,config=2586702516,health=441301785,kv=3642618011,logm=1765073220}) 2024-04-26T04:22:39.042 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.036+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:22:39.042 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.036+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=812728959}) 2024-04-26T04:22:39.042 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.036+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=437689756}) 2024-04-26T04:22:39.042 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.036+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:22:39.043 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.036+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=812728959}) 2024-04-26T04:22:39.043 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.036+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4169716155}) 2024-04-26T04:22:39.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:39.036+0000 7fb6755e0640 1 -- 172.21.15.63:0/2042894007 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fb65c035ec0 msgr2=0x7fb65c038380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:39.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:39.036+0000 7fb6755e0640 1 --2- 172.21.15.63:0/2042894007 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fb65c035ec0 0x7fb65c038380 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fb670068310 tx=0x7fb6600a6070 comp rx=0 tx=0).stop 2024-04-26T04:22:39.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:39.036+0000 7fb6755e0640 10 monclient: shutdown 2024-04-26T04:22:39.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:39.036+0000 7fb6755e0640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:22:39.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:39.036+0000 7fb6755e0640 1 -- 172.21.15.63:0/2042894007 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fb6700a47d0 msgr2=0x7fb6701cf800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:39.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:39.036+0000 7fb6755e0640 1 --2- 172.21.15.63:0/2042894007 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fb6700a47d0 0x7fb6701cf800 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fb66400c3e0 tx=0x7fb66400c8a0 comp rx=0 tx=0).stop 2024-04-26T04:22:39.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:39.036+0000 7fb6755e0640 1 -- 172.21.15.63:0/2042894007 shutdown_connections 2024-04-26T04:22:39.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:39.036+0000 7fb6755e0640 1 --2- 172.21.15.63:0/2042894007 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fb65c035ec0 0x7fb65c038380 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:39.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:39.036+0000 7fb6755e0640 1 --2- 172.21.15.63:0/2042894007 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb6700a4c00 0x7fb67013ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:39.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:39.036+0000 7fb6755e0640 1 --2- 172.21.15.63:0/2042894007 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fb6700a47d0 0x7fb6701cf800 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:39.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:39.036+0000 7fb6755e0640 1 --2- 172.21.15.63:0/2042894007 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6700a2cd0 0x7fb6701cf2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:39.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:39.036+0000 7fb6755e0640 1 -- 172.21.15.63:0/2042894007 >> 172.21.15.63:0/2042894007 conn(0x7fb6700a5f40 msgr2=0x7fb6700a1a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:22:39.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:39.036+0000 7fb6755e0640 1 -- 172.21.15.63:0/2042894007 shutdown_connections 2024-04-26T04:22:39.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:39.036+0000 7fb6755e0640 1 -- 172.21.15.63:0/2042894007 wait complete. 2024-04-26T04:22:39.053 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.040+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:22:39.053 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.040+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4107357152}) 2024-04-26T04:22:39.053 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.040+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3187148426}) 2024-04-26T04:22:39.053 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.040+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:22:39.054 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.040+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4107357152}) 2024-04-26T04:22:39.054 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.040+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2849184637}) 2024-04-26T04:22:39.054 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.040+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:22:39.054 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.040+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1709281625}) 2024-04-26T04:22:39.054 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.040+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1691494753}) 2024-04-26T04:22:39.054 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.040+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:22:39.055 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.040+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1709281625}) 2024-04-26T04:22:39.055 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.040+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3548709727}) 2024-04-26T04:22:39.055 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.044+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:22:39.055 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.044+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3114171847}) 2024-04-26T04:22:39.055 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.044+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3184195415}) 2024-04-26T04:22:39.055 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.044+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:22:39.056 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.044+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3114171847}) 2024-04-26T04:22:39.056 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.044+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1487110593}) 2024-04-26T04:22:39.056 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.048+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:22:39.056 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.048+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2639797793}) 2024-04-26T04:22:39.056 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.048+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2775982299}) 2024-04-26T04:22:39.056 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.048+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:22:39.057 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.048+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2639797793}) 2024-04-26T04:22:39.057 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.048+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=201882728}) 2024-04-26T04:22:39.057 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.048+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:22:39.057 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.048+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2189804941}) 2024-04-26T04:22:39.057 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.048+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2387745620}) 2024-04-26T04:22:39.057 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.048+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:22:39.057 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.048+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2189804941}) 2024-04-26T04:22:39.057 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.048+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3893187312}) 2024-04-26T04:22:39.061 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.052+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:22:39.061 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.052+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3901466741}) 2024-04-26T04:22:39.061 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.052+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=802761203}) 2024-04-26T04:22:39.061 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.052+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:22:39.061 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.052+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3901466741}) 2024-04-26T04:22:39.061 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.052+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=928961431}) 2024-04-26T04:22:39.061 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.052+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:22:39.061 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.052+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=687309113}) 2024-04-26T04:22:39.061 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.052+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1547791357}) 2024-04-26T04:22:39.061 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.052+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:22:39.062 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.052+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=687309113}) 2024-04-26T04:22:39.062 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.052+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2935403359}) 2024-04-26T04:22:39.062 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.056+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:22:39.062 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.056+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=45,mdsmap=3,mgr=6,mgr_metadata=1,mgrstat=8,mon_config_key=16,monmap=3,osd_metadata=5,osd_pg_creating=1,osdmap=12} crc {logm=1505470956,mdsmap=3764978221,mgr=1797672415,mgr_metadata=3709105003,mgrstat=2940339237,mon_config_key=1011518028,monmap=1192322635,osd_metadata=3823772801,osd_pg_creating=4217701614,osdmap=762672625}) 2024-04-26T04:22:39.062 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.056+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=46,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=7,mon_config_key=15,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=9} crc {logm=2939220137,mdsmap=2840952586,mgr=4110609431,mgr_command_descs=3057339438,mgr_metadata=3709105003,mgrstat=265674499,mon_config_key=467594455,monmap=1192322635,osd_metadata=1397308133,osd_pg_creating=4217701614,osdmap=2230319152}) 2024-04-26T04:22:39.062 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.056+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:22:39.062 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.056+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=45,mdsmap=3,mgr=6,mgr_metadata=1,mgrstat=8,mon_config_key=16,monmap=3,osd_metadata=5,osd_pg_creating=1,osdmap=12} crc {logm=1505470956,mdsmap=3764978221,mgr=1797672415,mgr_metadata=3709105003,mgrstat=2940339237,mon_config_key=1011518028,monmap=1192322635,osd_metadata=3823772801,osd_pg_creating=4217701614,osdmap=762672625}) 2024-04-26T04:22:39.062 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.056+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=47,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=17,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=7} crc {logm=3773962884,mdsmap=1401145748,mgr=2141873744,mgr_command_descs=3057339438,mgr_metadata=3709105003,mgrstat=2940339237,mon_config_key=4044468826,monmap=1192322635,osd_metadata=1397308133,osd_pg_creating=4217701614,osdmap=708798445}) 2024-04-26T04:22:39.063 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-04-26T04:22:39.064 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.060+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:22:39.064 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.060+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3548411838}) 2024-04-26T04:22:39.064 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.060+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3914260104}) 2024-04-26T04:22:39.064 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.060+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:22:39.064 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.060+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3548411838}) 2024-04-26T04:22:39.064 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.060+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=313336573}) 2024-04-26T04:22:39.065 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.060+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:22:39.065 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.060+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1657696850}) 2024-04-26T04:22:39.065 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.060+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1272342610}) 2024-04-26T04:22:39.065 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.060+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:22:39.065 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.060+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1657696850}) 2024-04-26T04:22:39.065 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.060+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1533172176}) 2024-04-26T04:22:39.628 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.624+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:22:39.628 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.624+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2059699494}) 2024-04-26T04:22:39.628 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.624+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3134174289}) 2024-04-26T04:22:39.628 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.624+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:22:39.628 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.624+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2059699494}) 2024-04-26T04:22:39.628 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.624+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3183820452}) 2024-04-26T04:22:39.632 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.628+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:22:39.633 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.628+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=9} crc {osdmap=1709423688}) 2024-04-26T04:22:39.633 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.628+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=8} crc {osdmap=773971961}) 2024-04-26T04:22:39.633 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.628+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:22:39.633 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.628+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=9} crc {osdmap=1709423688}) 2024-04-26T04:22:39.633 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:22:39.628+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=9} crc {osdmap=1709423687}) 2024-04-26T04:22:40.064 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-26T04:22:40.064 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T04:22:40.146 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[30501]: starting. 2024-04-26T04:22:40.146 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[30501]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-30500 2024-04-26T04:22:40.146 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[30501]: created object 0... 2024-04-26T04:22:40.146 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[30501]: created object 25... 2024-04-26T04:22:40.147 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[30501]: created object 49... 2024-04-26T04:22:40.147 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[30501]: finishing. 2024-04-26T04:22:40.147 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[30501]: shutting down. 2024-04-26T04:22:40.199 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.196+0000 7f1641354640 10 monclient: get_monmap_and_config 2024-04-26T04:22:40.199 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.196+0000 7f1641354640 10 monclient: build_initial_monmap 2024-04-26T04:22:40.199 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.196+0000 7f1641354640 10 monclient: monmap: 2024-04-26T04:22:40.199 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:22:40.199 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:40.199 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:22:40.199341+0000 2024-04-26T04:22:40.199 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:22:40.199341+0000 2024-04-26T04:22:40.200 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:22:40.200 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:40.200 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:22:40.200 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:22:40.200 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:22:40.200 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:22:40.200 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:22:40.200 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:40.203 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.196+0000 7f1641354640 1 Processor -- start 2024-04-26T04:22:40.203 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.196+0000 7f1641354640 1 -- start start 2024-04-26T04:22:40.203 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.196+0000 7f1641354640 10 monclient: init 2024-04-26T04:22:40.203 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.196+0000 7f1641354640 10 monclient: _reopen_session rank -1 2024-04-26T04:22:40.203 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.196+0000 7f1641354640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-04-26T04:22:40.203 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.196+0000 7f1641354640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f163c09fe60 addr v1:172.21.15.63:6789/0 2024-04-26T04:22:40.204 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.196+0000 7f1641354640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f163c0a0240 0x7f163c0a2a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:40.204 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.196+0000 7f1641354640 10 monclient(hunting): picked mon.noname-a con 0x7f163c0a0240 addr v2:172.21.15.63:3300/0 2024-04-26T04:22:40.204 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.196+0000 7f1641354640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f163c0a2f70 0x7f163c13e110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:40.204 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.196+0000 7f1641354640 10 monclient(hunting): picked mon.noname-b con 0x7f163c0a2f70 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:22:40.204 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.196+0000 7f1641354640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f163c0a0c60 con 0x7f163c09fe60 2024-04-26T04:22:40.204 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.196+0000 7f1641354640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:40.204 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.196+0000 7f1641354640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7f163c0a3730 con 0x7f163c0a0240 2024-04-26T04:22:40.204 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.196+0000 7f1641354640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:40.204 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.196+0000 7f1641354640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f163c0a0f40 con 0x7f163c0a2f70 2024-04-26T04:22:40.204 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.196+0000 7f1641354640 10 monclient(hunting): _renew_subs 2024-04-26T04:22:40.205 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.196+0000 7f1641354640 10 monclient(hunting): authenticate will time out at 1191.907348s 2024-04-26T04:22:40.205 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.196+0000 7f163b7fe640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f163c0a2f70 0x7f163c13e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:40.205 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.196+0000 7f163a7fc640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f163c0a0240 0x7f163c0a2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:40.205 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.196+0000 7f163a7fc640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f163c0a0240 0x7f163c0a2a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:46832/0 (socket says 172.21.15.63:46832) 2024-04-26T04:22:40.205 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.196+0000 7f163a7fc640 1 -- 172.21.15.63:0/2127881202 learned_addr learned my addr 172.21.15.63:0/2127881202 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:22:40.205 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.196+0000 7f163a7fc640 10 monclient(hunting): get_auth_request con 0x7f163c0a0240 auth_method 0 2024-04-26T04:22:40.205 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.196+0000 7f163b7fe640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f163c0a2f70 0x7f163c13e110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:60186/0 (socket says 172.21.15.63:60186) 2024-04-26T04:22:40.205 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.196+0000 7f163a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:40.205 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.196+0000 7f163a7fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:40.205 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.196+0000 7f163a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:40.205 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.196+0000 7f163affd640 1 --1- 172.21.15.63:0/2127881202 >> v1:172.21.15.63:6789/0 conn(0x7f163c09fe60 0x7f163c1372f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:35672/0 (socket says 172.21.15.63:35672) 2024-04-26T04:22:40.205 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.196+0000 7f163b7fe640 10 monclient(hunting): get_auth_request con 0x7f163c0a2f70 auth_method 0 2024-04-26T04:22:40.205 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.196+0000 7f163b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:40.206 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.196+0000 7f163b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:40.206 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.196+0000 7f163b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:40.206 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.196+0000 7f163b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:22:40.206 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.196+0000 7f163b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:22:40.206 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.196+0000 7f163b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:22:40.206 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.196+0000 7f163a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:22:40.206 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.196+0000 7f163a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:22:40.206 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.196+0000 7f163a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:22:40.207 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.196+0000 7f163a7fc640 10 monclient(hunting): handle_auth_done global_id 5223 payload 274 2024-04-26T04:22:40.207 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.196+0000 7f163a7fc640 1 -- 172.21.15.63:0/2127881202 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f163c0a2f70 msgr2=0x7f163c13e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:22:40.207 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.196+0000 7f163a7fc640 1 --2- 172.21.15.63:0/2127881202 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f163c0a2f70 0x7f163c13e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:40.207 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.196+0000 7f163a7fc640 1 -- 172.21.15.63:0/2127881202 >> v1:172.21.15.63:6789/0 conn(0x7f163c09fe60 legacy=0x7f163c1372f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:22:40.209 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.196+0000 7f163a7fc640 10 monclient: _finish_hunting 0 2024-04-26T04:22:40.209 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.196+0000 7f163a7fc640 1 monclient: found mon.noname-a 2024-04-26T04:22:40.209 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.196+0000 7f163a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:22:40.209 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.196+0000 7f163a7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:22:40.209 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.196+0000 7f163a7fc640 1 -- 172.21.15.63:0/2127881202 --> v2:172.21.15.63:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f163c0a1220 con 0x7f163c0a0240 2024-04-26T04:22:40.209 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.196+0000 7f163b7fe640 1 --2- 172.21.15.63:0/2127881202 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f163c0a2f70 0x7f163c13e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T04:22:40.216 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f163a7fc640 1 -- 172.21.15.63:0/2127881202 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f163c0a0240 msgr2=0x7f163c0a2a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:22:40.216 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f163a7fc640 1 --2- 172.21.15.63:0/2127881202 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f163c0a0240 0x7f163c0a2a30 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f1630009870 tx=0x7f163001c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6a09d049aca0d08 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:22:40.216 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f1639ffb640 1 -- 172.21.15.63:0/2127881202 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16300b1070 con 0x7f163c0a0240 2024-04-26T04:22:40.216 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f1639ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:40.216 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f1639ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:22:40.216 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f1639ffb640 10 monclient: dump: 2024-04-26T04:22:40.216 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:40.216 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:40.216 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:40.216 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:40.217 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:40.217 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:40.217 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:40.217 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:40.217 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:40.217 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:40.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f1639ffb640 10 monclient: _finish_auth 0 2024-04-26T04:22:40.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f1639ffb640 10 monclient: _check_auth_tickets 2024-04-26T04:22:40.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f1639ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:22:40.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f1639ffb640 1 -- 172.21.15.63:0/2127881202 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16300ac620 con 0x7f163c0a0240 2024-04-26T04:22:40.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f1641354640 5 monclient: authenticate success, global_id 5223 2024-04-26T04:22:40.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f1639ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T04:22:40.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f1639ffb640 1 -- 172.21.15.63:0/2127881202 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16300ac9f0 con 0x7f163c0a0240 2024-04-26T04:22:40.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f1639ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:40.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f1639ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:22:40.218 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f1639ffb640 10 monclient: dump: 2024-04-26T04:22:40.218 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:40.218 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:40.218 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:40.218 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:40.218 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:40.218 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:40.218 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:40.218 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:40.218 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:40.218 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:40.218 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f1641354640 10 monclient: get_monmap_and_config success 2024-04-26T04:22:40.218 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f1641354640 4 set_mon_vals no callback set 2024-04-26T04:22:40.218 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f1641354640 10 monclient: shutdown 2024-04-26T04:22:40.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f1641354640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:22:40.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f1641354640 1 -- 172.21.15.63:0/2127881202 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f163c0a0240 msgr2=0x7f163c0a2a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:40.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f1641354640 1 --2- 172.21.15.63:0/2127881202 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f163c0a0240 0x7f163c0a2a30 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f1630009870 tx=0x7f163001c3e0 comp rx=0 tx=0).stop 2024-04-26T04:22:40.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f1641354640 1 -- 172.21.15.63:0/2127881202 shutdown_connections 2024-04-26T04:22:40.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f1641354640 1 --2- 172.21.15.63:0/2127881202 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f163c0a2f70 0x7f163c13e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:40.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f1641354640 1 --2- 172.21.15.63:0/2127881202 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f163c0a0240 0x7f163c0a2a30 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:40.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f1641354640 1 -- 172.21.15.63:0/2127881202 >> 172.21.15.63:0/2127881202 conn(0x7f163c0a5fb0 msgr2=0x7f163c0a63c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:22:40.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f1641354640 1 -- 172.21.15.63:0/2127881202 shutdown_connections 2024-04-26T04:22:40.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f1641354640 1 -- 172.21.15.63:0/2127881202 wait complete. 2024-04-26T04:22:40.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f1641354640 10 monclient: build_initial_monmap 2024-04-26T04:22:40.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f1641354640 10 monclient: monmap: 2024-04-26T04:22:40.219 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:22:40.219 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:22:40.219 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:22:40.219 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:22:40.220 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:22:40.220 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:40.220 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:22:40.220 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:22:40.220 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:22:40.220 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:40.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f1641354640 1 Processor -- start 2024-04-26T04:22:40.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f1641354640 1 -- start start 2024-04-26T04:22:40.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f1641354640 10 monclient: init 2024-04-26T04:22:40.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f1641354640 10 monclient: _reopen_session rank -1 2024-04-26T04:22:40.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f1641354640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T04:22:40.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f1641354640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f163c09fe60 0x7f163c1ce840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:40.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f1641354640 10 monclient(hunting): picked mon.noname-b con 0x7f163c09fe60 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:22:40.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f1641354640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f163c0a0240 0x7f163c13ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:40.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f1641354640 10 monclient(hunting): picked mon.noname-a con 0x7f163c0a0240 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:22:40.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f1641354640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f163c0a2f70 0x7f163c13d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:40.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f1641354640 10 monclient(hunting): picked mon.noname-c con 0x7f163c0a2f70 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:22:40.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f1641354640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:40.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f1641354640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f163c141200 con 0x7f163c0a0240 2024-04-26T04:22:40.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f1641354640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:40.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f1641354640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f163c141080 con 0x7f163c09fe60 2024-04-26T04:22:40.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f1641354640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:40.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f1641354640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f163c141380 con 0x7f163c0a2f70 2024-04-26T04:22:40.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f1641354640 10 monclient(hunting): _renew_subs 2024-04-26T04:22:40.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f1641354640 10 monclient(hunting): authenticate will time out at 1191.914404s 2024-04-26T04:22:40.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f163a7fc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f163c0a0240 0x7f163c13ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:40.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f163b7fe640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f163c0a2f70 0x7f163c13d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:40.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f163a7fc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f163c0a0240 0x7f163c13ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:46848/0 (socket says 172.21.15.63:46848) 2024-04-26T04:22:40.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f163b7fe640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f163c0a2f70 0x7f163c13d350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:60192/0 (socket says 172.21.15.63:60192) 2024-04-26T04:22:40.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f163a7fc640 1 -- 172.21.15.63:0/1399064285 learned_addr learned my addr 172.21.15.63:0/1399064285 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:22:40.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f163a7fc640 10 monclient(hunting): get_auth_request con 0x7f163c0a0240 auth_method 0 2024-04-26T04:22:40.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f163a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:40.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f163a7fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:40.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f163a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:40.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f163b7fe640 10 monclient(hunting): get_auth_request con 0x7f163c0a2f70 auth_method 0 2024-04-26T04:22:40.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f163b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:40.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f163b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:40.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f163b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:40.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f163a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:22:40.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f163a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:22:40.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f163a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:22:40.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f163b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:22:40.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f163b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:22:40.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f163b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:22:40.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f163affd640 1 --2- 172.21.15.63:0/1399064285 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f163c09fe60 0x7f163c1ce840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:40.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f163a7fc640 10 monclient(hunting): handle_auth_done global_id 5226 payload 824 2024-04-26T04:22:40.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f163a7fc640 1 -- 172.21.15.63:0/1399064285 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f163c0a2f70 msgr2=0x7f163c13d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:40.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f163a7fc640 1 --2- 172.21.15.63:0/1399064285 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f163c0a2f70 0x7f163c13d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:40.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f163a7fc640 1 -- 172.21.15.63:0/1399064285 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f163c09fe60 msgr2=0x7f163c1ce840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:40.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f163a7fc640 1 --2- 172.21.15.63:0/1399064285 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f163c09fe60 0x7f163c1ce840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:40.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f163a7fc640 10 monclient: _finish_hunting 0 2024-04-26T04:22:40.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f163a7fc640 1 monclient: found mon.noname-a 2024-04-26T04:22:40.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f163a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:22:40.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f163a7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:22:40.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f163a7fc640 1 -- 172.21.15.63:0/1399064285 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f163c0a44c0 con 0x7f163c0a0240 2024-04-26T04:22:40.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f163affd640 10 monclient: get_auth_request con 0x7f163c09fe60 auth_method 0 2024-04-26T04:22:40.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.204+0000 7f163affd640 1 --2- 172.21.15.63:0/1399064285 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f163c09fe60 0x7f163c1ce840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T04:22:40.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.208+0000 7f163a7fc640 1 -- 172.21.15.63:0/1399064285 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f163c0a0240 msgr2=0x7f163c13ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:22:40.224 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.208+0000 7f163a7fc640 1 --2- 172.21.15.63:0/1399064285 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f163c0a0240 0x7f163c13ce10 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f163001cc80 tx=0x7f163001ac20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:22:40.224 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.208+0000 7f16237fe640 1 -- 172.21.15.63:0/1399064285 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16300b1040 con 0x7f163c0a0240 2024-04-26T04:22:40.224 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.208+0000 7f16237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:40.224 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.208+0000 7f16237fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:22:40.224 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.208+0000 7f16237fe640 10 monclient: dump: 2024-04-26T04:22:40.224 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:40.224 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:40.224 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:40.224 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:40.224 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:40.224 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:40.224 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:40.224 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:40.224 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:40.224 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:40.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.208+0000 7f16237fe640 10 monclient: _finish_auth 0 2024-04-26T04:22:40.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.208+0000 7f16237fe640 10 monclient: _check_auth_tickets 2024-04-26T04:22:40.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.208+0000 7f16237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:22:40.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.208+0000 7f16237fe640 1 -- 172.21.15.63:0/1399064285 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16300bc070 con 0x7f163c0a0240 2024-04-26T04:22:40.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.208+0000 7f1641354640 5 monclient: authenticate success, global_id 5226 2024-04-26T04:22:40.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.208+0000 7f1641354640 10 monclient: _renew_subs 2024-04-26T04:22:40.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.208+0000 7f1641354640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:22:40.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.208+0000 7f1641354640 1 -- 172.21.15.63:0/1399064285 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f163c13b8e0 con 0x7f163c0a0240 2024-04-26T04:22:40.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.208+0000 7f16237fe640 10 monclient: handle_config config(0 keys) 2024-04-26T04:22:40.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.208+0000 7f16237fe640 1 -- 172.21.15.63:0/1399064285 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16300aba30 con 0x7f163c0a0240 2024-04-26T04:22:40.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.208+0000 7f16237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:40.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.208+0000 7f16237fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:22:40.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.208+0000 7f16237fe640 10 monclient: dump: 2024-04-26T04:22:40.225 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:40.225 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:40.225 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:40.226 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:40.226 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:40.226 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:40.226 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:40.226 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:40.226 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:40.226 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:40.226 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.208+0000 7f16397fa640 4 set_mon_vals no callback set 2024-04-26T04:22:40.226 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.208+0000 7f1641354640 10 monclient: _renew_subs 2024-04-26T04:22:40.226 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.208+0000 7f1641354640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:22:40.226 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.208+0000 7f1641354640 1 -- 172.21.15.63:0/1399064285 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f163c13d890 con 0x7f163c0a0240 2024-04-26T04:22:40.226 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.208+0000 7f16237fe640 1 -- 172.21.15.63:0/1399064285 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f16300abbd0 con 0x7f163c0a0240 2024-04-26T04:22:40.226 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.208+0000 7f1641354640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:22:40.226 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.208+0000 7f1641354640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:22:40.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.208+0000 7f1641354640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:22:40.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.208+0000 7f1641354640 1 -- 172.21.15.63:0/1399064285 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f163c062e10 con 0x7f163c0a0240 2024-04-26T04:22:40.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.208+0000 7f16237fe640 1 --2- 172.21.15.63:0/1399064285 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f1628035ec0 0x7f1628038380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:40.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.208+0000 7f16237fe640 1 -- 172.21.15.63:0/1399064285 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(214..214 src has 1..214) ==== 4738+0+0 (secure 0 0 0) 0x7f16300e32e0 con 0x7f163c0a0240 2024-04-26T04:22:40.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.208+0000 7f163affd640 1 --2- 172.21.15.63:0/1399064285 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f1628035ec0 0x7f1628038380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:40.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.208+0000 7f163affd640 10 monclient: get_auth_request con 0x7f1628035ec0 auth_method 0 2024-04-26T04:22:40.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.212+0000 7f163affd640 1 --2- 172.21.15.63:0/1399064285 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f1628035ec0 0x7f1628038380 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f162c005710 tx=0x7f162c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:22:40.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.216+0000 7f16237fe640 1 -- 172.21.15.63:0/1399064285 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f16300160f0 con 0x7f163c0a0240 2024-04-26T04:22:40.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.216+0000 7f16237fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:22:40.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.216+0000 7f16237fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:22:40.438 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.436+0000 7f1641354640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T04:22:40.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.436+0000 7f1641354640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T04:22:40.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.436+0000 7f1641354640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:22:40.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.436+0000 7f1641354640 1 -- 172.21.15.63:0/1399064285 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f163c068430 con 0x7f163c0a0240 2024-04-26T04:22:40.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.436+0000 7f16237fe640 1 -- 172.21.15.63:0/1399064285 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f16300b63b0 con 0x7f163c0a0240 2024-04-26T04:22:40.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.436+0000 7f16237fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T04:22:40.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.436+0000 7f16237fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:22:40.440 INFO:teuthology.orchestra.run.smithi063.stdout:{"election_epoch":20,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cd1c6a0c-4e4b-4014-89c0-2e535479ef39","modified":"2024-04-26T04:18:03.702339Z","created":"2024-04-26T04:18:03.702339Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3300","nonce":0},{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:3300","nonce":0},{"type":"v1","addr":"172.21.15.186:6789","nonce":0}]},"addr":"172.21.15.186:6789/0","public_addr":"172.21.15.186:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3301","nonce":0},{"type":"v1","addr":"172.21.15.63:6790","nonce":0}]},"addr":"172.21.15.63:6790/0","public_addr":"172.21.15.63:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T04:22:40.441 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.436+0000 7f1641354640 1 -- 172.21.15.63:0/1399064285 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f1628035ec0 msgr2=0x7f1628038380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:40.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.436+0000 7f1641354640 1 --2- 172.21.15.63:0/1399064285 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f1628035ec0 0x7f1628038380 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f162c005710 tx=0x7f162c009040 comp rx=0 tx=0).stop 2024-04-26T04:22:40.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.436+0000 7f1641354640 10 monclient: shutdown 2024-04-26T04:22:40.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.436+0000 7f1641354640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:22:40.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.436+0000 7f1641354640 1 -- 172.21.15.63:0/1399064285 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f163c0a0240 msgr2=0x7f163c13ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:40.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.436+0000 7f1641354640 1 --2- 172.21.15.63:0/1399064285 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f163c0a0240 0x7f163c13ce10 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f163001cc80 tx=0x7f163001ac20 comp rx=0 tx=0).stop 2024-04-26T04:22:40.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.436+0000 7f1641354640 1 -- 172.21.15.63:0/1399064285 shutdown_connections 2024-04-26T04:22:40.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.436+0000 7f1641354640 1 --2- 172.21.15.63:0/1399064285 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f1628035ec0 0x7f1628038380 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:40.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.436+0000 7f1641354640 1 --2- 172.21.15.63:0/1399064285 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f163c0a2f70 0x7f163c13d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:40.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.436+0000 7f1641354640 1 --2- 172.21.15.63:0/1399064285 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f163c0a0240 0x7f163c13ce10 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:40.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.436+0000 7f1641354640 1 --2- 172.21.15.63:0/1399064285 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f163c09fe60 0x7f163c1ce840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:40.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.436+0000 7f1641354640 1 -- 172.21.15.63:0/1399064285 >> 172.21.15.63:0/1399064285 conn(0x7f163c0a5fb0 msgr2=0x7f163c0a19e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:22:40.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.436+0000 7f1641354640 1 -- 172.21.15.63:0/1399064285 shutdown_connections 2024-04-26T04:22:40.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.436+0000 7f1641354640 1 -- 172.21.15.63:0/1399064285 wait complete. 2024-04-26T04:22:40.458 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-26T04:22:40.458 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-04-26T04:22:40.458 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-26T04:22:40.459 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T04:22:40.586 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.584+0000 7f8c892d0640 10 monclient: get_monmap_and_config 2024-04-26T04:22:40.586 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.584+0000 7f8c892d0640 10 monclient: build_initial_monmap 2024-04-26T04:22:40.586 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.584+0000 7f8c892d0640 10 monclient: monmap: 2024-04-26T04:22:40.586 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:22:40.587 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:40.587 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:22:40.586426+0000 2024-04-26T04:22:40.587 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:22:40.586426+0000 2024-04-26T04:22:40.587 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:22:40.587 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:40.587 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:22:40.587 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:22:40.587 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:22:40.587 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:22:40.587 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:22:40.587 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:40.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.584+0000 7f8c892d0640 1 Processor -- start 2024-04-26T04:22:40.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.584+0000 7f8c892d0640 1 -- start start 2024-04-26T04:22:40.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.584+0000 7f8c892d0640 10 monclient: init 2024-04-26T04:22:40.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.584+0000 7f8c892d0640 10 monclient: _reopen_session rank -1 2024-04-26T04:22:40.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.584+0000 7f8c892d0640 10 monclient: _add_conns ranks=[0,2,4,1,3] 2024-04-26T04:22:40.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.584+0000 7f8c892d0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8c8409fe60 addr v1:172.21.15.63:6789/0 2024-04-26T04:22:40.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.584+0000 7f8c892d0640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f8c840a0240 0x7f8c840a2a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:40.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.584+0000 7f8c892d0640 10 monclient(hunting): picked mon.noname-b con 0x7f8c840a0240 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:22:40.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.584+0000 7f8c892d0640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f8c840a2f70 0x7f8c8413e110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:40.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.584+0000 7f8c892d0640 10 monclient(hunting): picked mon.noname-c con 0x7f8c840a2f70 addr v2:172.21.15.186:3300/0 2024-04-26T04:22:40.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.584+0000 7f8c892d0640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c840a0c60 con 0x7f8c8409fe60 2024-04-26T04:22:40.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.584+0000 7f8c892d0640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:40.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.584+0000 7f8c892d0640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f8c840a3730 con 0x7f8c840a2f70 2024-04-26T04:22:40.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.584+0000 7f8c892d0640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:40.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.584+0000 7f8c892d0640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f8c840a0f40 con 0x7f8c840a0240 2024-04-26T04:22:40.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.584+0000 7f8c892d0640 10 monclient(hunting): _renew_subs 2024-04-26T04:22:40.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.584+0000 7f8c892d0640 10 monclient(hunting): authenticate will time out at 1192.294090s 2024-04-26T04:22:40.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.584+0000 7f8c827fc640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f8c840a0240 0x7f8c840a2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:40.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.584+0000 7f8c827fc640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f8c840a0240 0x7f8c840a2a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:60198/0 (socket says 172.21.15.63:60198) 2024-04-26T04:22:40.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.584+0000 7f8c827fc640 1 -- 172.21.15.63:0/2748063728 learned_addr learned my addr 172.21.15.63:0/2748063728 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:22:40.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.584+0000 7f8c827fc640 10 monclient(hunting): get_auth_request con 0x7f8c840a0240 auth_method 0 2024-04-26T04:22:40.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.584+0000 7f8c827fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:40.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.584+0000 7f8c827fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:40.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.584+0000 7f8c827fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:40.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.584+0000 7f8c827fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:22:40.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.584+0000 7f8c827fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:22:40.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.584+0000 7f8c827fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:22:40.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.584+0000 7f8c837fe640 1 --2- 172.21.15.63:0/2748063728 >> v2:172.21.15.186:3300/0 conn(0x7f8c840a2f70 0x7f8c8413e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:40.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.584+0000 7f8c837fe640 10 monclient(hunting): get_auth_request con 0x7f8c840a2f70 auth_method 0 2024-04-26T04:22:40.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.584+0000 7f8c837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:40.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.584+0000 7f8c837fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:40.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.584+0000 7f8c837fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:40.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.584+0000 7f8c827fc640 10 monclient(hunting): handle_auth_done global_id 24134 payload 274 2024-04-26T04:22:40.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.584+0000 7f8c827fc640 1 -- 172.21.15.63:0/2748063728 >> v2:172.21.15.186:3300/0 conn(0x7f8c840a2f70 msgr2=0x7f8c8413e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:22:40.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.584+0000 7f8c827fc640 1 --2- 172.21.15.63:0/2748063728 >> v2:172.21.15.186:3300/0 conn(0x7f8c840a2f70 0x7f8c8413e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:40.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.584+0000 7f8c827fc640 1 -- 172.21.15.63:0/2748063728 >> v1:172.21.15.63:6789/0 conn(0x7f8c8409fe60 legacy=0x7f8c841372f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:22:40.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.584+0000 7f8c827fc640 10 monclient: _finish_hunting 0 2024-04-26T04:22:40.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.584+0000 7f8c827fc640 1 monclient: found mon.noname-b 2024-04-26T04:22:40.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.584+0000 7f8c827fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:22:40.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.584+0000 7f8c827fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.63:3301/0 2024-04-26T04:22:40.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.584+0000 7f8c827fc640 1 -- 172.21.15.63:0/2748063728 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c840a1220 con 0x7f8c840a0240 2024-04-26T04:22:40.602 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.588+0000 7f8c827fc640 1 -- 172.21.15.63:0/2748063728 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f8c840a0240 msgr2=0x7f8c840a2a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:22:40.602 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.588+0000 7f8c827fc640 1 --2- 172.21.15.63:0/2748063728 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f8c840a0240 0x7f8c840a2a30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8c78009870 tx=0x7f8c7801c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=245a7528cc1a3fcb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:22:40.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.588+0000 7f8c81ffb640 1 -- 172.21.15.63:0/2748063728 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c780b1070 con 0x7f8c840a0240 2024-04-26T04:22:40.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.588+0000 7f8c81ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:40.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.588+0000 7f8c81ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T04:22:40.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.588+0000 7f8c81ffb640 10 monclient: dump: 2024-04-26T04:22:40.603 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:40.603 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:40.603 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:40.603 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:40.603 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:40.603 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:40.603 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:40.603 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:40.603 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:40.604 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:40.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.588+0000 7f8c81ffb640 10 monclient: _finish_auth 0 2024-04-26T04:22:40.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.588+0000 7f8c81ffb640 10 monclient: _check_auth_tickets 2024-04-26T04:22:40.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.588+0000 7f8c81ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:22:40.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.588+0000 7f8c81ffb640 1 -- 172.21.15.63:0/2748063728 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c780ac620 con 0x7f8c840a0240 2024-04-26T04:22:40.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.588+0000 7f8c81ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T04:22:40.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.588+0000 7f8c81ffb640 1 -- 172.21.15.63:0/2748063728 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c780ac9f0 con 0x7f8c840a0240 2024-04-26T04:22:40.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.588+0000 7f8c892d0640 5 monclient: authenticate success, global_id 24134 2024-04-26T04:22:40.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.588+0000 7f8c892d0640 10 monclient: get_monmap_and_config success 2024-04-26T04:22:40.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.588+0000 7f8c892d0640 4 set_mon_vals no callback set 2024-04-26T04:22:40.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.588+0000 7f8c81ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:40.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.588+0000 7f8c81ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:22:40.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.588+0000 7f8c81ffb640 10 monclient: dump: 2024-04-26T04:22:40.604 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:40.605 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:40.605 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:40.605 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:40.605 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:40.605 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:40.605 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:40.605 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:40.605 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:40.605 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:40.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.588+0000 7f8c892d0640 10 monclient: shutdown 2024-04-26T04:22:40.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.588+0000 7f8c892d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:22:40.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.588+0000 7f8c892d0640 1 -- 172.21.15.63:0/2748063728 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f8c840a0240 msgr2=0x7f8c840a2a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:40.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.588+0000 7f8c892d0640 1 --2- 172.21.15.63:0/2748063728 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f8c840a0240 0x7f8c840a2a30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8c78009870 tx=0x7f8c7801c3e0 comp rx=0 tx=0).stop 2024-04-26T04:22:40.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.588+0000 7f8c892d0640 1 -- 172.21.15.63:0/2748063728 shutdown_connections 2024-04-26T04:22:40.605 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.588+0000 7f8c892d0640 1 --2- 172.21.15.63:0/2748063728 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f8c840a0240 0x7f8c840a2a30 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:40.606 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.588+0000 7f8c892d0640 1 --2- 172.21.15.63:0/2748063728 >> v2:172.21.15.186:3300/0 conn(0x7f8c840a2f70 0x7f8c8413e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:40.606 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.588+0000 7f8c892d0640 1 -- 172.21.15.63:0/2748063728 >> 172.21.15.63:0/2748063728 conn(0x7f8c840a5fb0 msgr2=0x7f8c840a63c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:22:40.606 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.588+0000 7f8c892d0640 1 -- 172.21.15.63:0/2748063728 shutdown_connections 2024-04-26T04:22:40.606 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.588+0000 7f8c892d0640 1 -- 172.21.15.63:0/2748063728 wait complete. 2024-04-26T04:22:40.606 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.592+0000 7f8c892d0640 10 monclient: build_initial_monmap 2024-04-26T04:22:40.606 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.592+0000 7f8c892d0640 10 monclient: monmap: 2024-04-26T04:22:40.606 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:22:40.606 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:22:40.606 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:22:40.606 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:22:40.606 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:22:40.606 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:40.606 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:22:40.606 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:22:40.606 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:22:40.607 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:40.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.592+0000 7f8c892d0640 1 Processor -- start 2024-04-26T04:22:40.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.592+0000 7f8c892d0640 1 -- start start 2024-04-26T04:22:40.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.592+0000 7f8c892d0640 10 monclient: init 2024-04-26T04:22:40.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.592+0000 7f8c892d0640 10 monclient: _reopen_session rank -1 2024-04-26T04:22:40.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.592+0000 7f8c892d0640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T04:22:40.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.592+0000 7f8c892d0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c8409fe60 0x7f8c84132e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:40.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.592+0000 7f8c892d0640 10 monclient(hunting): picked mon.noname-b con 0x7f8c8409fe60 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:22:40.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.592+0000 7f8c892d0640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8c840a0240 0x7f8c841333d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:40.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.592+0000 7f8c892d0640 10 monclient(hunting): picked mon.noname-a con 0x7f8c840a0240 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:22:40.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.592+0000 7f8c892d0640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f8c840a2f70 0x7f8c84137df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:40.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.592+0000 7f8c892d0640 10 monclient(hunting): picked mon.noname-c con 0x7f8c840a2f70 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:22:40.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.592+0000 7f8c892d0640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:40.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.592+0000 7f8c892d0640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f8c84141200 con 0x7f8c840a0240 2024-04-26T04:22:40.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.592+0000 7f8c892d0640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:40.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.592+0000 7f8c892d0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8c84141080 con 0x7f8c8409fe60 2024-04-26T04:22:40.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.592+0000 7f8c892d0640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:40.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.592+0000 7f8c892d0640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f8c84141380 con 0x7f8c840a2f70 2024-04-26T04:22:40.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.592+0000 7f8c892d0640 10 monclient(hunting): _renew_subs 2024-04-26T04:22:40.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.592+0000 7f8c892d0640 10 monclient(hunting): authenticate will time out at 1192.301433s 2024-04-26T04:22:40.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.592+0000 7f8c837fe640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f8c840a2f70 0x7f8c84137df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:40.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.592+0000 7f8c827fc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8c840a0240 0x7f8c841333d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:40.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.592+0000 7f8c837fe640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f8c840a2f70 0x7f8c84137df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:60204/0 (socket says 172.21.15.63:60204) 2024-04-26T04:22:40.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.592+0000 7f8c837fe640 1 -- 172.21.15.63:0/588076214 learned_addr learned my addr 172.21.15.63:0/588076214 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:22:40.609 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.592+0000 7f8c837fe640 10 monclient(hunting): get_auth_request con 0x7f8c840a2f70 auth_method 0 2024-04-26T04:22:40.609 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.592+0000 7f8c837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:40.609 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.592+0000 7f8c837fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:40.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.592+0000 7f8c837fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:40.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.592+0000 7f8c827fc640 10 monclient(hunting): get_auth_request con 0x7f8c840a0240 auth_method 0 2024-04-26T04:22:40.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.592+0000 7f8c827fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:40.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.592+0000 7f8c827fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:40.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.592+0000 7f8c827fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:40.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.592+0000 7f8c837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:22:40.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.592+0000 7f8c837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:22:40.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.592+0000 7f8c837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:22:40.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.592+0000 7f8c827fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:22:40.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.592+0000 7f8c827fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:22:40.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.592+0000 7f8c827fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:22:40.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.592+0000 7f8c82ffd640 1 --2- 172.21.15.63:0/588076214 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c8409fe60 0x7f8c84132e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:40.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.592+0000 7f8c837fe640 10 monclient(hunting): handle_auth_done global_id 24137 payload 824 2024-04-26T04:22:40.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.592+0000 7f8c837fe640 1 -- 172.21.15.63:0/588076214 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c8409fe60 msgr2=0x7f8c84132e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:40.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.592+0000 7f8c837fe640 1 --2- 172.21.15.63:0/588076214 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c8409fe60 0x7f8c84132e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:40.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.592+0000 7f8c837fe640 1 -- 172.21.15.63:0/588076214 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8c840a0240 msgr2=0x7f8c841333d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:40.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.592+0000 7f8c837fe640 1 --2- 172.21.15.63:0/588076214 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8c840a0240 0x7f8c841333d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:40.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.592+0000 7f8c837fe640 10 monclient: _finish_hunting 0 2024-04-26T04:22:40.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.592+0000 7f8c837fe640 1 monclient: found mon.noname-c 2024-04-26T04:22:40.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.592+0000 7f8c837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:22:40.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.592+0000 7f8c837fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.63:3301/0 2024-04-26T04:22:40.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.592+0000 7f8c837fe640 1 -- 172.21.15.63:0/588076214 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c8413b970 con 0x7f8c840a2f70 2024-04-26T04:22:40.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.596+0000 7f8c837fe640 1 -- 172.21.15.63:0/588076214 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f8c840a2f70 msgr2=0x7f8c84137df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:22:40.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.596+0000 7f8c837fe640 1 --2- 172.21.15.63:0/588076214 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f8c840a2f70 0x7f8c84137df0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c007e40 tx=0x7f8c7c00e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:22:40.612 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.596+0000 7f8c6b7fe640 1 -- 172.21.15.63:0/588076214 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c7c013790 con 0x7f8c840a2f70 2024-04-26T04:22:40.612 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.596+0000 7f8c6b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:40.612 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.596+0000 7f8c6b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:22:40.612 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.596+0000 7f8c6b7fe640 10 monclient: dump: 2024-04-26T04:22:40.612 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:40.612 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:40.612 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:40.612 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:40.612 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:40.612 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:40.612 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:40.612 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:40.612 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:40.612 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:40.612 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.596+0000 7f8c6b7fe640 10 monclient: _finish_auth 0 2024-04-26T04:22:40.613 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.596+0000 7f8c6b7fe640 10 monclient: _check_auth_tickets 2024-04-26T04:22:40.613 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.596+0000 7f8c6b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:22:40.613 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.596+0000 7f8c892d0640 5 monclient: authenticate success, global_id 24137 2024-04-26T04:22:40.613 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.596+0000 7f8c892d0640 10 monclient: _renew_subs 2024-04-26T04:22:40.613 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.596+0000 7f8c6b7fe640 1 -- 172.21.15.63:0/588076214 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c7c019070 con 0x7f8c840a2f70 2024-04-26T04:22:40.613 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.596+0000 7f8c892d0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:22:40.613 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.596+0000 7f8c892d0640 1 -- 172.21.15.63:0/588076214 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c840a38f0 con 0x7f8c840a2f70 2024-04-26T04:22:40.613 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.596+0000 7f8c6b7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T04:22:40.613 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.596+0000 7f8c6b7fe640 1 -- 172.21.15.63:0/588076214 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c7c013b90 con 0x7f8c840a2f70 2024-04-26T04:22:40.613 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.596+0000 7f8c6b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:40.613 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.596+0000 7f8c6b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:22:40.613 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.596+0000 7f8c6b7fe640 10 monclient: dump: 2024-04-26T04:22:40.613 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:40.613 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:40.613 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:40.614 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:40.614 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:40.614 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:40.614 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:40.614 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:40.614 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:40.614 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:40.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.596+0000 7f8c892d0640 10 monclient: _renew_subs 2024-04-26T04:22:40.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.596+0000 7f8c892d0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:22:40.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.596+0000 7f8c80ff9640 4 set_mon_vals no callback set 2024-04-26T04:22:40.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.596+0000 7f8c892d0640 1 -- 172.21.15.63:0/588076214 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c84139f60 con 0x7f8c840a2f70 2024-04-26T04:22:40.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.596+0000 7f8c892d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:22:40.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.596+0000 7f8c892d0640 1 --2- 172.21.15.63:0/588076214 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8c8413c350 0x7f8c84138330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:40.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.596+0000 7f8c892d0640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:40.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.596+0000 7f8c892d0640 1 -- 172.21.15.63:0/588076214 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f8c84062e10 con 0x7f8c8413c350 2024-04-26T04:22:40.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.596+0000 7f8c6b7fe640 1 -- 172.21.15.63:0/588076214 <== mon.2 v2:172.21.15.63:3301/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f8c7c01e460 con 0x7f8c840a2f70 2024-04-26T04:22:40.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.596+0000 7f8c82ffd640 1 --2- 172.21.15.63:0/588076214 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8c8413c350 0x7f8c84138330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:40.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.596+0000 7f8c82ffd640 10 monclient: get_auth_request con 0x7f8c8413c350 auth_method 0 2024-04-26T04:22:40.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.596+0000 7f8c82ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:40.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.596+0000 7f8c82ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:40.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.596+0000 7f8c82ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:40.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.596+0000 7f8c82ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:22:40.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.596+0000 7f8c82ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:22:40.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.596+0000 7f8c6b7fe640 1 --2- 172.21.15.63:0/588076214 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f8c70035ec0 0x7f8c70038380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:40.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.596+0000 7f8c82ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:22:40.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.596+0000 7f8c6b7fe640 1 -- 172.21.15.63:0/588076214 <== mon.2 v2:172.21.15.63:3301/0 5 ==== osd_map(214..214 src has 1..214) ==== 4738+0+0 (secure 0 0 0) 0x7f8c7c0496d0 con 0x7f8c840a2f70 2024-04-26T04:22:40.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.596+0000 7f8c82ffd640 10 monclient(hunting): handle_auth_done global_id 5232 payload 824 2024-04-26T04:22:40.616 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.596+0000 7f8c82ffd640 1 -- 172.21.15.63:0/588076214 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8c8413e130 con 0x7f8c8413c350 2024-04-26T04:22:40.616 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.596+0000 7f8c827fc640 1 --2- 172.21.15.63:0/588076214 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f8c70035ec0 0x7f8c70038380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:40.616 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.596+0000 7f8c827fc640 10 monclient: get_auth_request con 0x7f8c70035ec0 auth_method 0 2024-04-26T04:22:40.616 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.596+0000 7f8c82ffd640 1 -- 172.21.15.63:0/588076214 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8c8413c350 msgr2=0x7f8c84138330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:22:40.616 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.596+0000 7f8c82ffd640 1 --2- 172.21.15.63:0/588076214 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8c8413c350 0x7f8c84138330 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f8c7400d380 tx=0x7f8c7400d840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:22:40.616 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.600+0000 7f8c6b7fe640 1 -- 172.21.15.63:0/588076214 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c74016020 con 0x7f8c8413c350 2024-04-26T04:22:40.616 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.600+0000 7f8c6b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:40.616 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.600+0000 7f8c6b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:22:40.616 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.600+0000 7f8c6b7fe640 10 monclient: dump: 2024-04-26T04:22:40.616 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:40.616 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:40.616 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:40.616 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:40.616 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:40.617 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:40.617 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:40.617 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:40.617 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:40.617 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:40.617 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.600+0000 7f8c6b7fe640 1 -- 172.21.15.63:0/588076214 <== mon.0 v2:172.21.15.63:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8c74012a10 con 0x7f8c8413c350 2024-04-26T04:22:40.617 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.600+0000 7f8c6b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:22:40.617 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.600+0000 7f8c6b7fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:22:40.617 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.600+0000 7f8c6b7fe640 1 -- 172.21.15.63:0/588076214 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8c8413c350 msgr2=0x7f8c84138330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:40.617 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.600+0000 7f8c6b7fe640 1 --2- 172.21.15.63:0/588076214 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8c8413c350 0x7f8c84138330 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f8c7400d380 tx=0x7f8c7400d840 comp rx=0 tx=0).stop 2024-04-26T04:22:40.618 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.600+0000 7f8c6b7fe640 1 -- 172.21.15.63:0/588076214 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8c8413c350 msgr2=0x7f8c84138330 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:22:40.618 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.600+0000 7f8c6b7fe640 1 --2- 172.21.15.63:0/588076214 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8c8413c350 0x7f8c84138330 secure :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f8c7400d380 tx=0x7f8c7400d840 comp rx=0 tx=0).stop 2024-04-26T04:22:40.618 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.600+0000 7f8c827fc640 1 --2- 172.21.15.63:0/588076214 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f8c70035ec0 0x7f8c70038380 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f8c780ac000 tx=0x7f8c78008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:22:40.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.604+0000 7f8c892d0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T04:22:40.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.604+0000 7f8c892d0640 1 --2- 172.21.15.63:0/588076214 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8c84138870 0x7f8c84138c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:40.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.604+0000 7f8c892d0640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:40.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.604+0000 7f8c892d0640 1 -- 172.21.15.63:0/588076214 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f8c841391c0 con 0x7f8c84138870 2024-04-26T04:22:40.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.604+0000 7f8c82ffd640 1 --2- 172.21.15.63:0/588076214 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8c84138870 0x7f8c84138c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:40.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.604+0000 7f8c82ffd640 10 monclient: get_auth_request con 0x7f8c84138870 auth_method 0 2024-04-26T04:22:40.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.604+0000 7f8c82ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:40.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.604+0000 7f8c82ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:40.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.604+0000 7f8c82ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:40.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.604+0000 7f8c82ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:22:40.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.604+0000 7f8c82ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:22:40.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.604+0000 7f8c82ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:22:40.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.604+0000 7f8c82ffd640 10 monclient(hunting): handle_auth_done global_id 5235 payload 824 2024-04-26T04:22:40.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.604+0000 7f8c82ffd640 1 -- 172.21.15.63:0/588076214 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8c84068430 con 0x7f8c84138870 2024-04-26T04:22:40.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.608+0000 7f8c82ffd640 1 -- 172.21.15.63:0/588076214 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8c84138870 msgr2=0x7f8c84138c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:22:40.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.608+0000 7f8c82ffd640 1 --2- 172.21.15.63:0/588076214 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8c84138870 0x7f8c84138c80 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f8c7400d380 tx=0x7f8c74003770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:22:40.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.608+0000 7f8c6b7fe640 1 -- 172.21.15.63:0/588076214 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c74016040 con 0x7f8c84138870 2024-04-26T04:22:40.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.608+0000 7f8c6b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:40.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.608+0000 7f8c6b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:22:40.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.608+0000 7f8c6b7fe640 10 monclient: dump: 2024-04-26T04:22:40.621 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:40.621 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:40.621 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:40.621 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:40.621 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:40.622 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:40.622 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:40.622 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:40.622 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:40.622 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:40.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.608+0000 7f8c6b7fe640 1 -- 172.21.15.63:0/588076214 <== mon.0 v2:172.21.15.63:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4723 (secure 0 0 0) 0x7f8c7401a8c0 con 0x7f8c84138870 2024-04-26T04:22:40.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.608+0000 7f8c6b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T04:22:40.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.608+0000 7f8c6b7fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:22:40.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.608+0000 7f8c6b7fe640 1 -- 172.21.15.63:0/588076214 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8c84138870 msgr2=0x7f8c84138c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:40.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.608+0000 7f8c6b7fe640 1 --2- 172.21.15.63:0/588076214 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8c84138870 0x7f8c84138c80 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f8c7400d380 tx=0x7f8c74003770 comp rx=0 tx=0).stop 2024-04-26T04:22:40.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.608+0000 7f8c6b7fe640 1 -- 172.21.15.63:0/588076214 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8c84138870 msgr2=0x7f8c84138c80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:22:40.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.608+0000 7f8c6b7fe640 1 --2- 172.21.15.63:0/588076214 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8c84138870 0x7f8c84138c80 secure :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f8c7400d380 tx=0x7f8c74003770 comp rx=0 tx=0).stop 2024-04-26T04:22:40.623 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-26T04:22:40.623 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a", 2024-04-26T04:22:40.623 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 0, 2024-04-26T04:22:40.623 INFO:teuthology.orchestra.run.smithi063.stdout: "state": "leader", 2024-04-26T04:22:40.623 INFO:teuthology.orchestra.run.smithi063.stdout: "election_epoch": 20, 2024-04-26T04:22:40.623 INFO:teuthology.orchestra.run.smithi063.stdout: "uptime": 251761, 2024-04-26T04:22:40.623 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum": [ 2024-04-26T04:22:40.623 INFO:teuthology.orchestra.run.smithi063.stdout: 0, 2024-04-26T04:22:40.623 INFO:teuthology.orchestra.run.smithi063.stdout: 1, 2024-04-26T04:22:40.625 INFO:teuthology.orchestra.run.smithi063.stdout: 2 2024-04-26T04:22:40.625 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:22:40.625 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_age": 7, 2024-04-26T04:22:40.625 INFO:teuthology.orchestra.run.smithi063.stdout: "features": { 2024-04-26T04:22:40.625 INFO:teuthology.orchestra.run.smithi063.stdout: "required_con": "2450521708007866372", 2024-04-26T04:22:40.625 INFO:teuthology.orchestra.run.smithi063.stdout: "required_mon": [ 2024-04-26T04:22:40.625 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:22:40.625 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:22:40.625 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:22:40.625 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:22:40.625 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:22:40.626 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:22:40.626 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:22:40.626 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:22:40.626 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:22:40.626 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:22:40.626 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:22:40.626 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:22:40.626 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_con": "4540701547738038271", 2024-04-26T04:22:40.626 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_mon": [ 2024-04-26T04:22:40.627 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:22:40.627 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:22:40.627 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:22:40.627 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:22:40.627 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:22:40.627 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:22:40.627 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:22:40.627 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:22:40.627 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:22:40.627 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:22:40.627 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:22:40.627 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:22:40.627 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:40.627 INFO:teuthology.orchestra.run.smithi063.stdout: "outside_quorum": [], 2024-04-26T04:22:40.627 INFO:teuthology.orchestra.run.smithi063.stdout: "extra_probe_peers": [], 2024-04-26T04:22:40.628 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_provider": [], 2024-04-26T04:22:40.628 INFO:teuthology.orchestra.run.smithi063.stdout: "monmap": { 2024-04-26T04:22:40.628 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 1, 2024-04-26T04:22:40.628 INFO:teuthology.orchestra.run.smithi063.stdout: "fsid": "cd1c6a0c-4e4b-4014-89c0-2e535479ef39", 2024-04-26T04:22:40.628 INFO:teuthology.orchestra.run.smithi063.stdout: "modified": "2024-04-26T04:18:03.702339Z", 2024-04-26T04:22:40.628 INFO:teuthology.orchestra.run.smithi063.stdout: "created": "2024-04-26T04:18:03.702339Z", 2024-04-26T04:22:40.628 INFO:teuthology.orchestra.run.smithi063.stdout: "min_mon_release": 19, 2024-04-26T04:22:40.628 INFO:teuthology.orchestra.run.smithi063.stdout: "min_mon_release_name": "squid", 2024-04-26T04:22:40.628 INFO:teuthology.orchestra.run.smithi063.stdout: "election_strategy": 1, 2024-04-26T04:22:40.628 INFO:teuthology.orchestra.run.smithi063.stdout: "disallowed_leaders: ": "", 2024-04-26T04:22:40.628 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": false, 2024-04-26T04:22:40.628 INFO:teuthology.orchestra.run.smithi063.stdout: "tiebreaker_mon": "", 2024-04-26T04:22:40.628 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_ranks: ": "", 2024-04-26T04:22:40.628 INFO:teuthology.orchestra.run.smithi063.stdout: "features": { 2024-04-26T04:22:40.629 INFO:teuthology.orchestra.run.smithi063.stdout: "persistent": [ 2024-04-26T04:22:40.629 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:22:40.629 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:22:40.629 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:22:40.629 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:22:40.629 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:22:40.629 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:22:40.629 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:22:40.629 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:22:40.629 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:22:40.630 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:22:40.630 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:22:40.630 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:22:40.630 INFO:teuthology.orchestra.run.smithi063.stdout: "optional": [] 2024-04-26T04:22:40.630 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:40.630 INFO:teuthology.orchestra.run.smithi063.stdout: "mons": [ 2024-04-26T04:22:40.630 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:40.630 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 0, 2024-04-26T04:22:40.630 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a", 2024-04-26T04:22:40.630 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:22:40.630 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:22:40.630 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:40.630 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:22:40.631 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:3300", 2024-04-26T04:22:40.631 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:22:40.631 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:40.631 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:40.631 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:22:40.631 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6789", 2024-04-26T04:22:40.631 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:22:40.631 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:22:40.631 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:22:40.631 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:40.632 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6789/0", 2024-04-26T04:22:40.632 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6789/0", 2024-04-26T04:22:40.632 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:22:40.632 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:22:40.632 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:22:40.632 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:40.632 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:40.632 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 1, 2024-04-26T04:22:40.632 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b", 2024-04-26T04:22:40.632 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:22:40.632 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:22:40.632 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:40.632 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:22:40.632 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:3300", 2024-04-26T04:22:40.633 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:22:40.633 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:40.633 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:40.633 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:22:40.633 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:6789", 2024-04-26T04:22:40.633 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:22:40.633 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:22:40.633 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:22:40.633 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:40.633 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:6789/0", 2024-04-26T04:22:40.634 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.186:6789/0", 2024-04-26T04:22:40.634 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:22:40.634 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:22:40.634 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:22:40.634 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:40.634 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:40.634 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 2, 2024-04-26T04:22:40.634 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "c", 2024-04-26T04:22:40.634 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:22:40.634 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:22:40.634 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:40.634 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:22:40.635 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:3301", 2024-04-26T04:22:40.635 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:22:40.635 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:40.635 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:40.635 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:22:40.635 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6790", 2024-04-26T04:22:40.635 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:22:40.635 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:22:40.635 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:22:40.635 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:40.635 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6790/0", 2024-04-26T04:22:40.636 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6790/0", 2024-04-26T04:22:40.636 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:22:40.636 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:22:40.636 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:22:40.636 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:22:40.636 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:22:40.636 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:40.636 INFO:teuthology.orchestra.run.smithi063.stdout: "feature_map": { 2024-04-26T04:22:40.636 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": [ 2024-04-26T04:22:40.636 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:40.637 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:22:40.637 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:22:40.637 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 1 2024-04-26T04:22:40.637 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:22:40.637 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:22:40.637 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": [ 2024-04-26T04:22:40.637 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:40.637 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:22:40.637 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:22:40.637 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 3 2024-04-26T04:22:40.638 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:22:40.638 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:22:40.638 INFO:teuthology.orchestra.run.smithi063.stdout: "client": [ 2024-04-26T04:22:40.638 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:40.638 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:22:40.638 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:22:40.638 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 1 2024-04-26T04:22:40.638 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:22:40.638 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:22:40.638 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:40.639 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": false 2024-04-26T04:22:40.639 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-26T04:22:40.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.608+0000 7f8c892d0640 1 -- 172.21.15.63:0/588076214 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f8c70035ec0 msgr2=0x7f8c70038380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:40.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.608+0000 7f8c892d0640 1 --2- 172.21.15.63:0/588076214 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f8c70035ec0 0x7f8c70038380 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f8c780ac000 tx=0x7f8c78008040 comp rx=0 tx=0).stop 2024-04-26T04:22:40.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.608+0000 7f8c892d0640 10 monclient: shutdown 2024-04-26T04:22:40.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.608+0000 7f8c892d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:22:40.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.608+0000 7f8c892d0640 1 -- 172.21.15.63:0/588076214 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f8c840a2f70 msgr2=0x7f8c84137df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:40.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.608+0000 7f8c892d0640 1 --2- 172.21.15.63:0/588076214 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f8c840a2f70 0x7f8c84137df0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c007e40 tx=0x7f8c7c00e4b0 comp rx=0 tx=0).stop 2024-04-26T04:22:40.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.608+0000 7f8c837fe640 1 -- 172.21.15.63:0/588076214 reap_dead start 2024-04-26T04:22:40.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.608+0000 7f8c837fe640 1 -- 172.21.15.63:0/588076214 reap_dead start 2024-04-26T04:22:40.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.608+0000 7f8c892d0640 1 -- 172.21.15.63:0/588076214 shutdown_connections 2024-04-26T04:22:40.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.608+0000 7f8c892d0640 1 -- 172.21.15.63:0/588076214 >> 172.21.15.63:0/588076214 conn(0x7f8c840a5fb0 msgr2=0x7f8c840a1a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:22:40.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.608+0000 7f8c892d0640 1 -- 172.21.15.63:0/588076214 shutdown_connections 2024-04-26T04:22:40.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.608+0000 7f8c892d0640 1 -- 172.21.15.63:0/588076214 wait complete. 2024-04-26T04:22:40.642 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T04:22:40.642 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-26T04:22:40.642 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T04:22:40.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.752+0000 7f2f89c34640 10 monclient: get_monmap_and_config 2024-04-26T04:22:40.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.752+0000 7f2f89c34640 10 monclient: build_initial_monmap 2024-04-26T04:22:40.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.752+0000 7f2f89c34640 10 monclient: monmap: 2024-04-26T04:22:40.755 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:22:40.755 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:40.755 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:22:40.755118+0000 2024-04-26T04:22:40.755 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:22:40.755118+0000 2024-04-26T04:22:40.755 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:22:40.755 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:40.755 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:22:40.755 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:22:40.756 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:22:40.756 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:22:40.756 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:22:40.756 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:40.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.752+0000 7f2f89c34640 1 Processor -- start 2024-04-26T04:22:40.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.752+0000 7f2f89c34640 1 -- start start 2024-04-26T04:22:40.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.752+0000 7f2f89c34640 10 monclient: init 2024-04-26T04:22:40.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.752+0000 7f2f89c34640 10 monclient: _reopen_session rank -1 2024-04-26T04:22:40.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.752+0000 7f2f89c34640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-04-26T04:22:40.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.752+0000 7f2f89c34640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f2f84134e10 0x7f2f841351f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:40.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.752+0000 7f2f89c34640 10 monclient(hunting): picked mon.noname-b con 0x7f2f84134e10 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:22:40.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.752+0000 7f2f89c34640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2f8406e9e0 addr v1:172.21.15.63:6789/0 2024-04-26T04:22:40.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.752+0000 7f2f89c34640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f2f84135730 0x7f2f840a4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:40.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.752+0000 7f2f89c34640 10 monclient(hunting): picked mon.noname-a con 0x7f2f84135730 addr v2:172.21.15.63:3300/0 2024-04-26T04:22:40.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.752+0000 7f2f89c34640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f84071940 con 0x7f2f8406e9e0 2024-04-26T04:22:40.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.752+0000 7f2f89c34640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:40.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.752+0000 7f2f89c34640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7f2f84071c20 con 0x7f2f84135730 2024-04-26T04:22:40.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.752+0000 7f2f89c34640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:40.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.752+0000 7f2f89c34640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f2f8406c0d0 con 0x7f2f84134e10 2024-04-26T04:22:40.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.752+0000 7f2f89c34640 10 monclient(hunting): _renew_subs 2024-04-26T04:22:40.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.752+0000 7f2f89c34640 10 monclient(hunting): authenticate will time out at 1192.463281s 2024-04-26T04:22:40.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.752+0000 7f2f83fff640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f2f84135730 0x7f2f840a4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:40.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.752+0000 7f2f82ffd640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f2f8406e9e0 0x7f2f840a48c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:35694/0 (socket says 172.21.15.63:35694) 2024-04-26T04:22:40.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.752+0000 7f2f82ffd640 1 -- 172.21.15.63:0/1424349936 learned_addr learned my addr 172.21.15.63:0/1424349936 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-04-26T04:22:40.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.752+0000 7f2f83fff640 10 monclient(hunting): get_auth_request con 0x7f2f84135730 auth_method 0 2024-04-26T04:22:40.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.752+0000 7f2f83fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:40.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.752+0000 7f2f83fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:40.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.752+0000 7f2f83fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:40.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.752+0000 7f2f837fe640 1 --2- 172.21.15.63:0/1424349936 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f2f84134e10 0x7f2f841351f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:40.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.752+0000 7f2f83fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:22:40.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.752+0000 7f2f83fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:22:40.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.752+0000 7f2f83fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:22:40.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.752+0000 7f2f837fe640 10 monclient(hunting): get_auth_request con 0x7f2f84134e10 auth_method 0 2024-04-26T04:22:40.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.752+0000 7f2f837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:40.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.752+0000 7f2f837fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:40.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.752+0000 7f2f837fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:40.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.752+0000 7f2f83fff640 10 monclient(hunting): handle_auth_done global_id 5238 payload 274 2024-04-26T04:22:40.761 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.752+0000 7f2f83fff640 1 -- 172.21.15.63:0/1424349936 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f2f84134e10 msgr2=0x7f2f841351f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:22:40.761 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.752+0000 7f2f83fff640 1 --2- 172.21.15.63:0/1424349936 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f2f84134e10 0x7f2f841351f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:40.761 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.752+0000 7f2f83fff640 1 -- 172.21.15.63:0/1424349936 >> v1:172.21.15.63:6789/0 conn(0x7f2f8406e9e0 legacy=0x7f2f840a48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:22:40.761 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.752+0000 7f2f83fff640 10 monclient: _finish_hunting 0 2024-04-26T04:22:40.761 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.752+0000 7f2f83fff640 1 monclient: found mon.noname-a 2024-04-26T04:22:40.761 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.752+0000 7f2f83fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:22:40.761 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.752+0000 7f2f83fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:22:40.761 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.752+0000 7f2f83fff640 1 -- 172.21.15.63:0/1424349936 --> v2:172.21.15.63:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f84134ab0 con 0x7f2f84135730 2024-04-26T04:22:40.761 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.752+0000 7f2f837fe640 1 --2- 172.21.15.63:0/1424349936 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f2f84134e10 0x7f2f841351f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T04:22:40.763 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f83fff640 1 -- 172.21.15.63:0/1424349936 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f84135730 msgr2=0x7f2f840a4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:22:40.763 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f83fff640 1 --2- 172.21.15.63:0/1424349936 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f84135730 0x7f2f840a4fd0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c0097c0 tx=0x7f2f7c01c320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d6c2346b751a27c server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:22:40.763 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f827fc640 1 -- 172.21.15.63:0/1424349936 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f7c0b0070 con 0x7f2f84135730 2024-04-26T04:22:40.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:40.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f827fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:22:40.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f827fc640 10 monclient: dump: 2024-04-26T04:22:40.764 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:40.764 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:40.764 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:40.764 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:40.764 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:40.764 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:40.764 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:40.764 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:40.764 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:40.764 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:40.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f827fc640 10 monclient: _finish_auth 0 2024-04-26T04:22:40.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f827fc640 10 monclient: _check_auth_tickets 2024-04-26T04:22:40.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:22:40.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f827fc640 1 -- 172.21.15.63:0/1424349936 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f7c0ab600 con 0x7f2f84135730 2024-04-26T04:22:40.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f89c34640 5 monclient: authenticate success, global_id 5238 2024-04-26T04:22:40.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f89c34640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T04:22:40.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f827fc640 10 monclient: handle_config config(0 keys) 2024-04-26T04:22:40.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f89c34640 10 monclient: get_monmap_and_config success 2024-04-26T04:22:40.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f89c34640 4 set_mon_vals no callback set 2024-04-26T04:22:40.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f827fc640 1 -- 172.21.15.63:0/1424349936 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f7c0ab9d0 con 0x7f2f84135730 2024-04-26T04:22:40.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:40.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f827fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:22:40.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f827fc640 10 monclient: dump: 2024-04-26T04:22:40.765 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:40.766 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:40.766 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:40.766 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:40.766 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:40.766 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:40.766 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:40.766 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:40.766 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:40.766 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:40.766 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f89c34640 10 monclient: shutdown 2024-04-26T04:22:40.766 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f89c34640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:22:40.766 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f89c34640 1 -- 172.21.15.63:0/1424349936 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f84135730 msgr2=0x7f2f840a4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:40.766 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f89c34640 1 --2- 172.21.15.63:0/1424349936 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f84135730 0x7f2f840a4fd0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c0097c0 tx=0x7f2f7c01c320 comp rx=0 tx=0).stop 2024-04-26T04:22:40.766 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f89c34640 1 -- 172.21.15.63:0/1424349936 shutdown_connections 2024-04-26T04:22:40.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f89c34640 1 --2- 172.21.15.63:0/1424349936 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f84135730 0x7f2f840a4fd0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:40.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f89c34640 1 --2- 172.21.15.63:0/1424349936 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f2f84134e10 0x7f2f841351f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:40.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f89c34640 1 -- 172.21.15.63:0/1424349936 >> 172.21.15.63:0/1424349936 conn(0x7f2f840a60a0 msgr2=0x7f2f840a24b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:22:40.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f89c34640 1 -- 172.21.15.63:0/1424349936 shutdown_connections 2024-04-26T04:22:40.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f89c34640 1 -- 172.21.15.63:0/1424349936 wait complete. 2024-04-26T04:22:40.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f89c34640 10 monclient: build_initial_monmap 2024-04-26T04:22:40.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f89c34640 10 monclient: monmap: 2024-04-26T04:22:40.767 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:22:40.767 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:22:40.767 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:22:40.767 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:22:40.767 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:22:40.767 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:40.767 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:22:40.768 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:22:40.768 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:22:40.768 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:40.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f89c34640 1 Processor -- start 2024-04-26T04:22:40.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f89c34640 1 -- start start 2024-04-26T04:22:40.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f89c34640 10 monclient: init 2024-04-26T04:22:40.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f89c34640 10 monclient: _reopen_session rank -1 2024-04-26T04:22:40.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f89c34640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T04:22:40.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f89c34640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f8406e9e0 0x7f2f841cee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:40.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f89c34640 10 monclient(hunting): picked mon.noname-b con 0x7f2f8406e9e0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:22:40.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f89c34640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f84134e10 0x7f2f8413bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:40.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f89c34640 10 monclient(hunting): picked mon.noname-a con 0x7f2f84134e10 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:22:40.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f89c34640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f2f84135730 0x7f2f8413c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:40.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f89c34640 10 monclient(hunting): picked mon.noname-c con 0x7f2f84135730 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:22:40.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f89c34640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:40.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f89c34640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f2f8406e4a0 con 0x7f2f84134e10 2024-04-26T04:22:40.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f89c34640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:40.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f89c34640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2f840a5720 con 0x7f2f8406e9e0 2024-04-26T04:22:40.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f89c34640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:40.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f89c34640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f2f8413f840 con 0x7f2f84135730 2024-04-26T04:22:40.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f89c34640 10 monclient(hunting): _renew_subs 2024-04-26T04:22:40.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f89c34640 10 monclient(hunting): authenticate will time out at 1192.470674s 2024-04-26T04:22:40.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f83fff640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f2f84135730 0x7f2f8413c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:40.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f82ffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f84134e10 0x7f2f8413bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:40.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f83fff640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f2f84135730 0x7f2f8413c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:60228/0 (socket says 172.21.15.63:60228) 2024-04-26T04:22:40.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f83fff640 1 -- 172.21.15.63:0/1618562161 learned_addr learned my addr 172.21.15.63:0/1618562161 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:22:40.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f83fff640 10 monclient(hunting): get_auth_request con 0x7f2f84135730 auth_method 0 2024-04-26T04:22:40.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f83fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:40.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f83fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:40.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f83fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:40.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f82ffd640 10 monclient(hunting): get_auth_request con 0x7f2f84134e10 auth_method 0 2024-04-26T04:22:40.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f82ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:40.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f82ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:40.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f82ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:40.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f83fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:22:40.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f83fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:22:40.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f83fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:22:40.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f82ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:22:40.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f82ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:22:40.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f82ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:22:40.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f837fe640 1 --2- 172.21.15.63:0/1618562161 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f8406e9e0 0x7f2f841cee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:40.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f837fe640 10 monclient(hunting): get_auth_request con 0x7f2f8406e9e0 auth_method 0 2024-04-26T04:22:40.775 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:40.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f837fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:40.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f837fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:40.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f83fff640 10 monclient(hunting): handle_auth_done global_id 24143 payload 824 2024-04-26T04:22:40.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f83fff640 1 -- 172.21.15.63:0/1618562161 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f8406e9e0 msgr2=0x7f2f841cee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:40.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f83fff640 1 --2- 172.21.15.63:0/1618562161 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f8406e9e0 0x7f2f841cee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:40.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f83fff640 1 -- 172.21.15.63:0/1618562161 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f84134e10 msgr2=0x7f2f8413bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:40.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f83fff640 1 --2- 172.21.15.63:0/1618562161 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f84134e10 0x7f2f8413bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:40.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f83fff640 10 monclient: _finish_hunting 0 2024-04-26T04:22:40.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f83fff640 1 monclient: found mon.noname-c 2024-04-26T04:22:40.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f83fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:22:40.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f83fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.63:3301/0 2024-04-26T04:22:40.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f83fff640 1 -- 172.21.15.63:0/1618562161 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f8406ee30 con 0x7f2f84135730 2024-04-26T04:22:40.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.760+0000 7f2f82ffd640 1 --2- 172.21.15.63:0/1618562161 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f84134e10 0x7f2f8413bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T04:22:40.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.764+0000 7f2f83fff640 1 -- 172.21.15.63:0/1618562161 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f2f84135730 msgr2=0x7f2f8413c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:22:40.777 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.764+0000 7f2f83fff640 1 --2- 172.21.15.63:0/1618562161 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f2f84135730 0x7f2f8413c420 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c01cbc0 tx=0x7f2f7c0190c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:22:40.777 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.764+0000 7f2f7bfff640 1 -- 172.21.15.63:0/1618562161 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f7c0b8020 con 0x7f2f84135730 2024-04-26T04:22:40.777 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.764+0000 7f2f7bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:40.777 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.764+0000 7f2f7bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:22:40.777 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.764+0000 7f2f7bfff640 10 monclient: dump: 2024-04-26T04:22:40.777 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:40.777 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:40.777 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:40.777 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:40.777 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:40.777 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:40.777 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:40.777 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:40.777 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:40.778 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:40.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.764+0000 7f2f7bfff640 10 monclient: _finish_auth 0 2024-04-26T04:22:40.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.764+0000 7f2f7bfff640 10 monclient: _check_auth_tickets 2024-04-26T04:22:40.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.764+0000 7f2f7bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:22:40.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.764+0000 7f2f7bfff640 1 -- 172.21.15.63:0/1618562161 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f7c0b4810 con 0x7f2f84135730 2024-04-26T04:22:40.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.764+0000 7f2f7bfff640 10 monclient: handle_config config(0 keys) 2024-04-26T04:22:40.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.764+0000 7f2f7bfff640 1 -- 172.21.15.63:0/1618562161 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f7c0b0040 con 0x7f2f84135730 2024-04-26T04:22:40.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.764+0000 7f2f7bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:40.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.764+0000 7f2f7bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:22:40.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.764+0000 7f2f7bfff640 10 monclient: dump: 2024-04-26T04:22:40.778 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:40.778 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:40.778 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:40.778 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:40.778 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:40.779 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:40.779 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:40.779 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:40.779 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:40.779 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:40.779 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.764+0000 7f2f89c34640 5 monclient: authenticate success, global_id 24143 2024-04-26T04:22:40.779 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.764+0000 7f2f89c34640 10 monclient: _renew_subs 2024-04-26T04:22:40.779 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.764+0000 7f2f81ffb640 4 set_mon_vals no callback set 2024-04-26T04:22:40.779 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.764+0000 7f2f89c34640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:22:40.779 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.764+0000 7f2f89c34640 1 -- 172.21.15.63:0/1618562161 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f841341a0 con 0x7f2f84135730 2024-04-26T04:22:40.779 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.764+0000 7f2f89c34640 10 monclient: _renew_subs 2024-04-26T04:22:40.779 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.764+0000 7f2f89c34640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:22:40.779 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.764+0000 7f2f89c34640 1 -- 172.21.15.63:0/1618562161 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f8413a4a0 con 0x7f2f84135730 2024-04-26T04:22:40.779 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.764+0000 7f2f89c34640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:22:40.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.764+0000 7f2f89c34640 1 --2- 172.21.15.63:0/1618562161 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f2f84136880 0x7f2f84136c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:40.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.764+0000 7f2f89c34640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:40.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.764+0000 7f2f89c34640 1 -- 172.21.15.63:0/1618562161 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f2f841cf340 con 0x7f2f84136880 2024-04-26T04:22:40.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.764+0000 7f2f837fe640 1 --2- 172.21.15.63:0/1618562161 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f2f84136880 0x7f2f84136c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:40.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.764+0000 7f2f7bfff640 1 -- 172.21.15.63:0/1618562161 <== mon.2 v2:172.21.15.63:3301/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f2f7c0bb020 con 0x7f2f84135730 2024-04-26T04:22:40.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.764+0000 7f2f837fe640 10 monclient: get_auth_request con 0x7f2f84136880 auth_method 0 2024-04-26T04:22:40.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.764+0000 7f2f837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:40.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.764+0000 7f2f837fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:40.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.764+0000 7f2f837fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:40.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.764+0000 7f2f837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:22:40.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.764+0000 7f2f837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:22:40.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.764+0000 7f2f837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:22:40.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.764+0000 7f2f7bfff640 1 --2- 172.21.15.63:0/1618562161 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f2f68035ef0 0x7f2f680383b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:40.780 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.764+0000 7f2f7bfff640 1 -- 172.21.15.63:0/1618562161 <== mon.2 v2:172.21.15.63:3301/0 5 ==== osd_map(214..214 src has 1..214) ==== 4738+0+0 (secure 0 0 0) 0x7f2f7c0e2ae0 con 0x7f2f84135730 2024-04-26T04:22:40.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.764+0000 7f2f837fe640 10 monclient(hunting): handle_auth_done global_id 24146 payload 824 2024-04-26T04:22:40.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.764+0000 7f2f837fe640 1 -- 172.21.15.63:0/1618562161 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2f84062e10 con 0x7f2f84136880 2024-04-26T04:22:40.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.764+0000 7f2f82ffd640 1 --2- 172.21.15.63:0/1618562161 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f2f68035ef0 0x7f2f680383b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:40.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.764+0000 7f2f82ffd640 10 monclient: get_auth_request con 0x7f2f68035ef0 auth_method 0 2024-04-26T04:22:40.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.768+0000 7f2f837fe640 1 -- 172.21.15.63:0/1618562161 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f2f84136880 msgr2=0x7f2f84136c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:22:40.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.768+0000 7f2f837fe640 1 --2- 172.21.15.63:0/1618562161 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f2f84136880 0x7f2f84136c60 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c004610 tx=0x7f2f6c007330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:22:40.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.768+0000 7f2f7bfff640 1 -- 172.21.15.63:0/1618562161 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f6c013780 con 0x7f2f84136880 2024-04-26T04:22:40.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.768+0000 7f2f7bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:40.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.768+0000 7f2f7bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:22:40.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.768+0000 7f2f7bfff640 10 monclient: dump: 2024-04-26T04:22:40.781 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:40.781 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:40.781 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:40.781 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:40.782 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:40.782 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:40.782 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:40.782 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:40.782 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:40.782 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:40.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.768+0000 7f2f7bfff640 1 -- 172.21.15.63:0/1618562161 <== mon.2 v2:172.21.15.63:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2f6c019070 con 0x7f2f84136880 2024-04-26T04:22:40.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.768+0000 7f2f7bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:22:40.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.768+0000 7f2f7bfff640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:22:40.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.768+0000 7f2f7bfff640 1 -- 172.21.15.63:0/1618562161 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f2f84136880 msgr2=0x7f2f84136c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:40.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.768+0000 7f2f7bfff640 1 --2- 172.21.15.63:0/1618562161 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f2f84136880 0x7f2f84136c60 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c004610 tx=0x7f2f6c007330 comp rx=0 tx=0).stop 2024-04-26T04:22:40.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.768+0000 7f2f7bfff640 1 -- 172.21.15.63:0/1618562161 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f2f84136880 msgr2=0x7f2f84136c60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:22:40.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.768+0000 7f2f7bfff640 1 --2- 172.21.15.63:0/1618562161 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f2f84136880 0x7f2f84136c60 secure :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c004610 tx=0x7f2f6c007330 comp rx=0 tx=0).stop 2024-04-26T04:22:40.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.768+0000 7f2f82ffd640 1 --2- 172.21.15.63:0/1618562161 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f2f68035ef0 0x7f2f680383b0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f2f74007ca0 tx=0x7f2f74007bf0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:22:40.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.772+0000 7f2f89c34640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T04:22:40.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.772+0000 7f2f89c34640 1 --2- 172.21.15.63:0/1618562161 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f2f841371a0 0x7f2f8412f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:40.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.772+0000 7f2f89c34640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:40.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.772+0000 7f2f89c34640 1 -- 172.21.15.63:0/1618562161 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f2f84068260 con 0x7f2f841371a0 2024-04-26T04:22:40.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.772+0000 7f2f837fe640 1 --2- 172.21.15.63:0/1618562161 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f2f841371a0 0x7f2f8412f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:40.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.772+0000 7f2f837fe640 10 monclient: get_auth_request con 0x7f2f841371a0 auth_method 0 2024-04-26T04:22:40.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.772+0000 7f2f837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:40.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.772+0000 7f2f837fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:40.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.772+0000 7f2f837fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:40.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.772+0000 7f2f837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:22:40.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.772+0000 7f2f837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:22:40.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.772+0000 7f2f837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:22:40.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.776+0000 7f2f837fe640 10 monclient(hunting): handle_auth_done global_id 24149 payload 824 2024-04-26T04:22:40.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.776+0000 7f2f837fe640 1 -- 172.21.15.63:0/1618562161 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2f841375b0 con 0x7f2f841371a0 2024-04-26T04:22:40.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.776+0000 7f2f837fe640 1 -- 172.21.15.63:0/1618562161 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f2f841371a0 msgr2=0x7f2f8412f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:22:40.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.776+0000 7f2f837fe640 1 --2- 172.21.15.63:0/1618562161 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f2f841371a0 0x7f2f8412f490 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c004610 tx=0x7f2f6c004a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:22:40.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.776+0000 7f2f7bfff640 1 -- 172.21.15.63:0/1618562161 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f6c029640 con 0x7f2f841371a0 2024-04-26T04:22:40.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.776+0000 7f2f7bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:40.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.776+0000 7f2f7bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:22:40.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.776+0000 7f2f7bfff640 10 monclient: dump: 2024-04-26T04:22:40.786 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:40.786 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:40.786 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:40.786 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:40.786 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:40.786 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:40.786 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:40.786 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:40.786 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:40.786 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:40.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.776+0000 7f2f7bfff640 1 -- 172.21.15.63:0/1618562161 <== mon.2 v2:172.21.15.63:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4549 (secure 0 0 0) 0x7f2f6c014410 con 0x7f2f841371a0 2024-04-26T04:22:40.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.776+0000 7f2f7bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T04:22:40.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.776+0000 7f2f7bfff640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:22:40.786 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.776+0000 7f2f7bfff640 1 -- 172.21.15.63:0/1618562161 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f2f841371a0 msgr2=0x7f2f8412f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:40.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.776+0000 7f2f7bfff640 1 --2- 172.21.15.63:0/1618562161 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f2f841371a0 0x7f2f8412f490 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c004610 tx=0x7f2f6c004a50 comp rx=0 tx=0).stop 2024-04-26T04:22:40.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.776+0000 7f2f7bfff640 1 -- 172.21.15.63:0/1618562161 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f2f841371a0 msgr2=0x7f2f8412f490 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:22:40.787 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.776+0000 7f2f7bfff640 1 --2- 172.21.15.63:0/1618562161 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f2f841371a0 0x7f2f8412f490 secure :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c004610 tx=0x7f2f6c004a50 comp rx=0 tx=0).stop 2024-04-26T04:22:40.787 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-26T04:22:40.787 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "c", 2024-04-26T04:22:40.787 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 2, 2024-04-26T04:22:40.787 INFO:teuthology.orchestra.run.smithi063.stdout: "state": "peon", 2024-04-26T04:22:40.787 INFO:teuthology.orchestra.run.smithi063.stdout: "election_epoch": 20, 2024-04-26T04:22:40.787 INFO:teuthology.orchestra.run.smithi063.stdout: "uptime": 8375, 2024-04-26T04:22:40.787 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum": [ 2024-04-26T04:22:40.788 INFO:teuthology.orchestra.run.smithi063.stdout: 0, 2024-04-26T04:22:40.788 INFO:teuthology.orchestra.run.smithi063.stdout: 1, 2024-04-26T04:22:40.788 INFO:teuthology.orchestra.run.smithi063.stdout: 2 2024-04-26T04:22:40.788 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:22:40.788 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_age": 8, 2024-04-26T04:22:40.788 INFO:teuthology.orchestra.run.smithi063.stdout: "features": { 2024-04-26T04:22:40.788 INFO:teuthology.orchestra.run.smithi063.stdout: "required_con": "2450521708007866372", 2024-04-26T04:22:40.788 INFO:teuthology.orchestra.run.smithi063.stdout: "required_mon": [ 2024-04-26T04:22:40.788 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:22:40.788 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:22:40.788 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:22:40.788 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:22:40.788 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:22:40.789 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:22:40.789 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:22:40.789 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:22:40.789 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:22:40.789 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:22:40.789 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:22:40.789 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:22:40.789 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_con": "4540701547738038271", 2024-04-26T04:22:40.789 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_mon": [ 2024-04-26T04:22:40.789 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:22:40.789 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:22:40.789 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:22:40.789 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:22:40.789 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:22:40.790 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:22:40.790 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:22:40.790 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:22:40.790 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:22:40.790 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:22:40.790 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:22:40.790 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:22:40.790 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:40.790 INFO:teuthology.orchestra.run.smithi063.stdout: "outside_quorum": [], 2024-04-26T04:22:40.790 INFO:teuthology.orchestra.run.smithi063.stdout: "extra_probe_peers": [], 2024-04-26T04:22:40.790 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_provider": [], 2024-04-26T04:22:40.790 INFO:teuthology.orchestra.run.smithi063.stdout: "monmap": { 2024-04-26T04:22:40.790 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 1, 2024-04-26T04:22:40.791 INFO:teuthology.orchestra.run.smithi063.stdout: "fsid": "cd1c6a0c-4e4b-4014-89c0-2e535479ef39", 2024-04-26T04:22:40.791 INFO:teuthology.orchestra.run.smithi063.stdout: "modified": "2024-04-26T04:18:03.702339Z", 2024-04-26T04:22:40.791 INFO:teuthology.orchestra.run.smithi063.stdout: "created": "2024-04-26T04:18:03.702339Z", 2024-04-26T04:22:40.791 INFO:teuthology.orchestra.run.smithi063.stdout: "min_mon_release": 19, 2024-04-26T04:22:40.791 INFO:teuthology.orchestra.run.smithi063.stdout: "min_mon_release_name": "squid", 2024-04-26T04:22:40.791 INFO:teuthology.orchestra.run.smithi063.stdout: "election_strategy": 1, 2024-04-26T04:22:40.791 INFO:teuthology.orchestra.run.smithi063.stdout: "disallowed_leaders: ": "", 2024-04-26T04:22:40.791 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": false, 2024-04-26T04:22:40.791 INFO:teuthology.orchestra.run.smithi063.stdout: "tiebreaker_mon": "", 2024-04-26T04:22:40.791 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_ranks: ": "", 2024-04-26T04:22:40.791 INFO:teuthology.orchestra.run.smithi063.stdout: "features": { 2024-04-26T04:22:40.791 INFO:teuthology.orchestra.run.smithi063.stdout: "persistent": [ 2024-04-26T04:22:40.791 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:22:40.791 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:22:40.792 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:22:40.792 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:22:40.792 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:22:40.792 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:22:40.792 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:22:40.792 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:22:40.792 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:22:40.792 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:22:40.792 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:22:40.792 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:22:40.792 INFO:teuthology.orchestra.run.smithi063.stdout: "optional": [] 2024-04-26T04:22:40.792 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:40.792 INFO:teuthology.orchestra.run.smithi063.stdout: "mons": [ 2024-04-26T04:22:40.792 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:40.793 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 0, 2024-04-26T04:22:40.793 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a", 2024-04-26T04:22:40.793 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:22:40.793 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:22:40.793 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:40.793 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:22:40.793 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:3300", 2024-04-26T04:22:40.793 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:22:40.793 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:40.793 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:40.793 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:22:40.793 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6789", 2024-04-26T04:22:40.793 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:22:40.793 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:22:40.794 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:22:40.794 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:40.794 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6789/0", 2024-04-26T04:22:40.794 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6789/0", 2024-04-26T04:22:40.794 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:22:40.794 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:22:40.794 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:22:40.794 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:40.794 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:40.794 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 1, 2024-04-26T04:22:40.794 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b", 2024-04-26T04:22:40.794 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:22:40.794 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:22:40.795 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:40.795 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:22:40.795 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:3300", 2024-04-26T04:22:40.795 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:22:40.795 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:40.795 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:40.795 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:22:40.795 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:6789", 2024-04-26T04:22:40.795 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:22:40.795 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:22:40.795 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:22:40.795 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:40.795 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:6789/0", 2024-04-26T04:22:40.795 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.186:6789/0", 2024-04-26T04:22:40.796 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:22:40.796 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:22:40.796 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:22:40.796 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:40.796 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:40.796 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 2, 2024-04-26T04:22:40.796 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "c", 2024-04-26T04:22:40.796 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:22:40.796 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:22:40.796 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:40.796 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:22:40.796 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:3301", 2024-04-26T04:22:40.796 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:22:40.796 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:40.797 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:40.797 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:22:40.797 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6790", 2024-04-26T04:22:40.797 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:22:40.797 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:22:40.797 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:22:40.797 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:40.797 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6790/0", 2024-04-26T04:22:40.797 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6790/0", 2024-04-26T04:22:40.797 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:22:40.797 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:22:40.797 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:22:40.797 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:22:40.797 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:22:40.798 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:40.798 INFO:teuthology.orchestra.run.smithi063.stdout: "feature_map": { 2024-04-26T04:22:40.798 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": [ 2024-04-26T04:22:40.798 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:40.798 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:22:40.798 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:22:40.798 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 1 2024-04-26T04:22:40.798 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:22:40.798 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:22:40.798 INFO:teuthology.orchestra.run.smithi063.stdout: "client": [ 2024-04-26T04:22:40.798 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:40.798 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:22:40.798 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:22:40.798 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 2 2024-04-26T04:22:40.799 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:22:40.799 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:22:40.799 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:40.799 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": false 2024-04-26T04:22:40.799 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-26T04:22:40.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.776+0000 7f2f89c34640 1 -- 172.21.15.63:0/1618562161 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f2f68035ef0 msgr2=0x7f2f680383b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:40.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.776+0000 7f2f89c34640 1 --2- 172.21.15.63:0/1618562161 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f2f68035ef0 0x7f2f680383b0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f2f74007ca0 tx=0x7f2f74007bf0 comp rx=0 tx=0).stop 2024-04-26T04:22:40.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.776+0000 7f2f89c34640 10 monclient: shutdown 2024-04-26T04:22:40.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.776+0000 7f2f89c34640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:22:40.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.776+0000 7f2f89c34640 1 -- 172.21.15.63:0/1618562161 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f2f84135730 msgr2=0x7f2f8413c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:40.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.776+0000 7f2f89c34640 1 --2- 172.21.15.63:0/1618562161 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f2f84135730 0x7f2f8413c420 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c01cbc0 tx=0x7f2f7c0190c0 comp rx=0 tx=0).stop 2024-04-26T04:22:40.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.776+0000 7f2f83fff640 1 -- 172.21.15.63:0/1618562161 reap_dead start 2024-04-26T04:22:40.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.776+0000 7f2f89c34640 1 -- 172.21.15.63:0/1618562161 shutdown_connections 2024-04-26T04:22:40.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.776+0000 7f2f89c34640 1 --2- 172.21.15.63:0/1618562161 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f2f84135730 0x7f2f8413c420 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:40.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.776+0000 7f2f89c34640 1 -- 172.21.15.63:0/1618562161 >> 172.21.15.63:0/1618562161 conn(0x7f2f840a60a0 msgr2=0x7f2f8409ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:22:40.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.776+0000 7f2f89c34640 1 -- 172.21.15.63:0/1618562161 shutdown_connections 2024-04-26T04:22:40.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.776+0000 7f2f89c34640 1 -- 172.21.15.63:0/1618562161 wait complete. 2024-04-26T04:22:40.801 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T04:22:40.801 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-26T04:22:40.802 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T04:22:40.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.916+0000 7fa155307640 10 monclient: get_monmap_and_config 2024-04-26T04:22:40.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.916+0000 7fa155307640 10 monclient: build_initial_monmap 2024-04-26T04:22:40.918 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.916+0000 7fa155307640 10 monclient: monmap: 2024-04-26T04:22:40.918 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:22:40.918 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:40.918 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:22:40.918018+0000 2024-04-26T04:22:40.918 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:22:40.918018+0000 2024-04-26T04:22:40.918 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:22:40.919 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:40.919 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:22:40.919 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:22:40.919 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:22:40.919 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:22:40.919 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:22:40.919 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:40.921 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.916+0000 7fa155307640 1 Processor -- start 2024-04-26T04:22:40.921 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.916+0000 7fa155307640 1 -- start start 2024-04-26T04:22:40.921 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.916+0000 7fa155307640 10 monclient: init 2024-04-26T04:22:40.921 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.916+0000 7fa155307640 10 monclient: _reopen_session rank -1 2024-04-26T04:22:40.921 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.916+0000 7fa155307640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-04-26T04:22:40.922 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.916+0000 7fa155307640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa15009fe60 addr v1:172.21.15.63:6789/0 2024-04-26T04:22:40.922 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.916+0000 7fa155307640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7fa1500a0240 0x7fa1500a2a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:40.922 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.916+0000 7fa155307640 10 monclient(hunting): picked mon.noname-a con 0x7fa1500a0240 addr v2:172.21.15.63:3300/0 2024-04-26T04:22:40.922 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.916+0000 7fa155307640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa1500a2f70 0x7fa15013e110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:40.922 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.916+0000 7fa155307640 10 monclient(hunting): picked mon.noname-c con 0x7fa1500a2f70 addr v2:172.21.15.186:3300/0 2024-04-26T04:22:40.922 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.916+0000 7fa155307640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1500a0c60 con 0x7fa15009fe60 2024-04-26T04:22:40.922 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.916+0000 7fa155307640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:40.922 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.916+0000 7fa155307640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7fa1500a3730 con 0x7fa1500a0240 2024-04-26T04:22:40.922 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.916+0000 7fa155307640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:40.922 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.916+0000 7fa155307640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fa1500a0f40 con 0x7fa1500a2f70 2024-04-26T04:22:40.922 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.916+0000 7fa155307640 10 monclient(hunting): _renew_subs 2024-04-26T04:22:40.922 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.916+0000 7fa155307640 10 monclient(hunting): authenticate will time out at 1192.625747s 2024-04-26T04:22:40.922 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.916+0000 7fa14e7fc640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7fa1500a0240 0x7fa1500a2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:40.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.916+0000 7fa14effd640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7fa15009fe60 0x7fa1501372f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:35710/0 (socket says 172.21.15.63:35710) 2024-04-26T04:22:40.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.916+0000 7fa14effd640 1 -- 172.21.15.63:0/2847040543 learned_addr learned my addr 172.21.15.63:0/2847040543 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-04-26T04:22:40.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.916+0000 7fa14e7fc640 10 monclient(hunting): get_auth_request con 0x7fa1500a0240 auth_method 0 2024-04-26T04:22:40.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.916+0000 7fa14e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:40.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.916+0000 7fa14e7fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:40.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.916+0000 7fa14e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:40.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.916+0000 7fa14e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:22:40.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.916+0000 7fa14e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:22:40.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.916+0000 7fa14e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:22:40.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.916+0000 7fa14f7fe640 1 --2- 172.21.15.63:0/2847040543 >> v2:172.21.15.186:3300/0 conn(0x7fa1500a2f70 0x7fa15013e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:40.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.916+0000 7fa14f7fe640 10 monclient(hunting): get_auth_request con 0x7fa1500a2f70 auth_method 0 2024-04-26T04:22:40.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.916+0000 7fa14f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:40.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.916+0000 7fa14f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:40.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.916+0000 7fa14f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:40.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.916+0000 7fa14e7fc640 10 monclient(hunting): handle_auth_done global_id 5244 payload 274 2024-04-26T04:22:40.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.916+0000 7fa14e7fc640 1 -- 172.21.15.63:0/2847040543 >> v2:172.21.15.186:3300/0 conn(0x7fa1500a2f70 msgr2=0x7fa15013e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:22:40.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.916+0000 7fa14e7fc640 1 --2- 172.21.15.63:0/2847040543 >> v2:172.21.15.186:3300/0 conn(0x7fa1500a2f70 0x7fa15013e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:40.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.916+0000 7fa14e7fc640 1 -- 172.21.15.63:0/2847040543 >> v1:172.21.15.63:6789/0 conn(0x7fa15009fe60 legacy=0x7fa1501372f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:22:40.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.916+0000 7fa14e7fc640 10 monclient: _finish_hunting 0 2024-04-26T04:22:40.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.916+0000 7fa14e7fc640 1 monclient: found mon.noname-a 2024-04-26T04:22:40.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.916+0000 7fa14e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:22:40.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.916+0000 7fa14e7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:22:40.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.916+0000 7fa14e7fc640 1 -- 172.21.15.63:0/2847040543 --> v2:172.21.15.63:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1500a1220 con 0x7fa1500a0240 2024-04-26T04:22:40.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.920+0000 7fa14e7fc640 1 -- 172.21.15.63:0/2847040543 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa1500a0240 msgr2=0x7fa1500a2a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:22:40.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.920+0000 7fa14e7fc640 1 --2- 172.21.15.63:0/2847040543 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa1500a0240 0x7fa1500a2a30 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fa1500a3420 tx=0x7fa14001c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32053ec724a4b121 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:22:40.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.920+0000 7fa14dffb640 1 -- 172.21.15.63:0/2847040543 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1400b2070 con 0x7fa1500a0240 2024-04-26T04:22:40.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.920+0000 7fa14dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:40.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.920+0000 7fa14dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:22:40.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.920+0000 7fa14dffb640 10 monclient: dump: 2024-04-26T04:22:40.927 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:40.927 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:40.927 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:40.927 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:40.927 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:40.928 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:40.928 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:40.928 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:40.928 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:40.928 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:40.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.920+0000 7fa14dffb640 10 monclient: _finish_auth 0 2024-04-26T04:22:40.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.920+0000 7fa14dffb640 10 monclient: _check_auth_tickets 2024-04-26T04:22:40.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.920+0000 7fa14dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:22:40.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.920+0000 7fa14dffb640 1 -- 172.21.15.63:0/2847040543 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1400a5aa0 con 0x7fa1500a0240 2024-04-26T04:22:40.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.920+0000 7fa155307640 5 monclient: authenticate success, global_id 5244 2024-04-26T04:22:40.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.920+0000 7fa155307640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T04:22:40.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.920+0000 7fa14dffb640 10 monclient: handle_config config(0 keys) 2024-04-26T04:22:40.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.920+0000 7fa155307640 10 monclient: get_monmap_and_config success 2024-04-26T04:22:40.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.920+0000 7fa14dffb640 1 -- 172.21.15.63:0/2847040543 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1400ad440 con 0x7fa1500a0240 2024-04-26T04:22:40.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.920+0000 7fa14dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:40.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.920+0000 7fa155307640 4 set_mon_vals no callback set 2024-04-26T04:22:40.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.920+0000 7fa14dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:22:40.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.920+0000 7fa14dffb640 10 monclient: dump: 2024-04-26T04:22:40.929 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:40.929 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:40.929 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:40.929 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:40.929 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:40.929 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:40.929 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:40.929 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:40.929 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:40.929 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:40.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.920+0000 7fa155307640 10 monclient: shutdown 2024-04-26T04:22:40.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.920+0000 7fa155307640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:22:40.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.920+0000 7fa155307640 1 -- 172.21.15.63:0/2847040543 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa1500a0240 msgr2=0x7fa1500a2a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:40.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.920+0000 7fa155307640 1 --2- 172.21.15.63:0/2847040543 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa1500a0240 0x7fa1500a2a30 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fa1500a3420 tx=0x7fa14001c7b0 comp rx=0 tx=0).stop 2024-04-26T04:22:40.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.920+0000 7fa155307640 1 -- 172.21.15.63:0/2847040543 shutdown_connections 2024-04-26T04:22:40.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.920+0000 7fa155307640 1 --2- 172.21.15.63:0/2847040543 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa1500a0240 0x7fa1500a2a30 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:40.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.920+0000 7fa155307640 1 --2- 172.21.15.63:0/2847040543 >> v2:172.21.15.186:3300/0 conn(0x7fa1500a2f70 0x7fa15013e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:40.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.920+0000 7fa155307640 1 -- 172.21.15.63:0/2847040543 >> 172.21.15.63:0/2847040543 conn(0x7fa1500a5fb0 msgr2=0x7fa1500a63c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:22:40.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.920+0000 7fa155307640 1 -- 172.21.15.63:0/2847040543 shutdown_connections 2024-04-26T04:22:40.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.920+0000 7fa155307640 1 -- 172.21.15.63:0/2847040543 wait complete. 2024-04-26T04:22:40.931 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.924+0000 7fa155307640 10 monclient: build_initial_monmap 2024-04-26T04:22:40.931 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.924+0000 7fa155307640 10 monclient: monmap: 2024-04-26T04:22:40.931 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:22:40.931 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:22:40.931 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:22:40.931 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:22:40.931 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:22:40.931 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:40.931 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:22:40.931 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:22:40.931 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:22:40.931 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:40.932 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.924+0000 7fa155307640 1 Processor -- start 2024-04-26T04:22:40.932 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.924+0000 7fa155307640 1 -- start start 2024-04-26T04:22:40.932 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.924+0000 7fa155307640 10 monclient: init 2024-04-26T04:22:40.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.924+0000 7fa155307640 10 monclient: _reopen_session rank -1 2024-04-26T04:22:40.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.924+0000 7fa155307640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T04:22:40.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.924+0000 7fa155307640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fa15009fe60 0x7fa1501ce5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:40.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.924+0000 7fa155307640 10 monclient(hunting): picked mon.noname-c con 0x7fa15009fe60 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:22:40.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.924+0000 7fa155307640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1500a0240 0x7fa15013cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:40.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.924+0000 7fa155307640 10 monclient(hunting): picked mon.noname-b con 0x7fa1500a0240 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:22:40.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.924+0000 7fa155307640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa1500a2f70 0x7fa15013d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:40.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.924+0000 7fa155307640 10 monclient(hunting): picked mon.noname-a con 0x7fa1500a2f70 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:22:40.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.924+0000 7fa155307640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:40.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.924+0000 7fa155307640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fa150141290 con 0x7fa1500a2f70 2024-04-26T04:22:40.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.924+0000 7fa155307640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:40.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.924+0000 7fa155307640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa150141110 con 0x7fa1500a0240 2024-04-26T04:22:40.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.924+0000 7fa155307640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:40.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.924+0000 7fa155307640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7fa150141410 con 0x7fa15009fe60 2024-04-26T04:22:40.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.924+0000 7fa155307640 10 monclient(hunting): _renew_subs 2024-04-26T04:22:40.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.924+0000 7fa155307640 10 monclient(hunting): authenticate will time out at 1192.633762s 2024-04-26T04:22:40.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.924+0000 7fa14f7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa1500a2f70 0x7fa15013d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:40.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.924+0000 7fa14f7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa1500a2f70 0x7fa15013d530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:46912/0 (socket says 172.21.15.63:46912) 2024-04-26T04:22:40.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.924+0000 7fa14f7fe640 1 -- 172.21.15.63:0/1039766072 learned_addr learned my addr 172.21.15.63:0/1039766072 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:22:40.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.924+0000 7fa14f7fe640 10 monclient(hunting): get_auth_request con 0x7fa1500a2f70 auth_method 0 2024-04-26T04:22:40.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.924+0000 7fa14f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:40.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.924+0000 7fa14f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:40.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.924+0000 7fa14f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:40.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.924+0000 7fa14effd640 1 --2- 172.21.15.63:0/1039766072 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fa15009fe60 0x7fa1501ce5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:40.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.924+0000 7fa14effd640 10 monclient(hunting): get_auth_request con 0x7fa15009fe60 auth_method 0 2024-04-26T04:22:40.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.924+0000 7fa14effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:40.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.924+0000 7fa14effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:40.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.924+0000 7fa14effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:40.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.924+0000 7fa14f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:22:40.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.924+0000 7fa14f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:22:40.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.924+0000 7fa14f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:22:40.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.924+0000 7fa14effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:22:40.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.924+0000 7fa14effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:22:40.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.924+0000 7fa14effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:22:40.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.924+0000 7fa14e7fc640 1 --2- 172.21.15.63:0/1039766072 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1500a0240 0x7fa15013cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:40.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.924+0000 7fa14e7fc640 10 monclient(hunting): get_auth_request con 0x7fa1500a0240 auth_method 0 2024-04-26T04:22:40.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.924+0000 7fa14e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:40.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.924+0000 7fa14e7fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:40.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.924+0000 7fa14e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:40.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.924+0000 7fa14f7fe640 10 monclient(hunting): handle_auth_done global_id 5247 payload 824 2024-04-26T04:22:40.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.924+0000 7fa14f7fe640 1 -- 172.21.15.63:0/1039766072 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fa15009fe60 msgr2=0x7fa1501ce5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:40.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.924+0000 7fa14f7fe640 1 --2- 172.21.15.63:0/1039766072 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fa15009fe60 0x7fa1501ce5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:40.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.924+0000 7fa14f7fe640 1 -- 172.21.15.63:0/1039766072 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1500a0240 msgr2=0x7fa15013cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:40.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.924+0000 7fa14f7fe640 1 --2- 172.21.15.63:0/1039766072 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1500a0240 0x7fa15013cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:40.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.924+0000 7fa14f7fe640 10 monclient: _finish_hunting 0 2024-04-26T04:22:40.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.924+0000 7fa14f7fe640 1 monclient: found mon.noname-a 2024-04-26T04:22:40.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.924+0000 7fa14f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:22:40.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.924+0000 7fa14f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:22:40.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.924+0000 7fa14f7fe640 1 -- 172.21.15.63:0/1039766072 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa15013e050 con 0x7fa1500a2f70 2024-04-26T04:22:40.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.924+0000 7fa14effd640 1 --2- 172.21.15.63:0/1039766072 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fa15009fe60 0x7fa1501ce5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T04:22:40.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.924+0000 7fa14e7fc640 1 --2- 172.21.15.63:0/1039766072 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1500a0240 0x7fa15013cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T04:22:40.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.928+0000 7fa14f7fe640 1 -- 172.21.15.63:0/1039766072 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa1500a2f70 msgr2=0x7fa15013d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:22:40.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.928+0000 7fa14f7fe640 1 --2- 172.21.15.63:0/1039766072 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa1500a2f70 0x7fa15013d530 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fa14800d800 tx=0x7fa14800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:22:40.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.928+0000 7fa1477fe640 1 -- 172.21.15.63:0/1039766072 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa14800b8d0 con 0x7fa1500a2f70 2024-04-26T04:22:40.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.928+0000 7fa1477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:40.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.928+0000 7fa1477fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:22:40.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.928+0000 7fa1477fe640 10 monclient: dump: 2024-04-26T04:22:40.942 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:40.942 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:40.942 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:40.942 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:40.942 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:40.943 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:40.943 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:40.943 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:40.943 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:40.943 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:40.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.928+0000 7fa1477fe640 10 monclient: _finish_auth 0 2024-04-26T04:22:40.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.928+0000 7fa1477fe640 10 monclient: _check_auth_tickets 2024-04-26T04:22:40.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.928+0000 7fa1477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:22:40.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.928+0000 7fa1477fe640 1 -- 172.21.15.63:0/1039766072 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa148019070 con 0x7fa1500a2f70 2024-04-26T04:22:40.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.928+0000 7fa1477fe640 10 monclient: handle_config config(0 keys) 2024-04-26T04:22:40.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.928+0000 7fa1477fe640 1 -- 172.21.15.63:0/1039766072 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa14800bcb0 con 0x7fa1500a2f70 2024-04-26T04:22:40.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.928+0000 7fa155307640 5 monclient: authenticate success, global_id 5247 2024-04-26T04:22:40.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.928+0000 7fa155307640 10 monclient: _renew_subs 2024-04-26T04:22:40.944 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.928+0000 7fa155307640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:22:40.944 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.928+0000 7fa155307640 1 -- 172.21.15.63:0/1039766072 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1500a38f0 con 0x7fa1500a2f70 2024-04-26T04:22:40.944 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.928+0000 7fa14cff9640 4 set_mon_vals no callback set 2024-04-26T04:22:40.944 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.928+0000 7fa1477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:40.944 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.928+0000 7fa1477fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:22:40.944 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.928+0000 7fa1477fe640 10 monclient: dump: 2024-04-26T04:22:40.944 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:40.944 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:40.944 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:40.944 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:40.945 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:40.945 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:40.945 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:40.945 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:40.945 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:40.945 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:40.945 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.928+0000 7fa155307640 10 monclient: _renew_subs 2024-04-26T04:22:40.945 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.928+0000 7fa155307640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:22:40.945 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.928+0000 7fa155307640 1 -- 172.21.15.63:0/1039766072 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1501395a0 con 0x7fa1500a2f70 2024-04-26T04:22:40.945 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.928+0000 7fa155307640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:22:40.945 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.928+0000 7fa155307640 1 --2- 172.21.15.63:0/1039766072 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa15013b610 0x7fa150137df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:40.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.928+0000 7fa155307640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:40.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.928+0000 7fa155307640 1 -- 172.21.15.63:0/1039766072 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa15013b9f0 con 0x7fa15013b610 2024-04-26T04:22:40.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.928+0000 7fa1477fe640 1 -- 172.21.15.63:0/1039766072 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fa14801e460 con 0x7fa1500a2f70 2024-04-26T04:22:40.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.928+0000 7fa14effd640 1 --2- 172.21.15.63:0/1039766072 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa15013b610 0x7fa150137df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:40.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.928+0000 7fa1477fe640 1 --2- 172.21.15.63:0/1039766072 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fa130035f10 0x7fa1300383d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:40.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.928+0000 7fa14effd640 10 monclient: get_auth_request con 0x7fa15013b610 auth_method 0 2024-04-26T04:22:40.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.928+0000 7fa14effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:40.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.928+0000 7fa14effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:40.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.928+0000 7fa14effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:40.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.928+0000 7fa1477fe640 1 -- 172.21.15.63:0/1039766072 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(214..214 src has 1..214) ==== 4738+0+0 (secure 0 0 0) 0x7fa1480492a0 con 0x7fa1500a2f70 2024-04-26T04:22:40.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.928+0000 7fa14effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:22:40.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.928+0000 7fa14effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:22:40.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.928+0000 7fa14effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:22:40.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.928+0000 7fa14e7fc640 1 --2- 172.21.15.63:0/1039766072 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fa130035f10 0x7fa1300383d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:40.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.928+0000 7fa14e7fc640 10 monclient: get_auth_request con 0x7fa130035f10 auth_method 0 2024-04-26T04:22:40.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.928+0000 7fa14effd640 10 monclient(hunting): handle_auth_done global_id 5254 payload 824 2024-04-26T04:22:40.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.928+0000 7fa14effd640 1 -- 172.21.15.63:0/1039766072 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa150062e10 con 0x7fa15013b610 2024-04-26T04:22:40.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.932+0000 7fa14effd640 1 -- 172.21.15.63:0/1039766072 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa15013b610 msgr2=0x7fa150137df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:22:40.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.932+0000 7fa14effd640 1 --2- 172.21.15.63:0/1039766072 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa15013b610 0x7fa150137df0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fa1380072a0 tx=0x7fa138007760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:22:40.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.932+0000 7fa14e7fc640 1 --2- 172.21.15.63:0/1039766072 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fa130035f10 0x7fa1300383d0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fa1500a7010 tx=0x7fa1400a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:22:40.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.936+0000 7fa1477fe640 1 -- 172.21.15.63:0/1039766072 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa13800fce0 con 0x7fa15013b610 2024-04-26T04:22:40.949 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.936+0000 7fa1477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:40.949 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.936+0000 7fa1477fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T04:22:40.949 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.936+0000 7fa1477fe640 10 monclient: dump: 2024-04-26T04:22:40.949 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:40.949 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:40.949 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:40.949 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:40.949 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:40.949 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:40.950 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:40.950 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:40.950 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:40.950 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:40.950 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.940+0000 7fa1477fe640 1 -- 172.21.15.63:0/1039766072 <== mon.1 v2:172.21.15.186:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa13800fce0 con 0x7fa15013b610 2024-04-26T04:22:40.950 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.940+0000 7fa1477fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:22:40.950 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.940+0000 7fa1477fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:22:40.950 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.940+0000 7fa1477fe640 1 -- 172.21.15.63:0/1039766072 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa15013b610 msgr2=0x7fa150137df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:40.950 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.940+0000 7fa1477fe640 1 --2- 172.21.15.63:0/1039766072 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa15013b610 0x7fa150137df0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fa1380072a0 tx=0x7fa138007760 comp rx=0 tx=0).stop 2024-04-26T04:22:40.950 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.940+0000 7fa1477fe640 1 -- 172.21.15.63:0/1039766072 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa15013b610 msgr2=0x7fa150137df0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:22:40.950 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.940+0000 7fa1477fe640 1 --2- 172.21.15.63:0/1039766072 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa15013b610 0x7fa150137df0 secure :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fa1380072a0 tx=0x7fa138007760 comp rx=0 tx=0).stop 2024-04-26T04:22:40.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.948+0000 7fa155307640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T04:22:40.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.948+0000 7fa155307640 1 --2- 172.21.15.63:0/1039766072 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa150138330 0x7fa150138710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:40.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.948+0000 7fa155307640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:40.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.948+0000 7fa155307640 1 -- 172.21.15.63:0/1039766072 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa150138c50 con 0x7fa150138330 2024-04-26T04:22:40.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.948+0000 7fa14effd640 1 --2- 172.21.15.63:0/1039766072 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa150138330 0x7fa150138710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:40.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.948+0000 7fa14effd640 10 monclient: get_auth_request con 0x7fa150138330 auth_method 0 2024-04-26T04:22:40.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.948+0000 7fa14effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:40.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.948+0000 7fa14effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:40.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.948+0000 7fa14effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:40.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.948+0000 7fa14effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:22:40.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.948+0000 7fa14effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:22:40.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.948+0000 7fa14effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:22:40.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.948+0000 7fa14effd640 10 monclient(hunting): handle_auth_done global_id 5257 payload 824 2024-04-26T04:22:40.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.948+0000 7fa14effd640 1 -- 172.21.15.63:0/1039766072 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa150139e80 con 0x7fa150138330 2024-04-26T04:22:40.953 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[30502]: starting. 2024-04-26T04:22:40.953 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[30502]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-30500 2024-04-26T04:22:40.953 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[30502]: shutting down. 2024-04-26T04:22:40.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.948+0000 7fa14effd640 1 -- 172.21.15.63:0/1039766072 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa150138330 msgr2=0x7fa150138710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:22:40.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.948+0000 7fa14effd640 1 --2- 172.21.15.63:0/1039766072 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa150138330 0x7fa150138710 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fa1380072a0 tx=0x7fa138004950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:22:40.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.952+0000 7fa1477fe640 1 -- 172.21.15.63:0/1039766072 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa138006030 con 0x7fa150138330 2024-04-26T04:22:40.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.952+0000 7fa1477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:40.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.952+0000 7fa1477fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T04:22:40.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.952+0000 7fa1477fe640 10 monclient: dump: 2024-04-26T04:22:40.954 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:40.954 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:40.954 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:40.955 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:40.955 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:40.955 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:40.955 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:40.955 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:40.955 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:40.955 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:40.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.952+0000 7fa1477fe640 1 -- 172.21.15.63:0/1039766072 <== mon.1 v2:172.21.15.186:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4891 (secure 0 0 0) 0x7fa138006030 con 0x7fa150138330 2024-04-26T04:22:40.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.952+0000 7fa1477fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T04:22:40.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.952+0000 7fa1477fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:22:40.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.952+0000 7fa1477fe640 1 -- 172.21.15.63:0/1039766072 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa150138330 msgr2=0x7fa150138710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:40.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.952+0000 7fa1477fe640 1 --2- 172.21.15.63:0/1039766072 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa150138330 0x7fa150138710 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fa1380072a0 tx=0x7fa138004950 comp rx=0 tx=0).stop 2024-04-26T04:22:40.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.952+0000 7fa1477fe640 1 -- 172.21.15.63:0/1039766072 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa150138330 msgr2=0x7fa150138710 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:22:40.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.952+0000 7fa1477fe640 1 --2- 172.21.15.63:0/1039766072 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa150138330 0x7fa150138710 secure :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fa1380072a0 tx=0x7fa138004950 comp rx=0 tx=0).stop 2024-04-26T04:22:40.957 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-26T04:22:40.957 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b", 2024-04-26T04:22:40.957 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 1, 2024-04-26T04:22:40.957 INFO:teuthology.orchestra.run.smithi063.stdout: "state": "peon", 2024-04-26T04:22:40.957 INFO:teuthology.orchestra.run.smithi063.stdout: "election_epoch": 20, 2024-04-26T04:22:40.957 INFO:teuthology.orchestra.run.smithi063.stdout: "uptime": 252062, 2024-04-26T04:22:40.957 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum": [ 2024-04-26T04:22:40.957 INFO:teuthology.orchestra.run.smithi063.stdout: 0, 2024-04-26T04:22:40.957 INFO:teuthology.orchestra.run.smithi063.stdout: 1, 2024-04-26T04:22:40.957 INFO:teuthology.orchestra.run.smithi063.stdout: 2 2024-04-26T04:22:40.957 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:22:40.957 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_age": 8, 2024-04-26T04:22:40.958 INFO:teuthology.orchestra.run.smithi063.stdout: "features": { 2024-04-26T04:22:40.958 INFO:teuthology.orchestra.run.smithi063.stdout: "required_con": "2450521708007866372", 2024-04-26T04:22:40.958 INFO:teuthology.orchestra.run.smithi063.stdout: "required_mon": [ 2024-04-26T04:22:40.958 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:22:40.958 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:22:40.958 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:22:40.958 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:22:40.958 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:22:40.958 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:22:40.958 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:22:40.958 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:22:40.958 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:22:40.958 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:22:40.959 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:22:40.959 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:22:40.959 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_con": "4540701547738038271", 2024-04-26T04:22:40.959 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_mon": [ 2024-04-26T04:22:40.959 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:22:40.959 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:22:40.959 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:22:40.959 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:22:40.959 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:22:40.959 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:22:40.959 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:22:40.959 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:22:40.959 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:22:40.960 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:22:40.960 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:22:40.960 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:22:40.960 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:40.960 INFO:teuthology.orchestra.run.smithi063.stdout: "outside_quorum": [], 2024-04-26T04:22:40.960 INFO:teuthology.orchestra.run.smithi063.stdout: "extra_probe_peers": [], 2024-04-26T04:22:40.960 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_provider": [], 2024-04-26T04:22:40.960 INFO:teuthology.orchestra.run.smithi063.stdout: "monmap": { 2024-04-26T04:22:40.960 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 1, 2024-04-26T04:22:40.960 INFO:teuthology.orchestra.run.smithi063.stdout: "fsid": "cd1c6a0c-4e4b-4014-89c0-2e535479ef39", 2024-04-26T04:22:40.960 INFO:teuthology.orchestra.run.smithi063.stdout: "modified": "2024-04-26T04:18:03.702339Z", 2024-04-26T04:22:40.960 INFO:teuthology.orchestra.run.smithi063.stdout: "created": "2024-04-26T04:18:03.702339Z", 2024-04-26T04:22:40.960 INFO:teuthology.orchestra.run.smithi063.stdout: "min_mon_release": 19, 2024-04-26T04:22:40.961 INFO:teuthology.orchestra.run.smithi063.stdout: "min_mon_release_name": "squid", 2024-04-26T04:22:40.961 INFO:teuthology.orchestra.run.smithi063.stdout: "election_strategy": 1, 2024-04-26T04:22:40.961 INFO:teuthology.orchestra.run.smithi063.stdout: "disallowed_leaders: ": "", 2024-04-26T04:22:40.961 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": false, 2024-04-26T04:22:40.961 INFO:teuthology.orchestra.run.smithi063.stdout: "tiebreaker_mon": "", 2024-04-26T04:22:40.961 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_ranks: ": "", 2024-04-26T04:22:40.961 INFO:teuthology.orchestra.run.smithi063.stdout: "features": { 2024-04-26T04:22:40.961 INFO:teuthology.orchestra.run.smithi063.stdout: "persistent": [ 2024-04-26T04:22:40.961 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:22:40.961 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:22:40.961 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:22:40.961 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:22:40.961 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:22:40.961 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:22:40.962 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:22:40.962 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:22:40.962 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:22:40.962 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:22:40.962 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:22:40.962 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:22:40.962 INFO:teuthology.orchestra.run.smithi063.stdout: "optional": [] 2024-04-26T04:22:40.962 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:40.962 INFO:teuthology.orchestra.run.smithi063.stdout: "mons": [ 2024-04-26T04:22:40.962 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:40.962 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 0, 2024-04-26T04:22:40.962 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a", 2024-04-26T04:22:40.962 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:22:40.963 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:22:40.963 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:40.963 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:22:40.963 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:3300", 2024-04-26T04:22:40.963 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:22:40.963 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:40.963 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:40.963 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:22:40.963 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6789", 2024-04-26T04:22:40.963 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:22:40.963 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:22:40.963 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:22:40.963 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:40.963 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6789/0", 2024-04-26T04:22:40.964 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6789/0", 2024-04-26T04:22:40.964 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:22:40.964 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:22:40.964 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:22:40.964 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:40.964 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:40.964 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 1, 2024-04-26T04:22:40.964 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b", 2024-04-26T04:22:40.964 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:22:40.964 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:22:40.964 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:40.964 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:22:40.964 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:3300", 2024-04-26T04:22:40.964 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:22:40.964 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:40.965 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:40.965 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:22:40.965 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:6789", 2024-04-26T04:22:40.965 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:22:40.965 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:22:40.965 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:22:40.965 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:40.965 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:6789/0", 2024-04-26T04:22:40.965 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.186:6789/0", 2024-04-26T04:22:40.965 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:22:40.965 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:22:40.965 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:22:40.965 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:40.965 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:40.965 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 2, 2024-04-26T04:22:40.966 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "c", 2024-04-26T04:22:40.966 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:22:40.966 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:22:40.966 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:40.966 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:22:40.966 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:3301", 2024-04-26T04:22:40.966 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:22:40.966 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:40.966 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:40.966 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:22:40.966 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6790", 2024-04-26T04:22:40.966 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:22:40.966 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:22:40.967 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:22:40.967 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:40.967 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6790/0", 2024-04-26T04:22:40.967 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6790/0", 2024-04-26T04:22:40.967 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:22:40.967 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:22:40.967 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:22:40.967 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:22:40.967 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:22:40.967 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:40.967 INFO:teuthology.orchestra.run.smithi063.stdout: "feature_map": { 2024-04-26T04:22:40.967 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": [ 2024-04-26T04:22:40.967 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:40.967 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:22:40.967 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:22:40.968 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 1 2024-04-26T04:22:40.968 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:22:40.968 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:22:40.968 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": [ 2024-04-26T04:22:40.968 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:40.968 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:22:40.968 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:22:40.968 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 3 2024-04-26T04:22:40.968 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:22:40.968 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:22:40.968 INFO:teuthology.orchestra.run.smithi063.stdout: "client": [ 2024-04-26T04:22:40.968 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:40.968 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:22:40.968 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:22:40.969 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 5 2024-04-26T04:22:40.969 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:22:40.969 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:22:40.969 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": [ 2024-04-26T04:22:40.969 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:40.969 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:22:40.969 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:22:40.969 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 1 2024-04-26T04:22:40.969 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:22:40.969 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:22:40.969 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:40.969 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": false 2024-04-26T04:22:40.969 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-26T04:22:40.970 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.952+0000 7fa155307640 1 -- 172.21.15.63:0/1039766072 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fa130035f10 msgr2=0x7fa1300383d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:40.970 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.952+0000 7fa155307640 1 --2- 172.21.15.63:0/1039766072 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fa130035f10 0x7fa1300383d0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fa1500a7010 tx=0x7fa1400a7070 comp rx=0 tx=0).stop 2024-04-26T04:22:40.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.952+0000 7fa155307640 10 monclient: shutdown 2024-04-26T04:22:40.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.952+0000 7fa155307640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:22:40.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.952+0000 7fa155307640 1 -- 172.21.15.63:0/1039766072 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa1500a2f70 msgr2=0x7fa15013d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:40.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.952+0000 7fa155307640 1 --2- 172.21.15.63:0/1039766072 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa1500a2f70 0x7fa15013d530 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fa14800d800 tx=0x7fa14800dcc0 comp rx=0 tx=0).stop 2024-04-26T04:22:40.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.952+0000 7fa14f7fe640 1 -- 172.21.15.63:0/1039766072 reap_dead start 2024-04-26T04:22:40.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.952+0000 7fa14f7fe640 1 -- 172.21.15.63:0/1039766072 reap_dead start 2024-04-26T04:22:40.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.952+0000 7fa155307640 1 -- 172.21.15.63:0/1039766072 shutdown_connections 2024-04-26T04:22:40.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.952+0000 7fa155307640 1 -- 172.21.15.63:0/1039766072 >> 172.21.15.63:0/1039766072 conn(0x7fa1500a5fb0 msgr2=0x7fa1500a19b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:22:40.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.952+0000 7fa155307640 1 -- 172.21.15.63:0/1039766072 shutdown_connections 2024-04-26T04:22:40.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:40.952+0000 7fa155307640 1 -- 172.21.15.63:0/1039766072 wait complete. 2024-04-26T04:22:40.972 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T04:22:40.973 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-04-26T04:22:40.973 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-04-26T04:22:40.973 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-04-26T04:22:40.973 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-04-26T04:22:40.973 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-04-26T04:22:40.974 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T04:22:40.988 INFO:tasks.ceph.mon.b:Stopped 2024-04-26T04:22:40.988 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-04-26T04:22:40.988 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T04:22:41.098 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.096+0000 7f2f75f99640 10 monclient: get_monmap_and_config 2024-04-26T04:22:41.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.096+0000 7f2f75f99640 10 monclient: build_initial_monmap 2024-04-26T04:22:41.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.096+0000 7f2f75f99640 10 monclient: monmap: 2024-04-26T04:22:41.099 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:22:41.099 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:41.099 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:22:41.098774+0000 2024-04-26T04:22:41.099 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:22:41.098774+0000 2024-04-26T04:22:41.099 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:22:41.099 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:41.099 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:22:41.099 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:22:41.099 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:22:41.099 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:22:41.100 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:22:41.100 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:41.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.096+0000 7f2f75f99640 1 Processor -- start 2024-04-26T04:22:41.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.096+0000 7f2f75f99640 1 -- start start 2024-04-26T04:22:41.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.096+0000 7f2f75f99640 10 monclient: init 2024-04-26T04:22:41.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.096+0000 7f2f75f99640 10 monclient: _reopen_session rank -1 2024-04-26T04:22:41.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.096+0000 7f2f75f99640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-04-26T04:22:41.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.096+0000 7f2f75f99640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f2f7009b3b0 0x7f2f7009e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:41.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.096+0000 7f2f75f99640 10 monclient(hunting): picked mon.noname-a con 0x7f2f7009b3b0 addr v2:172.21.15.63:3300/0 2024-04-26T04:22:41.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.096+0000 7f2f75f99640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f2f700a0bb0 0x7f2f700a0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:41.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.096+0000 7f2f75f99640 10 monclient(hunting): picked mon.noname-c con 0x7f2f700a0bb0 addr v2:172.21.15.186:3300/0 2024-04-26T04:22:41.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.096+0000 7f2f75f99640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2f700a1500 addr v1:172.21.15.63:6789/0 2024-04-26T04:22:41.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.096+0000 7f2f75f99640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f7009dde0 con 0x7f2f700a1500 2024-04-26T04:22:41.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.096+0000 7f2f75f99640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:41.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.096+0000 7f2f75f99640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7f2f7009c100 con 0x7f2f7009b3b0 2024-04-26T04:22:41.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.096+0000 7f2f75f99640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:41.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.096+0000 7f2f75f99640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f2f7009b790 con 0x7f2f700a0bb0 2024-04-26T04:22:41.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.096+0000 7f2f75f99640 10 monclient(hunting): _renew_subs 2024-04-26T04:22:41.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.096+0000 7f2f75f99640 10 monclient(hunting): authenticate will time out at 1192.806391s 2024-04-26T04:22:41.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.096+0000 7f2f6f7fe640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f2f7009b3b0 0x7f2f7009e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:41.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.096+0000 7f2f6effd640 1 -- >> v2:172.21.15.186:3300/0 conn(0x7f2f700a0bb0 msgr2=0x7f2f700a0fc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.186:3300/0 2024-04-26T04:22:41.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.096+0000 7f2f6ffff640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f2f700a1500 0x7f2f7013b340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:35726/0 (socket says 172.21.15.63:35726) 2024-04-26T04:22:41.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.096+0000 7f2f6ffff640 1 -- 172.21.15.63:0/343915083 learned_addr learned my addr 172.21.15.63:0/343915083 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-04-26T04:22:41.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.096+0000 7f2f6effd640 1 --2- 172.21.15.63:0/343915083 >> v2:172.21.15.186:3300/0 conn(0x7f2f700a0bb0 0x7f2f700a0fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:22:41.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.096+0000 7f2f6f7fe640 10 monclient(hunting): get_auth_request con 0x7f2f7009b3b0 auth_method 0 2024-04-26T04:22:41.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.096+0000 7f2f6f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:41.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.096+0000 7f2f6f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:41.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.096+0000 7f2f6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:41.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.096+0000 7f2f6f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:22:41.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.096+0000 7f2f6f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:22:41.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.096+0000 7f2f6f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:22:41.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.096+0000 7f2f6f7fe640 10 monclient(hunting): handle_auth_done global_id 5265 payload 274 2024-04-26T04:22:41.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.096+0000 7f2f6f7fe640 1 -- 172.21.15.63:0/343915083 >> v2:172.21.15.186:3300/0 conn(0x7f2f700a0bb0 msgr2=0x7f2f700a0fc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T04:22:41.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.096+0000 7f2f6f7fe640 1 --2- 172.21.15.63:0/343915083 >> v2:172.21.15.186:3300/0 conn(0x7f2f700a0bb0 0x7f2f700a0fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:41.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.096+0000 7f2f6f7fe640 1 -- 172.21.15.63:0/343915083 >> v1:172.21.15.63:6789/0 conn(0x7f2f700a1500 legacy=0x7f2f7013b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:22:41.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.096+0000 7f2f6f7fe640 10 monclient: _finish_hunting 0 2024-04-26T04:22:41.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.096+0000 7f2f6f7fe640 1 monclient: found mon.noname-a 2024-04-26T04:22:41.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.096+0000 7f2f6f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:22:41.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.096+0000 7f2f6f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:22:41.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.096+0000 7f2f6f7fe640 1 -- 172.21.15.63:0/343915083 --> v2:172.21.15.63:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f7009c9b0 con 0x7f2f7009b3b0 2024-04-26T04:22:41.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.100+0000 7f2f6f7fe640 1 -- 172.21.15.63:0/343915083 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f7009b3b0 msgr2=0x7f2f7009e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:22:41.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.100+0000 7f2f6f7fe640 1 --2- 172.21.15.63:0/343915083 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f7009b3b0 0x7f2f7009e4e0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f2f7006d8e0 tx=0x7f2f6401c120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c107cca2ca020e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:22:41.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.100+0000 7f2f6e7fc640 1 -- 172.21.15.63:0/343915083 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f640b0070 con 0x7f2f7009b3b0 2024-04-26T04:22:41.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.100+0000 7f2f6e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:41.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.100+0000 7f2f6e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:22:41.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.100+0000 7f2f6e7fc640 10 monclient: dump: 2024-04-26T04:22:41.106 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:41.106 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:41.106 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:41.106 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:41.106 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:41.107 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:41.107 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:41.107 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:41.107 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:41.107 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:41.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.100+0000 7f2f6e7fc640 10 monclient: _finish_auth 0 2024-04-26T04:22:41.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.100+0000 7f2f6e7fc640 10 monclient: _check_auth_tickets 2024-04-26T04:22:41.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.100+0000 7f2f6e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:22:41.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.100+0000 7f2f6e7fc640 1 -- 172.21.15.63:0/343915083 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f640ab600 con 0x7f2f7009b3b0 2024-04-26T04:22:41.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.100+0000 7f2f6e7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T04:22:41.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.100+0000 7f2f6e7fc640 1 -- 172.21.15.63:0/343915083 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f640ab9d0 con 0x7f2f7009b3b0 2024-04-26T04:22:41.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.100+0000 7f2f6e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:41.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.100+0000 7f2f6e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:22:41.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.100+0000 7f2f6e7fc640 10 monclient: dump: 2024-04-26T04:22:41.107 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:41.108 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:41.108 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:41.108 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:41.108 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:41.108 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:41.108 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:41.108 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:41.108 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:41.108 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:41.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.100+0000 7f2f75f99640 5 monclient: authenticate success, global_id 5265 2024-04-26T04:22:41.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.100+0000 7f2f75f99640 10 monclient: get_monmap_and_config success 2024-04-26T04:22:41.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.100+0000 7f2f75f99640 4 set_mon_vals no callback set 2024-04-26T04:22:41.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.104+0000 7f2f75f99640 10 monclient: shutdown 2024-04-26T04:22:41.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.104+0000 7f2f75f99640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:22:41.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.104+0000 7f2f75f99640 1 -- 172.21.15.63:0/343915083 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f7009b3b0 msgr2=0x7f2f7009e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:41.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.104+0000 7f2f75f99640 1 --2- 172.21.15.63:0/343915083 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f7009b3b0 0x7f2f7009e4e0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f2f7006d8e0 tx=0x7f2f6401c120 comp rx=0 tx=0).stop 2024-04-26T04:22:41.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.104+0000 7f2f75f99640 1 -- 172.21.15.63:0/343915083 shutdown_connections 2024-04-26T04:22:41.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.104+0000 7f2f75f99640 1 --2- 172.21.15.63:0/343915083 >> v2:172.21.15.186:3300/0 conn(0x7f2f700a0bb0 0x7f2f700a0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:41.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.104+0000 7f2f75f99640 1 --2- 172.21.15.63:0/343915083 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f7009b3b0 0x7f2f7009e4e0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:41.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.104+0000 7f2f75f99640 1 -- 172.21.15.63:0/343915083 >> 172.21.15.63:0/343915083 conn(0x7f2f700a1e70 msgr2=0x7f2f700a2280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:22:41.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.104+0000 7f2f75f99640 1 -- 172.21.15.63:0/343915083 shutdown_connections 2024-04-26T04:22:41.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.104+0000 7f2f75f99640 1 -- 172.21.15.63:0/343915083 wait complete. 2024-04-26T04:22:41.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.104+0000 7f2f75f99640 10 monclient: build_initial_monmap 2024-04-26T04:22:41.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.104+0000 7f2f75f99640 10 monclient: monmap: 2024-04-26T04:22:41.113 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:22:41.113 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:22:41.113 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:22:41.113 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:22:41.113 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:22:41.114 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:41.114 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:22:41.114 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:22:41.114 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:22:41.114 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:41.114 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.104+0000 7f2f75f99640 1 Processor -- start 2024-04-26T04:22:41.114 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.104+0000 7f2f75f99640 1 -- start start 2024-04-26T04:22:41.114 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.104+0000 7f2f75f99640 10 monclient: init 2024-04-26T04:22:41.114 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.104+0000 7f2f75f99640 10 monclient: _reopen_session rank -1 2024-04-26T04:22:41.114 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.104+0000 7f2f75f99640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T04:22:41.114 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.104+0000 7f2f75f99640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f2f7009b3b0 0x7f2f7008d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:41.114 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.104+0000 7f2f75f99640 10 monclient(hunting): picked mon.noname-c con 0x7f2f7009b3b0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:22:41.114 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.104+0000 7f2f75f99640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f700a0bb0 0x7f2f7008d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:41.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.104+0000 7f2f75f99640 10 monclient(hunting): picked mon.noname-a con 0x7f2f700a0bb0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:22:41.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.104+0000 7f2f75f99640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f700a1500 0x7f2f7008dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:41.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.104+0000 7f2f75f99640 10 monclient(hunting): picked mon.noname-b con 0x7f2f700a1500 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:22:41.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.104+0000 7f2f75f99640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:41.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.104+0000 7f2f75f99640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f2f7013f490 con 0x7f2f700a0bb0 2024-04-26T04:22:41.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.104+0000 7f2f75f99640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:41.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.104+0000 7f2f75f99640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2f7009eae0 con 0x7f2f700a1500 2024-04-26T04:22:41.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.104+0000 7f2f75f99640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:41.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.104+0000 7f2f75f99640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f2f7013f610 con 0x7f2f7009b3b0 2024-04-26T04:22:41.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.104+0000 7f2f75f99640 10 monclient(hunting): _renew_subs 2024-04-26T04:22:41.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.104+0000 7f2f75f99640 10 monclient(hunting): authenticate will time out at 1192.813719s 2024-04-26T04:22:41.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.104+0000 7f2f6f7fe640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f2f7009b3b0 0x7f2f7008d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:41.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.104+0000 7f2f6effd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f700a0bb0 0x7f2f7008d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:41.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.104+0000 7f2f6f7fe640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f2f7009b3b0 0x7f2f7008d270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:60260/0 (socket says 172.21.15.63:60260) 2024-04-26T04:22:41.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.104+0000 7f2f6f7fe640 1 -- 172.21.15.63:0/3311190810 learned_addr learned my addr 172.21.15.63:0/3311190810 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:22:41.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.104+0000 7f2f6effd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f700a0bb0 0x7f2f7008d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:46936/0 (socket says 172.21.15.63:46936) 2024-04-26T04:22:41.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.104+0000 7f2f6f7fe640 10 monclient(hunting): get_auth_request con 0x7f2f7009b3b0 auth_method 0 2024-04-26T04:22:41.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.104+0000 7f2f6f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:41.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.104+0000 7f2f6f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:41.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.104+0000 7f2f6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:41.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.104+0000 7f2f6effd640 10 monclient(hunting): get_auth_request con 0x7f2f700a0bb0 auth_method 0 2024-04-26T04:22:41.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.104+0000 7f2f6effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:41.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.104+0000 7f2f6effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:41.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.104+0000 7f2f6effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:41.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.104+0000 7f2f6ffff640 1 -- 172.21.15.63:0/3311190810 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f700a1500 msgr2=0x7f2f7008dcf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.186:3300/0 2024-04-26T04:22:41.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.104+0000 7f2f6ffff640 1 --2- 172.21.15.63:0/3311190810 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f700a1500 0x7f2f7008dcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:22:41.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.104+0000 7f2f6effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:22:41.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.104+0000 7f2f6effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:22:41.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.104+0000 7f2f6effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:22:41.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.104+0000 7f2f6f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:22:41.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.104+0000 7f2f6f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:22:41.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.104+0000 7f2f6f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:22:41.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.104+0000 7f2f6effd640 10 monclient(hunting): handle_auth_done global_id 5268 payload 824 2024-04-26T04:22:41.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.104+0000 7f2f6effd640 1 -- 172.21.15.63:0/3311190810 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f2f7009b3b0 msgr2=0x7f2f7008d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:41.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.104+0000 7f2f6effd640 1 --2- 172.21.15.63:0/3311190810 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f2f7009b3b0 0x7f2f7008d270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:41.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.104+0000 7f2f6effd640 1 -- 172.21.15.63:0/3311190810 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f700a1500 msgr2=0x7f2f7008dcf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T04:22:41.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.104+0000 7f2f6effd640 1 --2- 172.21.15.63:0/3311190810 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f700a1500 0x7f2f7008dcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:41.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.104+0000 7f2f6effd640 10 monclient: _finish_hunting 0 2024-04-26T04:22:41.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.104+0000 7f2f6effd640 1 monclient: found mon.noname-a 2024-04-26T04:22:41.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.104+0000 7f2f6effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:22:41.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.104+0000 7f2f6effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:22:41.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.104+0000 7f2f6effd640 1 -- 172.21.15.63:0/3311190810 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f7008f790 con 0x7f2f700a0bb0 2024-04-26T04:22:41.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.104+0000 7f2f6f7fe640 1 --2- 172.21.15.63:0/3311190810 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f2f7009b3b0 0x7f2f7008d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T04:22:41.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.108+0000 7f2f6effd640 1 -- 172.21.15.63:0/3311190810 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f700a0bb0 msgr2=0x7f2f7008d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:22:41.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.108+0000 7f2f6effd640 1 --2- 172.21.15.63:0/3311190810 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f700a0bb0 0x7f2f7008d7b0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f2f6000b470 tx=0x7f2f6000b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:22:41.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.108+0000 7f2f57fff640 1 -- 172.21.15.63:0/3311190810 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f60005070 con 0x7f2f700a0bb0 2024-04-26T04:22:41.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.108+0000 7f2f57fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:41.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.108+0000 7f2f57fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:22:41.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.108+0000 7f2f57fff640 10 monclient: dump: 2024-04-26T04:22:41.119 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:41.119 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:41.119 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:41.119 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:41.119 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:41.119 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:41.119 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:41.119 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:41.120 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:41.120 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:41.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.108+0000 7f2f57fff640 10 monclient: _finish_auth 0 2024-04-26T04:22:41.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.108+0000 7f2f57fff640 10 monclient: _check_auth_tickets 2024-04-26T04:22:41.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.108+0000 7f2f57fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:22:41.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.108+0000 7f2f57fff640 1 -- 172.21.15.63:0/3311190810 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f60015070 con 0x7f2f700a0bb0 2024-04-26T04:22:41.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.108+0000 7f2f75f99640 5 monclient: authenticate success, global_id 5268 2024-04-26T04:22:41.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.108+0000 7f2f75f99640 10 monclient: _renew_subs 2024-04-26T04:22:41.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.108+0000 7f2f75f99640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:22:41.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.108+0000 7f2f75f99640 1 -- 172.21.15.63:0/3311190810 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f70072920 con 0x7f2f700a0bb0 2024-04-26T04:22:41.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.108+0000 7f2f57fff640 10 monclient: handle_config config(0 keys) 2024-04-26T04:22:41.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.108+0000 7f2f57fff640 1 -- 172.21.15.63:0/3311190810 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f60005450 con 0x7f2f700a0bb0 2024-04-26T04:22:41.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.108+0000 7f2f57fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:41.121 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.108+0000 7f2f57fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:22:41.121 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.108+0000 7f2f57fff640 10 monclient: dump: 2024-04-26T04:22:41.121 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:41.121 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:41.121 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:41.121 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:41.121 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:41.121 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:41.121 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:41.121 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:41.122 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:41.122 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:41.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.108+0000 7f2f75f99640 10 monclient: _renew_subs 2024-04-26T04:22:41.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.108+0000 7f2f6dffb640 4 set_mon_vals no callback set 2024-04-26T04:22:41.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.108+0000 7f2f75f99640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:22:41.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.108+0000 7f2f75f99640 1 -- 172.21.15.63:0/3311190810 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f70098d20 con 0x7f2f700a0bb0 2024-04-26T04:22:41.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.108+0000 7f2f75f99640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:22:41.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.108+0000 7f2f75f99640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:22:41.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.108+0000 7f2f75f99640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:22:41.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.108+0000 7f2f75f99640 1 -- 172.21.15.63:0/3311190810 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f70090700 con 0x7f2f700a0bb0 2024-04-26T04:22:41.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.108+0000 7f2f57fff640 1 -- 172.21.15.63:0/3311190810 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f2f6000f070 con 0x7f2f700a0bb0 2024-04-26T04:22:41.122 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.108+0000 7f2f57fff640 1 --2- 172.21.15.63:0/3311190810 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f2f5c035e70 0x7f2f5c038330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:41.123 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.108+0000 7f2f57fff640 1 -- 172.21.15.63:0/3311190810 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(215..215 src has 1..215) ==== 4346+0+0 (secure 0 0 0) 0x7f2f600484e0 con 0x7f2f700a0bb0 2024-04-26T04:22:41.123 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.108+0000 7f2f6f7fe640 1 --2- 172.21.15.63:0/3311190810 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f2f5c035e70 0x7f2f5c038330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:41.123 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.108+0000 7f2f6f7fe640 10 monclient: get_auth_request con 0x7f2f5c035e70 auth_method 0 2024-04-26T04:22:41.123 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.112+0000 7f2f6f7fe640 1 --2- 172.21.15.63:0/3311190810 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f2f5c035e70 0x7f2f5c038330 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f2f64009740 tx=0x7f2f64008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:22:41.123 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.112+0000 7f2f57fff640 1 -- 172.21.15.63:0/3311190810 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f2f6000d040 con 0x7f2f700a0bb0 2024-04-26T04:22:41.124 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.112+0000 7f2f57fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:22:41.124 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.112+0000 7f2f57fff640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:22:41.330 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.328+0000 7f2f75f99640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T04:22:41.330 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.328+0000 7f2f75f99640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T04:22:41.330 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.328+0000 7f2f75f99640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:22:41.331 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.328+0000 7f2f75f99640 1 -- 172.21.15.63:0/3311190810 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f2f70062e10 con 0x7f2f700a0bb0 2024-04-26T04:22:41.331 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.328+0000 7f2f57fff640 1 -- 172.21.15.63:0/3311190810 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f2f60018520 con 0x7f2f700a0bb0 2024-04-26T04:22:41.331 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.328+0000 7f2f57fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T04:22:41.331 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.328+0000 7f2f57fff640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:22:41.331 INFO:teuthology.orchestra.run.smithi063.stdout:{"election_epoch":20,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cd1c6a0c-4e4b-4014-89c0-2e535479ef39","modified":"2024-04-26T04:18:03.702339Z","created":"2024-04-26T04:18:03.702339Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3300","nonce":0},{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:3300","nonce":0},{"type":"v1","addr":"172.21.15.186:6789","nonce":0}]},"addr":"172.21.15.186:6789/0","public_addr":"172.21.15.186:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3301","nonce":0},{"type":"v1","addr":"172.21.15.63:6790","nonce":0}]},"addr":"172.21.15.63:6790/0","public_addr":"172.21.15.63:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T04:22:41.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.328+0000 7f2f75f99640 1 -- 172.21.15.63:0/3311190810 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f2f5c035e70 msgr2=0x7f2f5c038330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:41.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.328+0000 7f2f75f99640 1 --2- 172.21.15.63:0/3311190810 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f2f5c035e70 0x7f2f5c038330 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f2f64009740 tx=0x7f2f64008040 comp rx=0 tx=0).stop 2024-04-26T04:22:41.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.328+0000 7f2f75f99640 10 monclient: shutdown 2024-04-26T04:22:41.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.328+0000 7f2f75f99640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:22:41.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.328+0000 7f2f75f99640 1 -- 172.21.15.63:0/3311190810 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f700a0bb0 msgr2=0x7f2f7008d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:41.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.328+0000 7f2f75f99640 1 --2- 172.21.15.63:0/3311190810 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f700a0bb0 0x7f2f7008d7b0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f2f6000b470 tx=0x7f2f6000b930 comp rx=0 tx=0).stop 2024-04-26T04:22:41.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.328+0000 7f2f75f99640 1 -- 172.21.15.63:0/3311190810 shutdown_connections 2024-04-26T04:22:41.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.328+0000 7f2f75f99640 1 --2- 172.21.15.63:0/3311190810 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f2f5c035e70 0x7f2f5c038330 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:41.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.328+0000 7f2f75f99640 1 --2- 172.21.15.63:0/3311190810 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f700a1500 0x7f2f7008dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:41.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.328+0000 7f2f75f99640 1 --2- 172.21.15.63:0/3311190810 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2f700a0bb0 0x7f2f7008d7b0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:41.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.328+0000 7f2f75f99640 1 --2- 172.21.15.63:0/3311190810 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f2f7009b3b0 0x7f2f7008d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:41.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.328+0000 7f2f75f99640 1 -- 172.21.15.63:0/3311190810 >> 172.21.15.63:0/3311190810 conn(0x7f2f700a1e70 msgr2=0x7f2f70138440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:22:41.334 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.328+0000 7f2f75f99640 1 -- 172.21.15.63:0/3311190810 shutdown_connections 2024-04-26T04:22:41.334 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:41.328+0000 7f2f75f99640 1 -- 172.21.15.63:0/3311190810 wait complete. 2024-04-26T04:22:44.349 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T04:22:44.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.484+0000 7fb12425c640 10 monclient: get_monmap_and_config 2024-04-26T04:22:44.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.484+0000 7fb12425c640 10 monclient: build_initial_monmap 2024-04-26T04:22:44.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.484+0000 7fb12425c640 10 monclient: monmap: 2024-04-26T04:22:44.487 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:22:44.487 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:44.487 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:22:44.486919+0000 2024-04-26T04:22:44.487 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:22:44.486919+0000 2024-04-26T04:22:44.487 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:22:44.487 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:44.487 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:22:44.487 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:22:44.488 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:22:44.488 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:22:44.488 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:22:44.488 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:44.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.484+0000 7fb12425c640 1 Processor -- start 2024-04-26T04:22:44.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.484+0000 7fb12425c640 1 -- start start 2024-04-26T04:22:44.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.484+0000 7fb12425c640 10 monclient: init 2024-04-26T04:22:44.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.484+0000 7fb12425c640 10 monclient: _reopen_session rank -1 2024-04-26T04:22:44.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.484+0000 7fb12425c640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-04-26T04:22:44.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.484+0000 7fb12425c640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb11c0a2cd0 0x7fb11c0a30b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:44.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.484+0000 7fb12425c640 10 monclient(hunting): picked mon.noname-c con 0x7fb11c0a2cd0 addr v2:172.21.15.186:3300/0 2024-04-26T04:22:44.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.484+0000 7fb12425c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb11c0a47d0 addr v1:172.21.15.63:6789/0 2024-04-26T04:22:44.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.484+0000 7fb12425c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb11c0a4c00 addr v1:172.21.15.186:6789/0 2024-04-26T04:22:44.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.484+0000 7fb12425c640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb11c137b00 con 0x7fb11c0a47d0 2024-04-26T04:22:44.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.484+0000 7fb12425c640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb11c0a0fb0 con 0x7fb11c0a4c00 2024-04-26T04:22:44.490 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.484+0000 7fb12425c640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:44.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.484+0000 7fb12425c640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fb11c0a35f0 con 0x7fb11c0a2cd0 2024-04-26T04:22:44.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.484+0000 7fb12425c640 10 monclient(hunting): _renew_subs 2024-04-26T04:22:44.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.484+0000 7fb12425c640 10 monclient(hunting): authenticate will time out at 1196.195010s 2024-04-26T04:22:44.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.484+0000 7fb121ff8640 1 -- >> v2:172.21.15.186:3300/0 conn(0x7fb11c0a2cd0 msgr2=0x7fb11c0a30b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.186:3300/0 2024-04-26T04:22:44.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.484+0000 7fb1227f9640 1 -- >> v1:172.21.15.186:6789/0 conn(0x7fb11c0a4c00 legacy=0x7fb11c1373f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.186:6789/0 2024-04-26T04:22:44.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.484+0000 7fb121ff8640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb11c0a2cd0 0x7fb11c0a30b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:22:44.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.484+0000 7fb1217f7640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7fb11c0a47d0 0x7fb11c136ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:35732/0 (socket says 172.21.15.63:35732) 2024-04-26T04:22:44.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.484+0000 7fb1217f7640 1 -- 172.21.15.63:0/2187815650 learned_addr learned my addr 172.21.15.63:0/2187815650 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-04-26T04:22:44.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb1217f7640 1 -- 172.21.15.63:0/2187815650 >> v1:172.21.15.63:6789/0 conn(0x7fb11c0a47d0 legacy=0x7fb11c136ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:22:44.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb120ff6640 1 -- 172.21.15.63:0/2187815650 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1618639872 0 0) 0x7fb11c137b00 con 0x7fb11c0a47d0 2024-04-26T04:22:44.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb120ff6640 10 monclient(hunting): _negotiate 2024-04-26T04:22:44.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb120ff6640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:44.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb120ff6640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:44.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb120ff6640 10 monclient(hunting): my global_id is 5271 2024-04-26T04:22:44.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb120ff6640 1 -- 172.21.15.63:0/2187815650 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb104003530 con 0x7fb11c0a47d0 2024-04-26T04:22:44.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb120ff6640 1 -- 172.21.15.63:0/2187815650 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3335442157 0 0) 0x7fb104003530 con 0x7fb11c0a47d0 2024-04-26T04:22:44.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb120ff6640 1 -- 172.21.15.63:0/2187815650 >> v2:172.21.15.186:3300/0 conn(0x7fb11c0a2cd0 msgr2=0x7fb11c0a30b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T04:22:44.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb120ff6640 1 --2- 172.21.15.63:0/2187815650 >> v2:172.21.15.186:3300/0 conn(0x7fb11c0a2cd0 0x7fb11c0a30b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:44.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb120ff6640 1 -- 172.21.15.63:0/2187815650 >> v1:172.21.15.186:6789/0 conn(0x7fb11c0a4c00 legacy=0x7fb11c1373f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T04:22:44.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb120ff6640 10 monclient: _finish_hunting 0 2024-04-26T04:22:44.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb120ff6640 1 monclient: found mon.noname-a-legacy 2024-04-26T04:22:44.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb120ff6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:22:44.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb120ff6640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.63:6789/0 2024-04-26T04:22:44.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb120ff6640 1 -- 172.21.15.63:0/2187815650 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb11c0a5380 con 0x7fb11c0a47d0 2024-04-26T04:22:44.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb120ff6640 10 monclient: _finish_auth 0 2024-04-26T04:22:44.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb120ff6640 10 monclient: _check_auth_tickets 2024-04-26T04:22:44.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb120ff6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:22:44.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb12425c640 5 monclient: authenticate success, global_id 5271 2024-04-26T04:22:44.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb120ff6640 1 -- 172.21.15.63:0/2187815650 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2612289768 0 0) 0x7fb110001930 con 0x7fb11c0a47d0 2024-04-26T04:22:44.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb12425c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T04:22:44.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb120ff6640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:44.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb120ff6640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-26T04:22:44.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb120ff6640 10 monclient: dump: 2024-04-26T04:22:44.496 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:44.496 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:44.496 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:44.496 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:44.496 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:44.497 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:44.497 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:44.497 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:44.497 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:44.497 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:44.497 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb120ff6640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] but i'm connected to v1:172.21.15.63:6789/0, reconnecting 2024-04-26T04:22:44.497 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb120ff6640 10 monclient: _reopen_session rank -1 2024-04-26T04:22:44.497 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb120ff6640 1 -- 172.21.15.63:0/2187815650 >> v1:172.21.15.63:6789/0 conn(0x7fb11c0a47d0 legacy=0x7fb11c136ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:44.497 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb120ff6640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T04:22:44.497 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb120ff6640 1 --2- 172.21.15.63:0/2187815650 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fb104003aa0 0x7fb104003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:44.497 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb120ff6640 10 monclient(hunting): picked mon.c con 0x7fb104003aa0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:22:44.497 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb120ff6640 1 --2- 172.21.15.63:0/2187815650 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb104004810 0x7fb104004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:44.497 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb120ff6640 10 monclient(hunting): picked mon.b con 0x7fb104004810 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:22:44.497 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb120ff6640 1 --2- 172.21.15.63:0/2187815650 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb1040053d0 0x7fb10400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:44.497 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb120ff6640 10 monclient(hunting): picked mon.a con 0x7fb1040053d0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:22:44.498 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb120ff6640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:44.498 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb120ff6640 1 -- 172.21.15.63:0/2187815650 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fb11c0a35f0 con 0x7fb1040053d0 2024-04-26T04:22:44.498 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb120ff6640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:44.498 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb120ff6640 1 -- 172.21.15.63:0/2187815650 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb10400c2b0 con 0x7fb104004810 2024-04-26T04:22:44.498 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb120ff6640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:44.498 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb120ff6640 1 -- 172.21.15.63:0/2187815650 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7fb10400c4b0 con 0x7fb104003aa0 2024-04-26T04:22:44.498 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb120ff6640 10 monclient(hunting): _renew_subs 2024-04-26T04:22:44.498 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb120ff6640 10 monclient(hunting): _finish_auth 0 2024-04-26T04:22:44.498 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb12425c640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-04-26T04:22:44.498 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb1227f9640 1 --2- 172.21.15.63:0/2187815650 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb1040053d0 0x7fb10400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:44.498 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb1217f7640 1 -- 172.21.15.63:0/2187815650 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb104004810 msgr2=0x7fb104004c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.186:3300/0 2024-04-26T04:22:44.498 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb1227f9640 10 monclient(hunting): get_auth_request con 0x7fb1040053d0 auth_method 0 2024-04-26T04:22:44.498 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb1227f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:44.499 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb1227f9640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:44.499 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb1217f7640 1 --2- 172.21.15.63:0/2187815650 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb104004810 0x7fb104004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:22:44.499 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb1227f9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:22:44.499 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb121ff8640 1 --2- 172.21.15.63:0/2187815650 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fb104003aa0 0x7fb104003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:44.499 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb121ff8640 10 monclient(hunting): get_auth_request con 0x7fb104003aa0 auth_method 0 2024-04-26T04:22:44.499 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb121ff8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:44.499 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb121ff8640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:44.499 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb121ff8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:22:44.499 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb1227f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:22:44.499 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb1227f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:22:44.499 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb1227f9640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T04:22:44.499 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb121ff8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:22:44.499 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb121ff8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:22:44.499 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb121ff8640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T04:22:44.500 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb1227f9640 10 monclient(hunting): handle_auth_done global_id 5271 payload 293 2024-04-26T04:22:44.500 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb1227f9640 1 -- 172.21.15.63:0/2187815650 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fb104003aa0 msgr2=0x7fb104003ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:22:44.500 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb1227f9640 1 --2- 172.21.15.63:0/2187815650 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fb104003aa0 0x7fb104003ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:44.500 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb1227f9640 1 -- 172.21.15.63:0/2187815650 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb104004810 msgr2=0x7fb104004c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T04:22:44.500 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb1227f9640 1 --2- 172.21.15.63:0/2187815650 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb104004810 0x7fb104004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:44.500 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb1227f9640 10 monclient: _finish_hunting 0 2024-04-26T04:22:44.500 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb1227f9640 1 monclient: found mon.a 2024-04-26T04:22:44.500 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb1227f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:22:44.500 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb1227f9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:22:44.500 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb1227f9640 1 -- 172.21.15.63:0/2187815650 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb104003530 con 0x7fb1040053d0 2024-04-26T04:22:44.500 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb1227f9640 10 monclient: _finish_auth 0 2024-04-26T04:22:44.500 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb1227f9640 10 monclient: _check_auth_tickets 2024-04-26T04:22:44.500 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.488+0000 7fb1227f9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:22:44.500 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.492+0000 7fb1227f9640 1 -- 172.21.15.63:0/2187815650 reap_dead start 2024-04-26T04:22:44.506 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.492+0000 7fb1227f9640 1 -- 172.21.15.63:0/2187815650 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb1040053d0 msgr2=0x7fb10400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:22:44.506 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.492+0000 7fb1227f9640 1 --2- 172.21.15.63:0/2187815650 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb1040053d0 0x7fb10400b9d0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fb114009e30 tx=0x7fb11401c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=394c37351048d264 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:22:44.506 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.492+0000 7fb120ff6640 1 -- 172.21.15.63:0/2187815650 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb114009a30 con 0x7fb1040053d0 2024-04-26T04:22:44.506 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.492+0000 7fb120ff6640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:44.506 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.492+0000 7fb120ff6640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:22:44.506 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.492+0000 7fb120ff6640 10 monclient: dump: 2024-04-26T04:22:44.506 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:44.506 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:44.506 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:44.506 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:44.506 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:44.506 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:44.507 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:44.507 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:44.507 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:44.507 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:44.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.492+0000 7fb120ff6640 1 -- 172.21.15.63:0/2187815650 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1140a9070 con 0x7fb1040053d0 2024-04-26T04:22:44.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.492+0000 7fb12425c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T04:22:44.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.492+0000 7fb120ff6640 10 monclient: handle_config config(0 keys) 2024-04-26T04:22:44.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.492+0000 7fb12425c640 10 monclient: get_monmap_and_config success 2024-04-26T04:22:44.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.492+0000 7fb12425c640 4 set_mon_vals no callback set 2024-04-26T04:22:44.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.492+0000 7fb120ff6640 1 -- 172.21.15.63:0/2187815650 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1140a5b90 con 0x7fb1040053d0 2024-04-26T04:22:44.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.492+0000 7fb120ff6640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:44.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.492+0000 7fb120ff6640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:22:44.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.492+0000 7fb120ff6640 10 monclient: dump: 2024-04-26T04:22:44.507 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:44.507 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:44.508 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:44.508 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:44.508 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:44.508 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:44.508 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:44.508 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:44.508 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:44.508 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:44.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.492+0000 7fb12425c640 10 monclient: shutdown 2024-04-26T04:22:44.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.492+0000 7fb12425c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:22:44.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.492+0000 7fb12425c640 1 -- 172.21.15.63:0/2187815650 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb1040053d0 msgr2=0x7fb10400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:44.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.492+0000 7fb12425c640 1 --2- 172.21.15.63:0/2187815650 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb1040053d0 0x7fb10400b9d0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fb114009e30 tx=0x7fb11401c910 comp rx=0 tx=0).stop 2024-04-26T04:22:44.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.492+0000 7fb12425c640 1 -- 172.21.15.63:0/2187815650 shutdown_connections 2024-04-26T04:22:44.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.492+0000 7fb12425c640 1 --2- 172.21.15.63:0/2187815650 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb1040053d0 0x7fb10400b9d0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:44.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.492+0000 7fb12425c640 1 -- 172.21.15.63:0/2187815650 >> 172.21.15.63:0/2187815650 conn(0x7fb11c0a5f40 msgr2=0x7fb11c0a6320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:22:44.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.492+0000 7fb12425c640 1 -- 172.21.15.63:0/2187815650 shutdown_connections 2024-04-26T04:22:44.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.492+0000 7fb12425c640 1 -- 172.21.15.63:0/2187815650 wait complete. 2024-04-26T04:22:44.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.496+0000 7fb12425c640 10 monclient: build_initial_monmap 2024-04-26T04:22:44.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.496+0000 7fb12425c640 10 monclient: monmap: 2024-04-26T04:22:44.509 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:22:44.509 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:22:44.509 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:22:44.509 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:22:44.509 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:22:44.509 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:44.509 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:22:44.509 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:22:44.509 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:22:44.510 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:44.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.496+0000 7fb12425c640 1 Processor -- start 2024-04-26T04:22:44.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.496+0000 7fb12425c640 1 -- start start 2024-04-26T04:22:44.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.496+0000 7fb12425c640 10 monclient: init 2024-04-26T04:22:44.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.496+0000 7fb12425c640 10 monclient: _reopen_session rank -1 2024-04-26T04:22:44.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.496+0000 7fb12425c640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T04:22:44.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.496+0000 7fb12425c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1040053d0 0x7fb11c1cc1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:44.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.496+0000 7fb12425c640 10 monclient(hunting): picked mon.noname-b con 0x7fb1040053d0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:22:44.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.496+0000 7fb12425c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb11c123e00 0x7fb11c1cc710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:44.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.496+0000 7fb12425c640 10 monclient(hunting): picked mon.noname-a con 0x7fb11c123e00 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:22:44.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.496+0000 7fb12425c640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fb11c11fa90 0x7fb11c11ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:44.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.496+0000 7fb12425c640 10 monclient(hunting): picked mon.noname-c con 0x7fb11c11fa90 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:22:44.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.496+0000 7fb12425c640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:44.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.496+0000 7fb12425c640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fb11c0a6e10 con 0x7fb11c123e00 2024-04-26T04:22:44.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.496+0000 7fb12425c640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:44.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.496+0000 7fb12425c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb11c0a6c90 con 0x7fb1040053d0 2024-04-26T04:22:44.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.496+0000 7fb12425c640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:44.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.496+0000 7fb12425c640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7fb11c0a30b0 con 0x7fb11c11fa90 2024-04-26T04:22:44.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.496+0000 7fb12425c640 10 monclient(hunting): _renew_subs 2024-04-26T04:22:44.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.496+0000 7fb12425c640 10 monclient(hunting): authenticate will time out at 1196.205092s 2024-04-26T04:22:44.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.496+0000 7fb1217f7640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb11c123e00 0x7fb11c1cc710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:44.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.496+0000 7fb121ff8640 1 -- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1040053d0 msgr2=0x7fb11c1cc1d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.186:3300/0 2024-04-26T04:22:44.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.496+0000 7fb1227f9640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fb11c11fa90 0x7fb11c11ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:44.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.496+0000 7fb121ff8640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1040053d0 0x7fb11c1cc1d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:22:44.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.496+0000 7fb1217f7640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb11c123e00 0x7fb11c1cc710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:46948/0 (socket says 172.21.15.63:46948) 2024-04-26T04:22:44.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.496+0000 7fb1217f7640 1 -- 172.21.15.63:0/283146460 learned_addr learned my addr 172.21.15.63:0/283146460 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:22:44.512 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.496+0000 7fb1217f7640 10 monclient(hunting): get_auth_request con 0x7fb11c123e00 auth_method 0 2024-04-26T04:22:44.512 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.496+0000 7fb1217f7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:44.512 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.496+0000 7fb1217f7640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:44.512 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.496+0000 7fb1217f7640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:44.512 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.496+0000 7fb1227f9640 10 monclient(hunting): get_auth_request con 0x7fb11c11fa90 auth_method 0 2024-04-26T04:22:44.512 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.496+0000 7fb1227f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:44.512 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.496+0000 7fb1227f9640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:44.512 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.496+0000 7fb1227f9640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:44.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.496+0000 7fb1217f7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:22:44.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.496+0000 7fb1217f7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:22:44.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.496+0000 7fb1217f7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:22:44.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.496+0000 7fb1227f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:22:44.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.496+0000 7fb1227f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:22:44.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.496+0000 7fb1227f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:22:44.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.496+0000 7fb1217f7640 10 monclient(hunting): handle_auth_done global_id 5274 payload 824 2024-04-26T04:22:44.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.496+0000 7fb1217f7640 1 -- 172.21.15.63:0/283146460 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fb11c11fa90 msgr2=0x7fb11c11ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:44.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.496+0000 7fb1217f7640 1 --2- 172.21.15.63:0/283146460 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fb11c11fa90 0x7fb11c11ff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:44.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.496+0000 7fb1217f7640 1 -- 172.21.15.63:0/283146460 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1040053d0 msgr2=0x7fb11c1cc1d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T04:22:44.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.496+0000 7fb1217f7640 1 --2- 172.21.15.63:0/283146460 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1040053d0 0x7fb11c1cc1d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:44.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.496+0000 7fb1217f7640 10 monclient: _finish_hunting 0 2024-04-26T04:22:44.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.496+0000 7fb1217f7640 1 monclient: found mon.noname-a 2024-04-26T04:22:44.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.496+0000 7fb1217f7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:22:44.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.496+0000 7fb1217f7640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:22:44.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.496+0000 7fb1217f7640 1 -- 172.21.15.63:0/283146460 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb11c123120 con 0x7fb11c123e00 2024-04-26T04:22:44.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.496+0000 7fb1227f9640 1 --2- 172.21.15.63:0/283146460 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fb11c11fa90 0x7fb11c11ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T04:22:44.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.496+0000 7fb1217f7640 1 -- 172.21.15.63:0/283146460 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb11c123e00 msgr2=0x7fb11c1cc710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:22:44.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.496+0000 7fb1217f7640 1 --2- 172.21.15.63:0/283146460 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb11c123e00 0x7fb11c1cc710 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fb11000c790 tx=0x7fb11000cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:22:44.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.500+0000 7fb1217f7640 1 --2- 172.21.15.63:0/283146460 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb11c123e00 0x7fb11c1cc710 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fb11000c790 tx=0x7fb11000cc50 comp rx=0 tx=0).queue_received will delay after 1.71411e+09 on 0x7fb1100047f0 config(0 keys) 2024-04-26T04:22:44.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.500+0000 7fb11a7fc640 1 -- 172.21.15.63:0/283146460 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb11000cea0 con 0x7fb11c123e00 2024-04-26T04:22:44.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.500+0000 7fb11a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:44.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.500+0000 7fb11a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:22:44.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.500+0000 7fb11a7fc640 10 monclient: dump: 2024-04-26T04:22:44.514 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:44.514 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:44.515 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:44.515 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:44.515 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:44.515 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:44.515 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:44.515 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:44.515 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:44.515 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:44.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.500+0000 7fb11a7fc640 10 monclient: _finish_auth 0 2024-04-26T04:22:44.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.500+0000 7fb11a7fc640 10 monclient: _check_auth_tickets 2024-04-26T04:22:44.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.500+0000 7fb11a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:22:44.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.500+0000 7fb11a7fc640 1 -- 172.21.15.63:0/283146460 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1100047f0 con 0x7fb11c123e00 2024-04-26T04:22:44.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.500+0000 7fb12425c640 5 monclient: authenticate success, global_id 5274 2024-04-26T04:22:44.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.500+0000 7fb12425c640 10 monclient: _renew_subs 2024-04-26T04:22:44.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.500+0000 7fb12425c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:22:44.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.500+0000 7fb12425c640 1 -- 172.21.15.63:0/283146460 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb11c1215b0 con 0x7fb11c123e00 2024-04-26T04:22:44.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.500+0000 7fb11a7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T04:22:44.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.500+0000 7fb11bfff640 4 set_mon_vals no callback set 2024-04-26T04:22:44.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.500+0000 7fb12425c640 10 monclient: _renew_subs 2024-04-26T04:22:44.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.500+0000 7fb12425c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:22:44.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.500+0000 7fb12425c640 1 -- 172.21.15.63:0/283146460 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb11c122b60 con 0x7fb11c123e00 2024-04-26T04:22:44.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.500+0000 7fb12425c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:22:44.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.500+0000 7fb12425c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:22:44.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.500+0000 7fb12425c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:22:44.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.500+0000 7fb12425c640 1 -- 172.21.15.63:0/283146460 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb11c122d60 con 0x7fb11c123e00 2024-04-26T04:22:44.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.500+0000 7fb11a7fc640 1 -- 172.21.15.63:0/283146460 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb110004be0 con 0x7fb11c123e00 2024-04-26T04:22:44.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.500+0000 7fb11a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:44.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.500+0000 7fb11a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:22:44.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.500+0000 7fb11a7fc640 10 monclient: dump: 2024-04-26T04:22:44.517 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:44.517 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:44.517 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:44.517 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:44.517 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:44.517 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:44.517 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:44.517 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:44.517 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:44.517 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:44.517 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.500+0000 7fb11a7fc640 1 -- 172.21.15.63:0/283146460 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fb110011070 con 0x7fb11c123e00 2024-04-26T04:22:44.517 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.500+0000 7fb11a7fc640 1 --2- 172.21.15.63:0/283146460 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fb108035e70 0x7fb108038330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:44.517 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.500+0000 7fb121ff8640 1 --2- 172.21.15.63:0/283146460 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fb108035e70 0x7fb108038330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:44.517 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.500+0000 7fb121ff8640 10 monclient: get_auth_request con 0x7fb108035e70 auth_method 0 2024-04-26T04:22:44.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.504+0000 7fb121ff8640 1 --2- 172.21.15.63:0/283146460 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fb108035e70 0x7fb108038330 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb10c003910 tx=0x7fb10c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:22:44.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:44.504+0000 7fb11a7fc640 1 -- 172.21.15.63:0/283146460 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(215..215 src has 1..215) ==== 4346+0+0 (secure 0 0 0) 0x7fb110048a60 con 0x7fb11c123e00 2024-04-26T04:22:45.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:45.328+0000 7fb11a7fc640 1 -- 172.21.15.63:0/283146460 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fb11c122d60 con 0x7fb11c123e00 2024-04-26T04:22:45.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:45.328+0000 7fb11a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:22:45.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:45.328+0000 7fb11a7fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:22:45.566 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:45.564+0000 7fb12425c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T04:22:45.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:45.564+0000 7fb12425c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T04:22:45.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:45.564+0000 7fb12425c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:22:45.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:45.564+0000 7fb12425c640 1 -- 172.21.15.63:0/283146460 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb11c062e10 con 0x7fb11c123e00 2024-04-26T04:22:45.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:45.564+0000 7fb11a7fc640 1 -- 172.21.15.63:0/283146460 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7fb11004ae20 con 0x7fb11c123e00 2024-04-26T04:22:45.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:45.564+0000 7fb11a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T04:22:45.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:45.564+0000 7fb11a7fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:22:45.567 INFO:teuthology.orchestra.run.smithi063.stdout:{"election_epoch":20,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cd1c6a0c-4e4b-4014-89c0-2e535479ef39","modified":"2024-04-26T04:18:03.702339Z","created":"2024-04-26T04:18:03.702339Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3300","nonce":0},{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:3300","nonce":0},{"type":"v1","addr":"172.21.15.186:6789","nonce":0}]},"addr":"172.21.15.186:6789/0","public_addr":"172.21.15.186:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3301","nonce":0},{"type":"v1","addr":"172.21.15.63:6790","nonce":0}]},"addr":"172.21.15.63:6790/0","public_addr":"172.21.15.63:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T04:22:45.569 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:45.564+0000 7fb12425c640 1 -- 172.21.15.63:0/283146460 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fb108035e70 msgr2=0x7fb108038330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:45.569 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:45.564+0000 7fb12425c640 1 --2- 172.21.15.63:0/283146460 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fb108035e70 0x7fb108038330 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb10c003910 tx=0x7fb10c008040 comp rx=0 tx=0).stop 2024-04-26T04:22:45.569 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:45.564+0000 7fb12425c640 10 monclient: shutdown 2024-04-26T04:22:45.569 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:45.564+0000 7fb12425c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:22:45.569 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:45.564+0000 7fb12425c640 1 -- 172.21.15.63:0/283146460 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb11c123e00 msgr2=0x7fb11c1cc710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:45.569 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:45.564+0000 7fb12425c640 1 --2- 172.21.15.63:0/283146460 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb11c123e00 0x7fb11c1cc710 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fb11000c790 tx=0x7fb11000cc50 comp rx=0 tx=0).stop 2024-04-26T04:22:45.569 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:45.564+0000 7fb12425c640 1 -- 172.21.15.63:0/283146460 shutdown_connections 2024-04-26T04:22:45.569 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:45.564+0000 7fb12425c640 1 --2- 172.21.15.63:0/283146460 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fb108035e70 0x7fb108038330 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:45.569 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:45.564+0000 7fb12425c640 1 --2- 172.21.15.63:0/283146460 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fb11c11fa90 0x7fb11c11ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:45.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:45.564+0000 7fb12425c640 1 --2- 172.21.15.63:0/283146460 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb11c123e00 0x7fb11c1cc710 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:45.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:45.564+0000 7fb12425c640 1 --2- 172.21.15.63:0/283146460 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1040053d0 0x7fb11c1cc1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:45.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:45.564+0000 7fb12425c640 1 -- 172.21.15.63:0/283146460 >> 172.21.15.63:0/283146460 conn(0x7fb11c0a5f40 msgr2=0x7fb11c136370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:22:45.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:45.564+0000 7fb12425c640 1 -- 172.21.15.63:0/283146460 shutdown_connections 2024-04-26T04:22:45.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:45.564+0000 7fb12425c640 1 -- 172.21.15.63:0/283146460 wait complete. 2024-04-26T04:22:48.588 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T04:22:48.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.716+0000 7f392528c640 10 monclient: get_monmap_and_config 2024-04-26T04:22:48.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.716+0000 7f392528c640 10 monclient: build_initial_monmap 2024-04-26T04:22:48.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.716+0000 7f392528c640 10 monclient: monmap: 2024-04-26T04:22:48.720 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:22:48.720 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:48.720 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:22:48.719487+0000 2024-04-26T04:22:48.720 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:22:48.719487+0000 2024-04-26T04:22:48.720 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:22:48.720 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:48.720 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:22:48.720 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:22:48.720 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:22:48.720 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:22:48.721 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:22:48.721 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:48.723 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.716+0000 7f392528c640 1 Processor -- start 2024-04-26T04:22:48.723 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.716+0000 7f392528c640 1 -- start start 2024-04-26T04:22:48.723 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.716+0000 7f392528c640 10 monclient: init 2024-04-26T04:22:48.724 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.716+0000 7f392528c640 10 monclient: _reopen_session rank -1 2024-04-26T04:22:48.724 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.716+0000 7f392528c640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-04-26T04:22:48.724 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.716+0000 7f392528c640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f392009b3b0 0x7f392009e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:48.724 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.716+0000 7f392528c640 10 monclient(hunting): picked mon.noname-a con 0x7f392009b3b0 addr v2:172.21.15.63:3300/0 2024-04-26T04:22:48.724 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.716+0000 7f392528c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f39200a0bb0 addr v1:172.21.15.186:6789/0 2024-04-26T04:22:48.724 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.716+0000 7f392528c640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f39200a16d0 0x7f392013c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:48.724 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.716+0000 7f392528c640 10 monclient(hunting): picked mon.noname-b con 0x7f39200a16d0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:22:48.724 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.716+0000 7f392528c640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f392009dde0 con 0x7f39200a0bb0 2024-04-26T04:22:48.724 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.716+0000 7f392528c640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:48.724 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.716+0000 7f392528c640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7f392009c100 con 0x7f392009b3b0 2024-04-26T04:22:48.724 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.716+0000 7f392528c640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:48.724 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.716+0000 7f392528c640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f392009b790 con 0x7f39200a16d0 2024-04-26T04:22:48.724 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.716+0000 7f392528c640 10 monclient(hunting): _renew_subs 2024-04-26T04:22:48.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.716+0000 7f392528c640 10 monclient(hunting): authenticate will time out at 1200.427297s 2024-04-26T04:22:48.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.716+0000 7f391effd640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f392009b3b0 0x7f392009e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:48.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.716+0000 7f391f7fe640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f39200a16d0 0x7f392013c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:48.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.716+0000 7f391f7fe640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f39200a16d0 0x7f392013c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:60280/0 (socket says 172.21.15.63:60280) 2024-04-26T04:22:48.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.716+0000 7f391f7fe640 1 -- 172.21.15.63:0/2335345976 learned_addr learned my addr 172.21.15.63:0/2335345976 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:22:48.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.716+0000 7f391effd640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f392009b3b0 0x7f392009e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:46954/0 (socket says 172.21.15.63:46954) 2024-04-26T04:22:48.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.716+0000 7f391f7fe640 10 monclient(hunting): get_auth_request con 0x7f39200a16d0 auth_method 0 2024-04-26T04:22:48.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.716+0000 7f391f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:48.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.716+0000 7f391f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:48.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.716+0000 7f391f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:48.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.716+0000 7f391e7fc640 1 -- 172.21.15.63:0/2335345976 >> v1:172.21.15.186:6789/0 conn(0x7f39200a0bb0 legacy=0x7f39200a0fc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.186:6789/0 2024-04-26T04:22:48.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.716+0000 7f391effd640 10 monclient(hunting): get_auth_request con 0x7f392009b3b0 auth_method 0 2024-04-26T04:22:48.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.716+0000 7f391effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:48.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.716+0000 7f391effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:48.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.716+0000 7f391effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:48.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.716+0000 7f391effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:22:48.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.716+0000 7f391effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:22:48.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.716+0000 7f391effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:22:48.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.716+0000 7f391f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:22:48.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.716+0000 7f391f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:22:48.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.716+0000 7f391f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:22:48.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.716+0000 7f391effd640 10 monclient(hunting): handle_auth_done global_id 5277 payload 274 2024-04-26T04:22:48.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.716+0000 7f391effd640 1 -- 172.21.15.63:0/2335345976 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f39200a16d0 msgr2=0x7f392013c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:22:48.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.716+0000 7f391effd640 1 --2- 172.21.15.63:0/2335345976 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f39200a16d0 0x7f392013c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:48.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.716+0000 7f391effd640 1 -- 172.21.15.63:0/2335345976 >> v1:172.21.15.186:6789/0 conn(0x7f39200a0bb0 legacy=0x7f39200a0fc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T04:22:48.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.716+0000 7f391effd640 10 monclient: _finish_hunting 0 2024-04-26T04:22:48.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.716+0000 7f391effd640 1 monclient: found mon.noname-a 2024-04-26T04:22:48.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.716+0000 7f391effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:22:48.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.716+0000 7f391effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:22:48.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.716+0000 7f391effd640 1 -- 172.21.15.63:0/2335345976 --> v2:172.21.15.63:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f392009c9b0 con 0x7f392009b3b0 2024-04-26T04:22:48.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.716+0000 7f391f7fe640 1 --2- 172.21.15.63:0/2335345976 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f39200a16d0 0x7f392013c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T04:22:48.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f391effd640 1 -- 172.21.15.63:0/2335345976 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f392009b3b0 msgr2=0x7f392009e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:22:48.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f391effd640 1 --2- 172.21.15.63:0/2335345976 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f392009b3b0 0x7f392009e4e0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f3910009dc0 tx=0x7f391001c5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ee4197b26f9f18a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:22:48.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f391dffb640 1 -- 172.21.15.63:0/2335345976 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39100ad070 con 0x7f392009b3b0 2024-04-26T04:22:48.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f391dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:48.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f391dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:22:48.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f391dffb640 10 monclient: dump: 2024-04-26T04:22:48.730 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:48.730 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:48.731 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:48.731 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:48.731 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:48.731 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:48.731 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:48.731 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:48.731 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:48.731 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:48.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f391dffb640 10 monclient: _finish_auth 0 2024-04-26T04:22:48.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f391dffb640 10 monclient: _check_auth_tickets 2024-04-26T04:22:48.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f391dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:22:48.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f391dffb640 1 -- 172.21.15.63:0/2335345976 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39100a9630 con 0x7f392009b3b0 2024-04-26T04:22:48.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f392528c640 5 monclient: authenticate success, global_id 5277 2024-04-26T04:22:48.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f392528c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T04:22:48.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f391dffb640 10 monclient: handle_config config(0 keys) 2024-04-26T04:22:48.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f391dffb640 1 -- 172.21.15.63:0/2335345976 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39100a9a00 con 0x7f392009b3b0 2024-04-26T04:22:48.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f392528c640 10 monclient: get_monmap_and_config success 2024-04-26T04:22:48.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f392528c640 4 set_mon_vals no callback set 2024-04-26T04:22:48.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f391dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:48.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f391dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:22:48.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f391dffb640 10 monclient: dump: 2024-04-26T04:22:48.732 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:48.732 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:48.732 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:48.732 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:48.732 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:48.732 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:48.732 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:48.732 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:48.733 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:48.733 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:48.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f392528c640 10 monclient: shutdown 2024-04-26T04:22:48.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f392528c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:22:48.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f392528c640 1 -- 172.21.15.63:0/2335345976 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f392009b3b0 msgr2=0x7f392009e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:48.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f392528c640 1 --2- 172.21.15.63:0/2335345976 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f392009b3b0 0x7f392009e4e0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f3910009dc0 tx=0x7f391001c5f0 comp rx=0 tx=0).stop 2024-04-26T04:22:48.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f392528c640 1 -- 172.21.15.63:0/2335345976 shutdown_connections 2024-04-26T04:22:48.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f392528c640 1 --2- 172.21.15.63:0/2335345976 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f39200a16d0 0x7f392013c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:48.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f392528c640 1 --2- 172.21.15.63:0/2335345976 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f392009b3b0 0x7f392009e4e0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:48.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f392528c640 1 -- 172.21.15.63:0/2335345976 >> 172.21.15.63:0/2335345976 conn(0x7f39200a1e70 msgr2=0x7f39200a2280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:22:48.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f392528c640 1 -- 172.21.15.63:0/2335345976 shutdown_connections 2024-04-26T04:22:48.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f392528c640 1 -- 172.21.15.63:0/2335345976 wait complete. 2024-04-26T04:22:48.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f392528c640 10 monclient: build_initial_monmap 2024-04-26T04:22:48.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f392528c640 10 monclient: monmap: 2024-04-26T04:22:48.734 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:22:48.734 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:22:48.734 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:22:48.734 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:22:48.734 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:22:48.734 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:48.734 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:22:48.734 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:22:48.734 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:22:48.734 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:48.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f392528c640 1 Processor -- start 2024-04-26T04:22:48.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f392528c640 1 -- start start 2024-04-26T04:22:48.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f392528c640 10 monclient: init 2024-04-26T04:22:48.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f392528c640 10 monclient: _reopen_session rank -1 2024-04-26T04:22:48.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f392528c640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T04:22:48.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f392528c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f392009b3b0 0x7f39201ced40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:48.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f392528c640 10 monclient(hunting): picked mon.noname-a con 0x7f392009b3b0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:22:48.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f392528c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f39200a0bb0 0x7f392009a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:48.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f392528c640 10 monclient(hunting): picked mon.noname-b con 0x7f39200a0bb0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:22:48.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f392528c640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f39200a16d0 0x7f392009ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:48.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f392528c640 10 monclient(hunting): picked mon.noname-c con 0x7f39200a16d0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:22:48.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f392528c640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:48.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f392528c640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f392013f440 con 0x7f392009b3b0 2024-04-26T04:22:48.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f392528c640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:48.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f392528c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f392013f2c0 con 0x7f39200a0bb0 2024-04-26T04:22:48.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f392528c640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:48.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f392528c640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f392013f5c0 con 0x7f39200a16d0 2024-04-26T04:22:48.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f392528c640 10 monclient(hunting): _renew_subs 2024-04-26T04:22:48.736 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f392528c640 10 monclient(hunting): authenticate will time out at 1200.434723s 2024-04-26T04:22:48.736 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f391effd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f392009b3b0 0x7f39201ced40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:48.736 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f391f7fe640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f39200a16d0 0x7f392009ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:48.736 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f391e7fc640 1 -- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f39200a0bb0 msgr2=0x7f392009a650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.186:3300/0 2024-04-26T04:22:48.736 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f391e7fc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f39200a0bb0 0x7f392009a650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:22:48.736 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f391effd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f392009b3b0 0x7f39201ced40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:46970/0 (socket says 172.21.15.63:46970) 2024-04-26T04:22:48.736 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f391f7fe640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f39200a16d0 0x7f392009ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:60290/0 (socket says 172.21.15.63:60290) 2024-04-26T04:22:48.736 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f391effd640 1 -- 172.21.15.63:0/3085599717 learned_addr learned my addr 172.21.15.63:0/3085599717 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:22:48.736 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f391effd640 10 monclient(hunting): get_auth_request con 0x7f392009b3b0 auth_method 0 2024-04-26T04:22:48.736 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f391effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:48.736 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f391effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:48.736 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f391effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:48.736 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f391f7fe640 10 monclient(hunting): get_auth_request con 0x7f39200a16d0 auth_method 0 2024-04-26T04:22:48.736 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f391f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:48.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f391f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:48.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f391f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:48.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f391effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:22:48.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f391effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:22:48.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f391effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:22:48.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f391f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:22:48.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f391f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:22:48.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f391f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:22:48.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f391effd640 10 monclient(hunting): handle_auth_done global_id 5280 payload 824 2024-04-26T04:22:48.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f391effd640 1 -- 172.21.15.63:0/3085599717 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f39200a16d0 msgr2=0x7f392009ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:48.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f391effd640 1 --2- 172.21.15.63:0/3085599717 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f39200a16d0 0x7f392009ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:48.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f391effd640 1 -- 172.21.15.63:0/3085599717 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f39200a0bb0 msgr2=0x7f392009a650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T04:22:48.738 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f391effd640 1 --2- 172.21.15.63:0/3085599717 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f39200a0bb0 0x7f392009a650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:48.738 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f391effd640 10 monclient: _finish_hunting 0 2024-04-26T04:22:48.738 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f391effd640 1 monclient: found mon.noname-a 2024-04-26T04:22:48.738 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f391effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:22:48.738 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f391effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:22:48.738 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f391effd640 1 -- 172.21.15.63:0/3085599717 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3920095150 con 0x7f392009b3b0 2024-04-26T04:22:48.738 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.724+0000 7f391f7fe640 1 --2- 172.21.15.63:0/3085599717 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f39200a16d0 0x7f392009ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T04:22:48.740 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.728+0000 7f391effd640 1 -- 172.21.15.63:0/3085599717 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f392009b3b0 msgr2=0x7f39201ced40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:22:48.740 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.728+0000 7f391effd640 1 --2- 172.21.15.63:0/3085599717 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f392009b3b0 0x7f39201ced40 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f391001aca0 tx=0x7f3910019390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:22:48.740 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.728+0000 7f39177fe640 1 -- 172.21.15.63:0/3085599717 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39100ad040 con 0x7f392009b3b0 2024-04-26T04:22:48.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.728+0000 7f39177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:48.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.728+0000 7f39177fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:22:48.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.728+0000 7f39177fe640 10 monclient: dump: 2024-04-26T04:22:48.741 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:48.741 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:48.741 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:48.741 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:48.741 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:48.741 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:48.741 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:48.741 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:48.741 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:48.741 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:48.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.728+0000 7f39177fe640 10 monclient: _finish_auth 0 2024-04-26T04:22:48.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.728+0000 7f39177fe640 10 monclient: _check_auth_tickets 2024-04-26T04:22:48.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.728+0000 7f39177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:22:48.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.728+0000 7f39177fe640 1 -- 172.21.15.63:0/3085599717 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39100b2870 con 0x7f392009b3b0 2024-04-26T04:22:48.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.728+0000 7f39177fe640 10 monclient: handle_config config(0 keys) 2024-04-26T04:22:48.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.728+0000 7f39177fe640 1 -- 172.21.15.63:0/3085599717 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39100b2c80 con 0x7f392009b3b0 2024-04-26T04:22:48.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.728+0000 7f392528c640 5 monclient: authenticate success, global_id 5280 2024-04-26T04:22:48.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.728+0000 7f392528c640 10 monclient: _renew_subs 2024-04-26T04:22:48.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.728+0000 7f392528c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:22:48.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.728+0000 7f392528c640 1 -- 172.21.15.63:0/3085599717 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f392009cc50 con 0x7f392009b3b0 2024-04-26T04:22:48.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.728+0000 7f39177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:48.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.728+0000 7f39177fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:22:48.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.728+0000 7f391cff9640 4 set_mon_vals no callback set 2024-04-26T04:22:48.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.728+0000 7f39177fe640 10 monclient: dump: 2024-04-26T04:22:48.742 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:48.742 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:48.743 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:48.743 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:48.743 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:48.743 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:48.743 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:48.743 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:48.743 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:48.743 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:48.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.728+0000 7f392528c640 10 monclient: _renew_subs 2024-04-26T04:22:48.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.728+0000 7f392528c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:22:48.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.728+0000 7f392528c640 1 -- 172.21.15.63:0/3085599717 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3920094840 con 0x7f392009b3b0 2024-04-26T04:22:48.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.728+0000 7f39177fe640 1 -- 172.21.15.63:0/3085599717 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f39100b0070 con 0x7f392009b3b0 2024-04-26T04:22:48.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.728+0000 7f392528c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:22:48.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.728+0000 7f392528c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:22:48.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.728+0000 7f392528c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:22:48.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.728+0000 7f392528c640 1 -- 172.21.15.63:0/3085599717 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3920094c20 con 0x7f392009b3b0 2024-04-26T04:22:48.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.728+0000 7f39177fe640 1 --2- 172.21.15.63:0/3085599717 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f3900035e70 0x7f3900038330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:48.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.728+0000 7f39177fe640 1 -- 172.21.15.63:0/3085599717 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(215..215 src has 1..215) ==== 4346+0+0 (secure 0 0 0) 0x7f39100e3a00 con 0x7f392009b3b0 2024-04-26T04:22:48.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.728+0000 7f391e7fc640 1 --2- 172.21.15.63:0/3085599717 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f3900035e70 0x7f3900038330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:48.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.728+0000 7f391e7fc640 10 monclient: get_auth_request con 0x7f3900035e70 auth_method 0 2024-04-26T04:22:48.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.732+0000 7f391e7fc640 1 --2- 172.21.15.63:0/3085599717 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f3900035e70 0x7f3900038330 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f3908006fd0 tx=0x7f3908006d40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:22:48.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.732+0000 7f39177fe640 1 -- 172.21.15.63:0/3085599717 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f3920094c20 con 0x7f392009b3b0 2024-04-26T04:22:48.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.732+0000 7f39177fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:22:48.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.732+0000 7f39177fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:22:48.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.952+0000 7f392528c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T04:22:48.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.952+0000 7f392528c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T04:22:48.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.952+0000 7f392528c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:22:48.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.952+0000 7f392528c640 1 -- 172.21.15.63:0/3085599717 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f3920094f00 con 0x7f392009b3b0 2024-04-26T04:22:48.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.952+0000 7f39177fe640 1 -- 172.21.15.63:0/3085599717 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f3920094f00 con 0x7f392009b3b0 2024-04-26T04:22:48.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.952+0000 7f39177fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T04:22:48.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.952+0000 7f39177fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:22:48.956 INFO:teuthology.orchestra.run.smithi063.stdout:{"election_epoch":20,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cd1c6a0c-4e4b-4014-89c0-2e535479ef39","modified":"2024-04-26T04:18:03.702339Z","created":"2024-04-26T04:18:03.702339Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3300","nonce":0},{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:3300","nonce":0},{"type":"v1","addr":"172.21.15.186:6789","nonce":0}]},"addr":"172.21.15.186:6789/0","public_addr":"172.21.15.186:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3301","nonce":0},{"type":"v1","addr":"172.21.15.63:6790","nonce":0}]},"addr":"172.21.15.63:6790/0","public_addr":"172.21.15.63:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T04:22:48.958 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.956+0000 7f392528c640 1 -- 172.21.15.63:0/3085599717 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f3900035e70 msgr2=0x7f3900038330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:48.958 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.956+0000 7f392528c640 1 --2- 172.21.15.63:0/3085599717 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f3900035e70 0x7f3900038330 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f3908006fd0 tx=0x7f3908006d40 comp rx=0 tx=0).stop 2024-04-26T04:22:48.958 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.956+0000 7f392528c640 10 monclient: shutdown 2024-04-26T04:22:48.958 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.956+0000 7f392528c640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:22:48.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.956+0000 7f392528c640 1 -- 172.21.15.63:0/3085599717 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f392009b3b0 msgr2=0x7f39201ced40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:48.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.956+0000 7f392528c640 1 --2- 172.21.15.63:0/3085599717 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f392009b3b0 0x7f39201ced40 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f391001aca0 tx=0x7f3910019390 comp rx=0 tx=0).stop 2024-04-26T04:22:48.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.956+0000 7f392528c640 1 -- 172.21.15.63:0/3085599717 shutdown_connections 2024-04-26T04:22:48.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.956+0000 7f392528c640 1 --2- 172.21.15.63:0/3085599717 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f3900035e70 0x7f3900038330 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:48.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.956+0000 7f392528c640 1 --2- 172.21.15.63:0/3085599717 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f39200a16d0 0x7f392009ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:48.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.956+0000 7f392528c640 1 --2- 172.21.15.63:0/3085599717 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f39200a0bb0 0x7f392009a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:48.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.956+0000 7f392528c640 1 --2- 172.21.15.63:0/3085599717 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f392009b3b0 0x7f39201ced40 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:48.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.956+0000 7f392528c640 1 -- 172.21.15.63:0/3085599717 >> 172.21.15.63:0/3085599717 conn(0x7f39200a1e70 msgr2=0x7f39200a3060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:22:48.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.956+0000 7f392528c640 1 -- 172.21.15.63:0/3085599717 shutdown_connections 2024-04-26T04:22:48.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:48.956+0000 7f392528c640 1 -- 172.21.15.63:0/3085599717 wait complete. 2024-04-26T04:22:51.974 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T04:22:52.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:52.104+0000 7fda28273640 10 monclient: get_monmap_and_config 2024-04-26T04:22:52.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:52.104+0000 7fda28273640 10 monclient: build_initial_monmap 2024-04-26T04:22:52.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:52.104+0000 7fda28273640 10 monclient: monmap: 2024-04-26T04:22:52.108 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:22:52.108 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:52.108 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:22:52.107522+0000 2024-04-26T04:22:52.108 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:22:52.107522+0000 2024-04-26T04:22:52.108 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:22:52.108 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:52.108 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:22:52.108 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:22:52.108 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:22:52.108 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:22:52.108 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:22:52.108 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:52.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:52.104+0000 7fda28273640 1 Processor -- start 2024-04-26T04:22:52.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:52.104+0000 7fda28273640 1 -- start start 2024-04-26T04:22:52.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:52.104+0000 7fda28273640 10 monclient: init 2024-04-26T04:22:52.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:52.104+0000 7fda28273640 10 monclient: _reopen_session rank -1 2024-04-26T04:22:52.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:52.104+0000 7fda28273640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-04-26T04:22:52.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:52.104+0000 7fda28273640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7fda2009c6a0 0x7fda200996b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:52.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:52.104+0000 7fda28273640 10 monclient(hunting): picked mon.noname-a con 0x7fda2009c6a0 addr v2:172.21.15.63:3300/0 2024-04-26T04:22:52.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:52.104+0000 7fda28273640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fda2009ca80 addr v1:172.21.15.186:6789/0 2024-04-26T04:22:52.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:52.104+0000 7fda28273640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fda20138740 addr v1:172.21.15.63:6789/0 2024-04-26T04:22:52.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:52.104+0000 7fda28273640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda2009d9f0 con 0x7fda20138740 2024-04-26T04:22:52.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:52.104+0000 7fda28273640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda200992d0 con 0x7fda2009ca80 2024-04-26T04:22:52.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:52.104+0000 7fda28273640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:52.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:52.104+0000 7fda28273640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7fda2009c1d0 con 0x7fda2009c6a0 2024-04-26T04:22:52.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:52.104+0000 7fda28273640 10 monclient(hunting): _renew_subs 2024-04-26T04:22:52.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:52.104+0000 7fda28273640 10 monclient(hunting): authenticate will time out at 1203.815625s 2024-04-26T04:22:52.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:52.104+0000 7fda26810640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7fda20138740 0x7fda2013ab30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:34480/0 (socket says 172.21.15.63:34480) 2024-04-26T04:22:52.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:52.104+0000 7fda26810640 1 -- 172.21.15.63:0/3571603115 learned_addr learned my addr 172.21.15.63:0/3571603115 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-04-26T04:22:52.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:52.104+0000 7fda2600f640 1 --2- 172.21.15.63:0/3571603115 >> v2:172.21.15.63:3300/0 conn(0x7fda2009c6a0 0x7fda200996b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:52.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:52.104+0000 7fda2580e640 1 -- 172.21.15.63:0/3571603115 >> v1:172.21.15.186:6789/0 conn(0x7fda2009ca80 legacy=0x7fda2009ce90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.186:6789/0 2024-04-26T04:22:52.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:52.104+0000 7fda2600f640 10 monclient(hunting): get_auth_request con 0x7fda2009c6a0 auth_method 0 2024-04-26T04:22:52.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:52.104+0000 7fda2600f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:52.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:52.104+0000 7fda2600f640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:52.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:52.104+0000 7fda2600f640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:52.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:52.104+0000 7fda2600f640 1 -- 172.21.15.63:0/3571603115 >> v2:172.21.15.63:3300/0 conn(0x7fda2009c6a0 msgr2=0x7fda200996b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T04:22:52.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:52.104+0000 7fda2600f640 1 -- 172.21.15.63:0/3571603115 >> v2:172.21.15.63:3300/0 conn(0x7fda2009c6a0 msgr2=0x7fda200996b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:22:52.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:52.104+0000 7fda2600f640 1 --2- 172.21.15.63:0/3571603115 >> v2:172.21.15.63:3300/0 conn(0x7fda2009c6a0 0x7fda200996b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:22:52.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:52.108+0000 7fda2600f640 1 --2- 172.21.15.63:0/3571603115 >> v2:172.21.15.63:3300/0 conn(0x7fda2009c6a0 0x7fda200996b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:22:52.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:52.108+0000 7fda26810640 1 -- 172.21.15.63:0/3571603115 >> v1:172.21.15.63:6789/0 conn(0x7fda20138740 legacy=0x7fda2013ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:22:52.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:52.308+0000 7fda2580e640 1 -- 172.21.15.63:0/3571603115 >> v1:172.21.15.186:6789/0 conn(0x7fda2009ca80 legacy=0x7fda2009ce90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.186:6789/0 2024-04-26T04:22:52.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:52.312+0000 7fda2600f640 1 --2- 172.21.15.63:0/3571603115 >> v2:172.21.15.63:3300/0 conn(0x7fda2009c6a0 0x7fda200996b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:52.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:52.312+0000 7fda2600f640 10 monclient(hunting): get_auth_request con 0x7fda2009c6a0 auth_method 0 2024-04-26T04:22:52.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:52.312+0000 7fda2600f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:52.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:52.312+0000 7fda2600f640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:52.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:52.312+0000 7fda2600f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:22:52.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:52.312+0000 7fda2600f640 1 -- 172.21.15.63:0/3571603115 >> v2:172.21.15.63:3300/0 conn(0x7fda2009c6a0 msgr2=0x7fda200996b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T04:22:52.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:52.312+0000 7fda2600f640 1 -- 172.21.15.63:0/3571603115 >> v2:172.21.15.63:3300/0 conn(0x7fda2009c6a0 msgr2=0x7fda200996b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:22:52.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:52.312+0000 7fda2600f640 1 --2- 172.21.15.63:0/3571603115 >> v2:172.21.15.63:3300/0 conn(0x7fda2009c6a0 0x7fda200996b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:22:52.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:52.312+0000 7fda2600f640 1 --2- 172.21.15.63:0/3571603115 >> v2:172.21.15.63:3300/0 conn(0x7fda2009c6a0 0x7fda200996b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T04:22:52.714 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:52.712+0000 7fda2580e640 1 -- 172.21.15.63:0/3571603115 >> v1:172.21.15.186:6789/0 conn(0x7fda2009ca80 legacy=0x7fda2009ce90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.186:6789/0 2024-04-26T04:22:52.718 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:52.716+0000 7fda2600f640 1 --2- 172.21.15.63:0/3571603115 >> v2:172.21.15.63:3300/0 conn(0x7fda2009c6a0 0x7fda200996b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:52.718 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:52.716+0000 7fda2600f640 10 monclient(hunting): get_auth_request con 0x7fda2009c6a0 auth_method 0 2024-04-26T04:22:52.718 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:52.716+0000 7fda2600f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:52.718 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:52.716+0000 7fda2600f640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:52.718 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:52.716+0000 7fda2600f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:22:52.718 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:52.716+0000 7fda2600f640 1 -- 172.21.15.63:0/3571603115 >> v2:172.21.15.63:3300/0 conn(0x7fda2009c6a0 msgr2=0x7fda200996b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T04:22:52.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:52.716+0000 7fda2600f640 1 -- 172.21.15.63:0/3571603115 >> v2:172.21.15.63:3300/0 conn(0x7fda2009c6a0 msgr2=0x7fda200996b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:22:52.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:52.716+0000 7fda2600f640 1 --2- 172.21.15.63:0/3571603115 >> v2:172.21.15.63:3300/0 conn(0x7fda2009c6a0 0x7fda200996b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:22:52.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:52.716+0000 7fda2600f640 1 --2- 172.21.15.63:0/3571603115 >> v2:172.21.15.63:3300/0 conn(0x7fda2009c6a0 0x7fda200996b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T04:22:53.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:53.516+0000 7fda2580e640 1 -- 172.21.15.63:0/3571603115 >> v1:172.21.15.186:6789/0 conn(0x7fda2009ca80 legacy=0x7fda2009ce90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.186:6789/0 2024-04-26T04:22:53.522 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:53.520+0000 7fda2600f640 1 --2- 172.21.15.63:0/3571603115 >> v2:172.21.15.63:3300/0 conn(0x7fda2009c6a0 0x7fda200996b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:53.522 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:53.520+0000 7fda2600f640 10 monclient(hunting): get_auth_request con 0x7fda2009c6a0 auth_method 0 2024-04-26T04:22:53.522 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:53.520+0000 7fda2600f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:53.522 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:53.520+0000 7fda2600f640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:53.522 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:53.520+0000 7fda2600f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:22:53.522 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:53.520+0000 7fda2600f640 1 -- 172.21.15.63:0/3571603115 >> v2:172.21.15.63:3300/0 conn(0x7fda2009c6a0 msgr2=0x7fda200996b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T04:22:53.522 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:53.520+0000 7fda2600f640 1 -- 172.21.15.63:0/3571603115 >> v2:172.21.15.63:3300/0 conn(0x7fda2009c6a0 msgr2=0x7fda200996b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:22:53.523 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:53.520+0000 7fda2600f640 1 --2- 172.21.15.63:0/3571603115 >> v2:172.21.15.63:3300/0 conn(0x7fda2009c6a0 0x7fda200996b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:22:53.523 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:53.520+0000 7fda2600f640 1 --2- 172.21.15.63:0/3571603115 >> v2:172.21.15.63:3300/0 conn(0x7fda2009c6a0 0x7fda200996b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T04:22:55.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:55.108+0000 7fda17fff640 10 monclient(hunting): tick 2024-04-26T04:22:55.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:55.108+0000 7fda17fff640 10 monclient(hunting): _check_auth_tickets 2024-04-26T04:22:55.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:55.108+0000 7fda17fff640 1 monclient(hunting): continuing hunt 2024-04-26T04:22:55.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:55.108+0000 7fda17fff640 10 monclient(hunting): _reopen_session rank -1 2024-04-26T04:22:55.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:55.108+0000 7fda17fff640 1 -- 172.21.15.63:0/3571603115 >> v2:172.21.15.63:3300/0 conn(0x7fda2009c6a0 msgr2=0x7fda200996b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T04:22:55.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:55.108+0000 7fda17fff640 1 --2- 172.21.15.63:0/3571603115 >> v2:172.21.15.63:3300/0 conn(0x7fda2009c6a0 0x7fda200996b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:55.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:55.108+0000 7fda17fff640 1 -- 172.21.15.63:0/3571603115 >> v1:172.21.15.186:6789/0 conn(0x7fda2009ca80 legacy=0x7fda2009ce90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T04:22:55.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:55.108+0000 7fda17fff640 1 -- 172.21.15.63:0/3571603115 >> v1:172.21.15.63:6789/0 conn(0x7fda20138740 legacy=0x7fda2013ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:55.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:55.108+0000 7fda17fff640 10 monclient: _add_conns ranks=[2,4] 2024-04-26T04:22:55.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:55.108+0000 7fda17fff640 1 --2- 172.21.15.63:0/3571603115 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fda04003720 0x7fda04005b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:55.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:55.108+0000 7fda17fff640 10 monclient(hunting): picked mon.noname-b con 0x7fda04003720 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:22:55.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:55.108+0000 7fda17fff640 1 --2- 172.21.15.63:0/3571603115 >> v2:172.21.15.186:3300/0 conn(0x7fda040062a0 0x7fda04008740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:55.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:55.108+0000 7fda17fff640 10 monclient(hunting): picked mon.noname-c con 0x7fda040062a0 addr v2:172.21.15.186:3300/0 2024-04-26T04:22:55.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:55.108+0000 7fda17fff640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:55.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:55.108+0000 7fda17fff640 1 -- 172.21.15.63:0/3571603115 --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fda2009c1d0 con 0x7fda040062a0 2024-04-26T04:22:55.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:55.108+0000 7fda17fff640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:55.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:55.108+0000 7fda17fff640 1 -- 172.21.15.63:0/3571603115 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7fda04008e00 con 0x7fda04003720 2024-04-26T04:22:55.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:55.108+0000 7fda17fff640 10 monclient(hunting): _renew_subs 2024-04-26T04:22:55.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:55.108+0000 7fda2600f640 1 --2- 172.21.15.63:0/3571603115 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fda04003720 0x7fda04005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:55.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:55.108+0000 7fda2600f640 10 monclient(hunting): get_auth_request con 0x7fda04003720 auth_method 0 2024-04-26T04:22:55.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:55.108+0000 7fda2600f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:55.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:55.108+0000 7fda2600f640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:55.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:55.108+0000 7fda2600f640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:55.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:55.108+0000 7fda2580e640 1 -- 172.21.15.63:0/3571603115 >> v2:172.21.15.186:3300/0 conn(0x7fda040062a0 msgr2=0x7fda04008740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.186:3300/0 2024-04-26T04:22:55.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:55.108+0000 7fda2580e640 1 --2- 172.21.15.63:0/3571603115 >> v2:172.21.15.186:3300/0 conn(0x7fda040062a0 0x7fda04008740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:22:55.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:55.108+0000 7fda2600f640 1 -- 172.21.15.63:0/3571603115 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fda04003720 msgr2=0x7fda04005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T04:22:55.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:55.108+0000 7fda2600f640 1 -- 172.21.15.63:0/3571603115 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fda04003720 msgr2=0x7fda04005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:22:55.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:55.108+0000 7fda2600f640 1 --2- 172.21.15.63:0/3571603115 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fda04003720 0x7fda04005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:22:55.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:55.108+0000 7fda2600f640 1 --2- 172.21.15.63:0/3571603115 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fda04003720 0x7fda04005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:22:55.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:55.312+0000 7fda2580e640 1 -- 172.21.15.63:0/3571603115 >> v2:172.21.15.186:3300/0 conn(0x7fda040062a0 msgr2=0x7fda04008740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.186:3300/0 2024-04-26T04:22:55.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:55.312+0000 7fda2580e640 1 --2- 172.21.15.63:0/3571603115 >> v2:172.21.15.186:3300/0 conn(0x7fda040062a0 0x7fda04008740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T04:22:55.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:55.312+0000 7fda2600f640 1 --2- 172.21.15.63:0/3571603115 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fda04003720 0x7fda04005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:55.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:55.312+0000 7fda2600f640 10 monclient(hunting): get_auth_request con 0x7fda04003720 auth_method 0 2024-04-26T04:22:55.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:55.312+0000 7fda2600f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:55.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:55.312+0000 7fda2600f640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:55.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:55.312+0000 7fda2600f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:22:55.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:55.312+0000 7fda2600f640 1 -- 172.21.15.63:0/3571603115 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fda04003720 msgr2=0x7fda04005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T04:22:55.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:55.312+0000 7fda2600f640 1 -- 172.21.15.63:0/3571603115 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fda04003720 msgr2=0x7fda04005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:22:55.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:55.312+0000 7fda2600f640 1 --2- 172.21.15.63:0/3571603115 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fda04003720 0x7fda04005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:22:55.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:55.312+0000 7fda2600f640 1 --2- 172.21.15.63:0/3571603115 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fda04003720 0x7fda04005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T04:22:55.718 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:55.716+0000 7fda2580e640 1 -- 172.21.15.63:0/3571603115 >> v2:172.21.15.186:3300/0 conn(0x7fda040062a0 msgr2=0x7fda04008740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.186:3300/0 2024-04-26T04:22:55.718 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:55.716+0000 7fda2580e640 1 --2- 172.21.15.63:0/3571603115 >> v2:172.21.15.186:3300/0 conn(0x7fda040062a0 0x7fda04008740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T04:22:55.718 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:55.716+0000 7fda2600f640 1 --2- 172.21.15.63:0/3571603115 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fda04003720 0x7fda04005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:55.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:55.716+0000 7fda2600f640 10 monclient(hunting): get_auth_request con 0x7fda04003720 auth_method 0 2024-04-26T04:22:55.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:55.716+0000 7fda2600f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:55.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:55.716+0000 7fda2600f640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:55.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:55.716+0000 7fda2600f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:22:55.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:55.716+0000 7fda2600f640 1 -- 172.21.15.63:0/3571603115 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fda04003720 msgr2=0x7fda04005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T04:22:55.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:55.716+0000 7fda2600f640 1 -- 172.21.15.63:0/3571603115 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fda04003720 msgr2=0x7fda04005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:22:55.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:55.716+0000 7fda2600f640 1 --2- 172.21.15.63:0/3571603115 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fda04003720 0x7fda04005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:22:55.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:55.716+0000 7fda2600f640 1 --2- 172.21.15.63:0/3571603115 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fda04003720 0x7fda04005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T04:22:56.523 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.520+0000 7fda2580e640 1 -- 172.21.15.63:0/3571603115 >> v2:172.21.15.186:3300/0 conn(0x7fda040062a0 msgr2=0x7fda04008740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.186:3300/0 2024-04-26T04:22:56.523 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.520+0000 7fda2600f640 1 --2- 172.21.15.63:0/3571603115 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fda04003720 0x7fda04005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:56.523 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.520+0000 7fda2580e640 1 --2- 172.21.15.63:0/3571603115 >> v2:172.21.15.186:3300/0 conn(0x7fda040062a0 0x7fda04008740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T04:22:56.523 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.520+0000 7fda2600f640 10 monclient(hunting): get_auth_request con 0x7fda04003720 auth_method 0 2024-04-26T04:22:56.523 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.520+0000 7fda2600f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:56.523 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.520+0000 7fda2600f640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:56.523 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.520+0000 7fda2600f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:22:56.523 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.520+0000 7fda2600f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:22:56.523 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.520+0000 7fda2600f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:22:56.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.520+0000 7fda2600f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:22:56.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.520+0000 7fda2600f640 10 monclient(hunting): handle_auth_done global_id 24185 payload 274 2024-04-26T04:22:56.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.520+0000 7fda2600f640 1 -- 172.21.15.63:0/3571603115 >> v2:172.21.15.186:3300/0 conn(0x7fda040062a0 msgr2=0x7fda04008740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T04:22:56.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.520+0000 7fda2600f640 1 --2- 172.21.15.63:0/3571603115 >> v2:172.21.15.186:3300/0 conn(0x7fda040062a0 0x7fda04008740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:56.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.520+0000 7fda2600f640 10 monclient: _finish_hunting 0 2024-04-26T04:22:56.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.520+0000 7fda2600f640 1 monclient: found mon.noname-b 2024-04-26T04:22:56.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.520+0000 7fda2600f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:22:56.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.520+0000 7fda2600f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.63:3301/0 2024-04-26T04:22:56.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.520+0000 7fda2600f640 1 -- 172.21.15.63:0/3571603115 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda2009dc30 con 0x7fda04003720 2024-04-26T04:22:56.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.528+0000 7fda2600f640 1 -- 172.21.15.63:0/3571603115 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fda04003720 msgr2=0x7fda04005b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:22:56.531 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.528+0000 7fda2600f640 1 --2- 172.21.15.63:0/3571603115 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fda04003720 0x7fda04005b70 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fda1c000ee0 tx=0x7fda1c020540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dee68ff4ea5370f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:22:56.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.528+0000 7fda2500d640 1 -- 172.21.15.63:0/3571603115 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda1c00b040 con 0x7fda04003720 2024-04-26T04:22:56.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.528+0000 7fda2500d640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:56.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.528+0000 7fda2500d640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T04:22:56.533 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.528+0000 7fda2500d640 10 monclient: dump: 2024-04-26T04:22:56.533 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:56.533 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:56.533 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:56.533 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:56.533 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:56.534 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:56.534 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:56.534 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:56.534 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:56.534 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:56.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.528+0000 7fda2500d640 10 monclient: _finish_auth 0 2024-04-26T04:22:56.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.528+0000 7fda2500d640 10 monclient: _check_auth_tickets 2024-04-26T04:22:56.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.528+0000 7fda2500d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:22:56.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.528+0000 7fda2500d640 1 -- 172.21.15.63:0/3571603115 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda1c004230 con 0x7fda04003720 2024-04-26T04:22:56.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.528+0000 7fda28273640 5 monclient: authenticate success, global_id 24185 2024-04-26T04:22:56.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.528+0000 7fda28273640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T04:22:56.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.528+0000 7fda2500d640 10 monclient: handle_config config(0 keys) 2024-04-26T04:22:56.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.528+0000 7fda28273640 10 monclient: get_monmap_and_config success 2024-04-26T04:22:56.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.528+0000 7fda28273640 4 set_mon_vals no callback set 2024-04-26T04:22:56.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.528+0000 7fda2500d640 1 -- 172.21.15.63:0/3571603115 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda1c003620 con 0x7fda04003720 2024-04-26T04:22:56.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.528+0000 7fda2500d640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:56.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.528+0000 7fda2500d640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:22:56.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.528+0000 7fda2500d640 10 monclient: dump: 2024-04-26T04:22:56.537 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:56.537 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:56.537 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:56.537 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:56.538 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:56.538 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:56.538 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:56.538 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:56.538 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:56.538 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:56.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.528+0000 7fda28273640 10 monclient: shutdown 2024-04-26T04:22:56.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.528+0000 7fda28273640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:22:56.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.528+0000 7fda28273640 1 -- 172.21.15.63:0/3571603115 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fda04003720 msgr2=0x7fda04005b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:56.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.528+0000 7fda28273640 1 --2- 172.21.15.63:0/3571603115 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fda04003720 0x7fda04005b70 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fda1c000ee0 tx=0x7fda1c020540 comp rx=0 tx=0).stop 2024-04-26T04:22:56.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.528+0000 7fda26810640 1 -- 172.21.15.63:0/3571603115 reap_dead start 2024-04-26T04:22:56.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.528+0000 7fda28273640 1 -- 172.21.15.63:0/3571603115 shutdown_connections 2024-04-26T04:22:56.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.528+0000 7fda28273640 1 -- 172.21.15.63:0/3571603115 >> 172.21.15.63:0/3571603115 conn(0x7fda200a1e70 msgr2=0x7fda200a2280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:22:56.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.528+0000 7fda28273640 1 -- 172.21.15.63:0/3571603115 shutdown_connections 2024-04-26T04:22:56.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.528+0000 7fda28273640 1 -- 172.21.15.63:0/3571603115 wait complete. 2024-04-26T04:22:56.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.532+0000 7fda28273640 10 monclient: build_initial_monmap 2024-04-26T04:22:56.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.532+0000 7fda28273640 10 monclient: monmap: 2024-04-26T04:22:56.546 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:22:56.547 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:22:56.547 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:22:56.547 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:22:56.547 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:22:56.547 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:56.547 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:22:56.547 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:22:56.547 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:22:56.547 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:56.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.532+0000 7fda28273640 1 Processor -- start 2024-04-26T04:22:56.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.532+0000 7fda28273640 1 -- start start 2024-04-26T04:22:56.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.532+0000 7fda28273640 10 monclient: init 2024-04-26T04:22:56.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.532+0000 7fda28273640 10 monclient: _reopen_session rank -1 2024-04-26T04:22:56.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.532+0000 7fda28273640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T04:22:56.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.532+0000 7fda28273640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fda201cadc0 0x7fda201338b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:56.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.532+0000 7fda28273640 10 monclient(hunting): picked mon.noname-a con 0x7fda201cadc0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:22:56.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.532+0000 7fda28273640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fda20133df0 0x7fda201c78c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:56.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.532+0000 7fda28273640 10 monclient(hunting): picked mon.noname-c con 0x7fda20133df0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:22:56.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.532+0000 7fda28273640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda20071720 0x7fda201c7e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:56.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.532+0000 7fda28273640 10 monclient(hunting): picked mon.noname-b con 0x7fda20071720 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:22:56.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.532+0000 7fda28273640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:56.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.532+0000 7fda28273640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fda2013e3e0 con 0x7fda201cadc0 2024-04-26T04:22:56.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.532+0000 7fda28273640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:56.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.532+0000 7fda28273640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fda2013e260 con 0x7fda20071720 2024-04-26T04:22:56.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.532+0000 7fda28273640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:56.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.532+0000 7fda28273640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7fda2013e560 con 0x7fda20133df0 2024-04-26T04:22:56.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.532+0000 7fda28273640 10 monclient(hunting): _renew_subs 2024-04-26T04:22:56.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.532+0000 7fda28273640 10 monclient(hunting): authenticate will time out at 1208.243718s 2024-04-26T04:22:56.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.532+0000 7fda2580e640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fda20133df0 0x7fda201c78c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:56.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.532+0000 7fda2600f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fda201cadc0 0x7fda201338b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:56.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.532+0000 7fda2580e640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fda20133df0 0x7fda201c78c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:42354/0 (socket says 172.21.15.63:42354) 2024-04-26T04:22:56.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.532+0000 7fda2580e640 1 -- 172.21.15.63:0/2471111092 learned_addr learned my addr 172.21.15.63:0/2471111092 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:22:56.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.532+0000 7fda2580e640 10 monclient(hunting): get_auth_request con 0x7fda20133df0 auth_method 0 2024-04-26T04:22:56.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.532+0000 7fda2580e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:56.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.532+0000 7fda2580e640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:56.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.532+0000 7fda2580e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:56.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.532+0000 7fda26810640 1 -- 172.21.15.63:0/2471111092 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda20071720 msgr2=0x7fda201c7e00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.186:3300/0 2024-04-26T04:22:56.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.532+0000 7fda2600f640 10 monclient(hunting): get_auth_request con 0x7fda201cadc0 auth_method 0 2024-04-26T04:22:56.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.532+0000 7fda2600f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:56.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.532+0000 7fda2600f640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:56.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.532+0000 7fda2600f640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:56.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.532+0000 7fda26810640 1 --2- 172.21.15.63:0/2471111092 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda20071720 0x7fda201c7e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:22:56.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.532+0000 7fda2580e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:22:56.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.532+0000 7fda2580e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:22:56.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.536+0000 7fda2580e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:22:56.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.536+0000 7fda2600f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:22:56.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.536+0000 7fda2600f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:22:56.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.536+0000 7fda2600f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:22:56.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.536+0000 7fda2600f640 10 monclient(hunting): handle_auth_done global_id 5283 payload 824 2024-04-26T04:22:56.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.536+0000 7fda2600f640 1 -- 172.21.15.63:0/2471111092 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fda20133df0 msgr2=0x7fda201c78c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:56.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.536+0000 7fda2600f640 1 --2- 172.21.15.63:0/2471111092 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fda20133df0 0x7fda201c78c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:56.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.536+0000 7fda2600f640 1 -- 172.21.15.63:0/2471111092 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda20071720 msgr2=0x7fda201c7e00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T04:22:56.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.536+0000 7fda2600f640 1 --2- 172.21.15.63:0/2471111092 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda20071720 0x7fda201c7e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:56.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.536+0000 7fda2600f640 10 monclient: _finish_hunting 0 2024-04-26T04:22:56.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.536+0000 7fda2600f640 1 monclient: found mon.noname-a 2024-04-26T04:22:56.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.536+0000 7fda2600f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:22:56.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.536+0000 7fda2600f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:22:56.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.536+0000 7fda2600f640 1 -- 172.21.15.63:0/2471111092 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda20071f10 con 0x7fda201cadc0 2024-04-26T04:22:56.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.536+0000 7fda2580e640 1 --2- 172.21.15.63:0/2471111092 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fda20133df0 0x7fda201c78c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T04:22:56.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.536+0000 7fda2600f640 1 -- 172.21.15.63:0/2471111092 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fda201cadc0 msgr2=0x7fda201338b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:22:56.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.536+0000 7fda2600f640 1 --2- 172.21.15.63:0/2471111092 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fda201cadc0 0x7fda201338b0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fda04008fd0 tx=0x7fda1c01ed80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:22:56.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.536+0000 7fda16ffd640 1 -- 172.21.15.63:0/2471111092 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda1c0c3070 con 0x7fda201cadc0 2024-04-26T04:22:56.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.536+0000 7fda16ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:56.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.536+0000 7fda16ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:22:56.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.536+0000 7fda16ffd640 10 monclient: dump: 2024-04-26T04:22:56.553 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:56.553 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:56.553 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:56.554 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:56.554 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:56.554 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:56.554 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:56.554 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:56.554 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:56.554 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:56.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.536+0000 7fda16ffd640 10 monclient: _finish_auth 0 2024-04-26T04:22:56.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.536+0000 7fda16ffd640 10 monclient: _check_auth_tickets 2024-04-26T04:22:56.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.536+0000 7fda16ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:22:56.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.536+0000 7fda28273640 5 monclient: authenticate success, global_id 5283 2024-04-26T04:22:56.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.536+0000 7fda16ffd640 1 -- 172.21.15.63:0/2471111092 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda1c003820 con 0x7fda201cadc0 2024-04-26T04:22:56.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.536+0000 7fda28273640 10 monclient: _renew_subs 2024-04-26T04:22:56.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.536+0000 7fda28273640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:22:56.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.536+0000 7fda28273640 1 -- 172.21.15.63:0/2471111092 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda20072440 con 0x7fda201cadc0 2024-04-26T04:22:56.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.536+0000 7fda16ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T04:22:56.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.536+0000 7fda16ffd640 1 -- 172.21.15.63:0/2471111092 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda1c003bf0 con 0x7fda201cadc0 2024-04-26T04:22:56.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.536+0000 7fda16ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:56.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.536+0000 7fda16ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:22:56.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.536+0000 7fda16ffd640 10 monclient: dump: 2024-04-26T04:22:56.555 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:56.555 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:56.555 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:56.555 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:56.555 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:56.555 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:56.555 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:56.556 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:56.556 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:56.556 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:56.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.536+0000 7fda28273640 10 monclient: _renew_subs 2024-04-26T04:22:56.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.536+0000 7fda28273640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:22:56.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.536+0000 7fda28273640 1 -- 172.21.15.63:0/2471111092 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda20070a90 con 0x7fda201cadc0 2024-04-26T04:22:56.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.536+0000 7fda2480c640 4 set_mon_vals no callback set 2024-04-26T04:22:56.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.540+0000 7fda28273640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:22:56.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.540+0000 7fda28273640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:22:56.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.540+0000 7fda16ffd640 1 -- 172.21.15.63:0/2471111092 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fda1c0c6070 con 0x7fda201cadc0 2024-04-26T04:22:56.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.540+0000 7fda28273640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:22:56.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.540+0000 7fda28273640 1 -- 172.21.15.63:0/2471111092 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda20062e10 con 0x7fda201cadc0 2024-04-26T04:22:56.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.540+0000 7fda16ffd640 1 --2- 172.21.15.63:0/2471111092 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fda04036d10 0x7fda040391d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:56.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.540+0000 7fda16ffd640 1 -- 172.21.15.63:0/2471111092 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(215..215 src has 1..215) ==== 4346+0+0 (secure 0 0 0) 0x7fda1c0e6960 con 0x7fda201cadc0 2024-04-26T04:22:56.557 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.540+0000 7fda2580e640 1 --2- 172.21.15.63:0/2471111092 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fda04036d10 0x7fda040391d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:56.557 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.540+0000 7fda2580e640 10 monclient: get_auth_request con 0x7fda04036d10 auth_method 0 2024-04-26T04:22:56.557 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.544+0000 7fda2580e640 1 --2- 172.21.15.63:0/2471111092 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fda04036d10 0x7fda040391d0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fda100091d0 tx=0x7fda10009140 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:22:56.557 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.544+0000 7fda16ffd640 1 -- 172.21.15.63:0/2471111092 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fda1c0b2020 con 0x7fda201cadc0 2024-04-26T04:22:56.557 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.544+0000 7fda16ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:22:56.557 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.544+0000 7fda16ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:22:56.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.768+0000 7fda28273640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T04:22:56.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.768+0000 7fda28273640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T04:22:56.772 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.768+0000 7fda28273640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:22:56.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.768+0000 7fda28273640 1 -- 172.21.15.63:0/2471111092 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fda200a4e50 con 0x7fda201cadc0 2024-04-26T04:22:56.773 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.768+0000 7fda16ffd640 1 -- 172.21.15.63:0/2471111092 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1554 (secure 0 0 0) 0x7fda1c008050 con 0x7fda201cadc0 2024-04-26T04:22:56.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.768+0000 7fda16ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T04:22:56.774 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.768+0000 7fda16ffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:22:56.774 INFO:teuthology.orchestra.run.smithi063.stdout:{"election_epoch":24,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cd1c6a0c-4e4b-4014-89c0-2e535479ef39","modified":"2024-04-26T04:18:03.702339Z","created":"2024-04-26T04:18:03.702339Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3300","nonce":0},{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:3300","nonce":0},{"type":"v1","addr":"172.21.15.186:6789","nonce":0}]},"addr":"172.21.15.186:6789/0","public_addr":"172.21.15.186:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3301","nonce":0},{"type":"v1","addr":"172.21.15.63:6790","nonce":0}]},"addr":"172.21.15.63:6790/0","public_addr":"172.21.15.63:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T04:22:56.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.772+0000 7fda28273640 1 -- 172.21.15.63:0/2471111092 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fda04036d10 msgr2=0x7fda040391d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:56.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.772+0000 7fda28273640 1 --2- 172.21.15.63:0/2471111092 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fda04036d10 0x7fda040391d0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fda100091d0 tx=0x7fda10009140 comp rx=0 tx=0).stop 2024-04-26T04:22:56.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.772+0000 7fda28273640 10 monclient: shutdown 2024-04-26T04:22:56.776 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.772+0000 7fda28273640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:22:56.777 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.772+0000 7fda28273640 1 -- 172.21.15.63:0/2471111092 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fda201cadc0 msgr2=0x7fda201338b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:56.777 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.772+0000 7fda28273640 1 --2- 172.21.15.63:0/2471111092 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fda201cadc0 0x7fda201338b0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fda04008fd0 tx=0x7fda1c01ed80 comp rx=0 tx=0).stop 2024-04-26T04:22:56.777 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.772+0000 7fda28273640 1 -- 172.21.15.63:0/2471111092 shutdown_connections 2024-04-26T04:22:56.777 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.772+0000 7fda28273640 1 --2- 172.21.15.63:0/2471111092 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fda04036d10 0x7fda040391d0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:56.777 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.772+0000 7fda28273640 1 --2- 172.21.15.63:0/2471111092 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda20071720 0x7fda201c7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:56.777 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.772+0000 7fda28273640 1 --2- 172.21.15.63:0/2471111092 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fda20133df0 0x7fda201c78c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:56.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.772+0000 7fda28273640 1 --2- 172.21.15.63:0/2471111092 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fda201cadc0 0x7fda201338b0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:56.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.772+0000 7fda28273640 1 -- 172.21.15.63:0/2471111092 >> 172.21.15.63:0/2471111092 conn(0x7fda200a1e70 msgr2=0x7fda200a2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:22:56.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.772+0000 7fda28273640 1 -- 172.21.15.63:0/2471111092 shutdown_connections 2024-04-26T04:22:56.778 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.772+0000 7fda28273640 1 -- 172.21.15.63:0/2471111092 wait complete. 2024-04-26T04:22:56.792 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-04-26T04:22:56.792 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-04-26T04:22:56.793 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T04:22:56.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.904+0000 7f5945de2640 10 monclient: get_monmap_and_config 2024-04-26T04:22:56.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.904+0000 7f5945de2640 10 monclient: build_initial_monmap 2024-04-26T04:22:56.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.904+0000 7f5945de2640 10 monclient: monmap: 2024-04-26T04:22:56.910 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:22:56.910 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:56.910 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:22:56.909729+0000 2024-04-26T04:22:56.910 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:22:56.909729+0000 2024-04-26T04:22:56.910 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:22:56.910 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:56.910 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:22:56.910 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:22:56.910 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:22:56.910 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:22:56.910 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:22:56.911 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:56.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.908+0000 7f5945de2640 1 Processor -- start 2024-04-26T04:22:56.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.908+0000 7f5945de2640 1 -- start start 2024-04-26T04:22:56.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.908+0000 7f5945de2640 10 monclient: init 2024-04-26T04:22:56.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.908+0000 7f5945de2640 10 monclient: _reopen_session rank -1 2024-04-26T04:22:56.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.908+0000 7f5945de2640 10 monclient: _add_conns ranks=[0,2,4,1,3] 2024-04-26T04:22:56.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.908+0000 7f5945de2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f594009b3b0 addr v1:172.21.15.63:6789/0 2024-04-26T04:22:56.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.908+0000 7f5945de2640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f5940137320 0x7f5940139710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:56.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.908+0000 7f5945de2640 10 monclient(hunting): picked mon.noname-b con 0x7f5940137320 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:22:56.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.908+0000 7f5945de2640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f5940139c50 0x7f594013c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:56.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.908+0000 7f5945de2640 10 monclient(hunting): picked mon.noname-c con 0x7f5940139c50 addr v2:172.21.15.186:3300/0 2024-04-26T04:22:56.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.908+0000 7f5945de2640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f594009dde0 con 0x7f594009b3b0 2024-04-26T04:22:56.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.908+0000 7f5945de2640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:56.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.908+0000 7f5945de2640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f594009c100 con 0x7f5940139c50 2024-04-26T04:22:56.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.908+0000 7f5945de2640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:56.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.908+0000 7f5945de2640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f594009b790 con 0x7f5940137320 2024-04-26T04:22:56.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.908+0000 7f5945de2640 10 monclient(hunting): _renew_subs 2024-04-26T04:22:56.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.908+0000 7f5945de2640 10 monclient(hunting): authenticate will time out at 1208.617526s 2024-04-26T04:22:56.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.908+0000 7f593f7fe640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f594009b3b0 0x7f594009e4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:34486/0 (socket says 172.21.15.63:34486) 2024-04-26T04:22:56.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.908+0000 7f593f7fe640 1 -- 172.21.15.63:0/1783688726 learned_addr learned my addr 172.21.15.63:0/1783688726 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-04-26T04:22:56.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.908+0000 7f593effd640 1 --2- 172.21.15.63:0/1783688726 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f5940137320 0x7f5940139710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:56.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.908+0000 7f593ffff640 1 -- 172.21.15.63:0/1783688726 >> v2:172.21.15.186:3300/0 conn(0x7f5940139c50 msgr2=0x7f594013c040 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.186:3300/0 2024-04-26T04:22:56.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.908+0000 7f593effd640 10 monclient(hunting): get_auth_request con 0x7f5940137320 auth_method 0 2024-04-26T04:22:56.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.908+0000 7f593effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:56.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.908+0000 7f593effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:56.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.908+0000 7f593effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:56.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.908+0000 7f593ffff640 1 --2- 172.21.15.63:0/1783688726 >> v2:172.21.15.186:3300/0 conn(0x7f5940139c50 0x7f594013c040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:22:56.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.908+0000 7f593effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:22:56.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.908+0000 7f593effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:22:56.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.908+0000 7f593effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:22:56.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.908+0000 7f593effd640 10 monclient(hunting): handle_auth_done global_id 24191 payload 274 2024-04-26T04:22:56.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.908+0000 7f593effd640 1 -- 172.21.15.63:0/1783688726 >> v2:172.21.15.186:3300/0 conn(0x7f5940139c50 msgr2=0x7f594013c040 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T04:22:56.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.908+0000 7f593effd640 1 --2- 172.21.15.63:0/1783688726 >> v2:172.21.15.186:3300/0 conn(0x7f5940139c50 0x7f594013c040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:56.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.908+0000 7f593effd640 1 -- 172.21.15.63:0/1783688726 >> v1:172.21.15.63:6789/0 conn(0x7f594009b3b0 legacy=0x7f594009e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:22:56.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.908+0000 7f593effd640 10 monclient: _finish_hunting 0 2024-04-26T04:22:56.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.908+0000 7f593effd640 1 monclient: found mon.noname-b 2024-04-26T04:22:56.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.908+0000 7f593effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:22:56.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.908+0000 7f593effd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.63:3301/0 2024-04-26T04:22:56.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.908+0000 7f593effd640 1 -- 172.21.15.63:0/1783688726 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f594009c9b0 con 0x7f5940137320 2024-04-26T04:22:56.920 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.912+0000 7f593effd640 1 -- 172.21.15.63:0/1783688726 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f5940137320 msgr2=0x7f5940139710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:22:56.920 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.912+0000 7f593effd640 1 --2- 172.21.15.63:0/1783688726 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f5940137320 0x7f5940139710 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f59340097c0 tx=0x7f593401c160 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db148af85710c68b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:22:56.920 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.912+0000 7f593e7fc640 1 -- 172.21.15.63:0/1783688726 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59340b0070 con 0x7f5940137320 2024-04-26T04:22:56.920 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.912+0000 7f593e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:56.921 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.912+0000 7f593e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T04:22:56.921 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.912+0000 7f593e7fc640 10 monclient: dump: 2024-04-26T04:22:56.921 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:56.921 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:56.921 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:56.921 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:56.921 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:56.921 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:56.921 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:56.921 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:56.921 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:56.921 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:56.921 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.912+0000 7f593e7fc640 10 monclient: _finish_auth 0 2024-04-26T04:22:56.921 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.912+0000 7f593e7fc640 10 monclient: _check_auth_tickets 2024-04-26T04:22:56.922 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.912+0000 7f593e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:22:56.922 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.912+0000 7f593e7fc640 1 -- 172.21.15.63:0/1783688726 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59340ab600 con 0x7f5940137320 2024-04-26T04:22:56.922 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.912+0000 7f593e7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T04:22:56.922 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.912+0000 7f593e7fc640 1 -- 172.21.15.63:0/1783688726 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59340ab9d0 con 0x7f5940137320 2024-04-26T04:22:56.922 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.912+0000 7f593e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:56.922 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.912+0000 7f593e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:22:56.922 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.912+0000 7f593e7fc640 10 monclient: dump: 2024-04-26T04:22:56.922 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:56.922 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:56.922 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:56.922 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:56.922 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:56.922 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:56.922 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:56.923 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:56.923 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:56.923 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:56.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.912+0000 7f5945de2640 5 monclient: authenticate success, global_id 24191 2024-04-26T04:22:56.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.912+0000 7f5945de2640 10 monclient: get_monmap_and_config success 2024-04-26T04:22:56.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.912+0000 7f5945de2640 4 set_mon_vals no callback set 2024-04-26T04:22:56.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.912+0000 7f5945de2640 10 monclient: shutdown 2024-04-26T04:22:56.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.912+0000 7f5945de2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:22:56.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.912+0000 7f5945de2640 1 -- 172.21.15.63:0/1783688726 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f5940137320 msgr2=0x7f5940139710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:56.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.912+0000 7f5945de2640 1 --2- 172.21.15.63:0/1783688726 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f5940137320 0x7f5940139710 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f59340097c0 tx=0x7f593401c160 comp rx=0 tx=0).stop 2024-04-26T04:22:56.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.912+0000 7f5945de2640 1 -- 172.21.15.63:0/1783688726 shutdown_connections 2024-04-26T04:22:56.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.912+0000 7f5945de2640 1 --2- 172.21.15.63:0/1783688726 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f5940137320 0x7f5940139710 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:56.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.912+0000 7f5945de2640 1 --2- 172.21.15.63:0/1783688726 >> v2:172.21.15.186:3300/0 conn(0x7f5940139c50 0x7f594013c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:56.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.912+0000 7f5945de2640 1 -- 172.21.15.63:0/1783688726 >> 172.21.15.63:0/1783688726 conn(0x7f59400a1e70 msgr2=0x7f59400a2280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:22:56.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.912+0000 7f5945de2640 1 -- 172.21.15.63:0/1783688726 shutdown_connections 2024-04-26T04:22:56.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.912+0000 7f5945de2640 1 -- 172.21.15.63:0/1783688726 wait complete. 2024-04-26T04:22:56.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.916+0000 7f5945de2640 10 monclient: build_initial_monmap 2024-04-26T04:22:56.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.916+0000 7f5945de2640 10 monclient: monmap: 2024-04-26T04:22:56.924 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:22:56.924 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:22:56.924 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:22:56.924 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:22:56.924 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:22:56.924 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:56.924 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:22:56.924 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:22:56.925 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:22:56.925 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:56.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.916+0000 7f5945de2640 1 Processor -- start 2024-04-26T04:22:56.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.916+0000 7f5945de2640 1 -- start start 2024-04-26T04:22:56.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.916+0000 7f5945de2640 10 monclient: init 2024-04-26T04:22:56.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.916+0000 7f5945de2640 10 monclient: _reopen_session rank -1 2024-04-26T04:22:56.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.916+0000 7f5945de2640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T04:22:56.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.916+0000 7f5945de2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f594009b3b0 0x7f59401ce960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:56.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.916+0000 7f5945de2640 10 monclient(hunting): picked mon.noname-b con 0x7f594009b3b0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:22:56.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.916+0000 7f5945de2640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f5940137320 0x7f5940135710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:56.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.916+0000 7f5945de2640 10 monclient(hunting): picked mon.noname-c con 0x7f5940137320 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:22:56.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.916+0000 7f5945de2640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5940139c50 0x7f5940135c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:56.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.916+0000 7f5945de2640 10 monclient(hunting): picked mon.noname-a con 0x7f5940139c50 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:22:56.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.916+0000 7f5945de2640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:56.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.916+0000 7f5945de2640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f594013f170 con 0x7f5940139c50 2024-04-26T04:22:56.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.916+0000 7f5945de2640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:56.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.916+0000 7f5945de2640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f594013eff0 con 0x7f594009b3b0 2024-04-26T04:22:56.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.916+0000 7f5945de2640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:56.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.916+0000 7f5945de2640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f594013f2f0 con 0x7f5940137320 2024-04-26T04:22:56.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.916+0000 7f5945de2640 10 monclient(hunting): _renew_subs 2024-04-26T04:22:56.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.916+0000 7f5945de2640 10 monclient(hunting): authenticate will time out at 1208.625262s 2024-04-26T04:22:56.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.916+0000 7f593effd640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f5940137320 0x7f5940135710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:56.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.916+0000 7f593effd640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f5940137320 0x7f5940135710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:42366/0 (socket says 172.21.15.63:42366) 2024-04-26T04:22:56.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.916+0000 7f593ffff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5940139c50 0x7f5940135c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:56.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.916+0000 7f593effd640 1 -- 172.21.15.63:0/1464603515 learned_addr learned my addr 172.21.15.63:0/1464603515 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:22:56.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.916+0000 7f593effd640 10 monclient(hunting): get_auth_request con 0x7f5940137320 auth_method 0 2024-04-26T04:22:56.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.916+0000 7f593f7fe640 1 -- 172.21.15.63:0/1464603515 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f594009b3b0 msgr2=0x7f59401ce960 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.186:3300/0 2024-04-26T04:22:56.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.916+0000 7f593effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:56.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.916+0000 7f593effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:56.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.916+0000 7f593effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:56.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.916+0000 7f593f7fe640 1 --2- 172.21.15.63:0/1464603515 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f594009b3b0 0x7f59401ce960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:22:56.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.916+0000 7f593ffff640 10 monclient(hunting): get_auth_request con 0x7f5940139c50 auth_method 0 2024-04-26T04:22:56.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.916+0000 7f593ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:56.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.916+0000 7f593ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:56.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.916+0000 7f593ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:56.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.916+0000 7f593effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:22:56.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.916+0000 7f593effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:22:56.927 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.916+0000 7f593effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:22:56.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.916+0000 7f593ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:22:56.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.916+0000 7f593ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:22:56.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.916+0000 7f593ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:22:56.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.916+0000 7f593effd640 10 monclient(hunting): handle_auth_done global_id 24194 payload 824 2024-04-26T04:22:56.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.916+0000 7f593effd640 1 -- 172.21.15.63:0/1464603515 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f594009b3b0 msgr2=0x7f59401ce960 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T04:22:56.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.916+0000 7f593effd640 1 --2- 172.21.15.63:0/1464603515 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f594009b3b0 0x7f59401ce960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:56.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.916+0000 7f593effd640 1 -- 172.21.15.63:0/1464603515 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5940139c50 msgr2=0x7f5940135c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:56.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.916+0000 7f593effd640 1 --2- 172.21.15.63:0/1464603515 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5940139c50 0x7f5940135c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:56.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.916+0000 7f593effd640 10 monclient: _finish_hunting 0 2024-04-26T04:22:56.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.916+0000 7f593effd640 1 monclient: found mon.noname-c 2024-04-26T04:22:56.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.916+0000 7f593effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:22:56.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.916+0000 7f593effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.63:3301/0 2024-04-26T04:22:56.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.916+0000 7f593effd640 1 -- 172.21.15.63:0/1464603515 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5940130bf0 con 0x7f5940137320 2024-04-26T04:22:56.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.916+0000 7f593ffff640 1 --2- 172.21.15.63:0/1464603515 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5940139c50 0x7f5940135c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T04:22:56.932 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.920+0000 7f593effd640 1 -- 172.21.15.63:0/1464603515 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f5940137320 msgr2=0x7f5940135710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:22:56.932 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.920+0000 7f593effd640 1 --2- 172.21.15.63:0/1464603515 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f5940137320 0x7f5940135710 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f593401a810 tx=0x7f593401a9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:22:56.932 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.920+0000 7f5927fff640 1 -- 172.21.15.63:0/1464603515 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59340b0050 con 0x7f5940137320 2024-04-26T04:22:56.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.920+0000 7f5927fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:56.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.920+0000 7f5927fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:22:56.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.920+0000 7f5927fff640 10 monclient: dump: 2024-04-26T04:22:56.933 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:56.933 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:56.933 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:56.933 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:56.933 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:56.933 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:56.933 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:56.933 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:56.933 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:56.933 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:56.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.920+0000 7f5927fff640 10 monclient: _finish_auth 0 2024-04-26T04:22:56.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.920+0000 7f5927fff640 10 monclient: _check_auth_tickets 2024-04-26T04:22:56.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.920+0000 7f5927fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:22:56.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.920+0000 7f5927fff640 1 -- 172.21.15.63:0/1464603515 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59340bc070 con 0x7f5940137320 2024-04-26T04:22:56.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.920+0000 7f5927fff640 10 monclient: handle_config config(0 keys) 2024-04-26T04:22:56.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.920+0000 7f5927fff640 1 -- 172.21.15.63:0/1464603515 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59340b5c40 con 0x7f5940137320 2024-04-26T04:22:56.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.920+0000 7f5927fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:56.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.920+0000 7f5927fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:22:56.934 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.920+0000 7f5927fff640 10 monclient: dump: 2024-04-26T04:22:56.934 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:56.934 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:56.934 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:56.934 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:56.934 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:56.934 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:56.934 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:56.934 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:56.934 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:56.935 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:56.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.920+0000 7f5945de2640 5 monclient: authenticate success, global_id 24194 2024-04-26T04:22:56.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.920+0000 7f5945de2640 10 monclient: _renew_subs 2024-04-26T04:22:56.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.920+0000 7f5945de2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:22:56.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.920+0000 7f5945de2640 1 -- 172.21.15.63:0/1464603515 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f594009cc50 con 0x7f5940137320 2024-04-26T04:22:56.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.920+0000 7f593dffb640 4 set_mon_vals no callback set 2024-04-26T04:22:56.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.920+0000 7f5945de2640 10 monclient: _renew_subs 2024-04-26T04:22:56.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.920+0000 7f5945de2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:22:56.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.920+0000 7f5945de2640 1 -- 172.21.15.63:0/1464603515 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f59401311b0 con 0x7f5940137320 2024-04-26T04:22:56.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.920+0000 7f5945de2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:22:56.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.920+0000 7f5945de2640 1 --2- 172.21.15.63:0/1464603515 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f594012d320 0x7f594012d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:56.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.920+0000 7f5945de2640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:56.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.920+0000 7f5945de2640 1 -- 172.21.15.63:0/1464603515 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f5940136190 con 0x7f594012d320 2024-04-26T04:22:56.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.920+0000 7f593f7fe640 1 --2- 172.21.15.63:0/1464603515 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f594012d320 0x7f594012d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:56.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.920+0000 7f593f7fe640 10 monclient: get_auth_request con 0x7f594012d320 auth_method 0 2024-04-26T04:22:56.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.920+0000 7f593f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:56.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.920+0000 7f593f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:56.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.920+0000 7f593f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:56.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.920+0000 7f5927fff640 1 -- 172.21.15.63:0/1464603515 <== mon.2 v2:172.21.15.63:3301/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f59340c1460 con 0x7f5940137320 2024-04-26T04:22:56.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.920+0000 7f593f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:22:56.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.920+0000 7f593f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:22:56.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.920+0000 7f593f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:22:56.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.920+0000 7f593f7fe640 10 monclient(hunting): handle_auth_done global_id 5289 payload 824 2024-04-26T04:22:56.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.920+0000 7f5927fff640 1 --2- 172.21.15.63:0/1464603515 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f592c035ec0 0x7f592c038380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:56.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.920+0000 7f593f7fe640 1 -- 172.21.15.63:0/1464603515 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5940062e10 con 0x7f594012d320 2024-04-26T04:22:56.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.920+0000 7f5927fff640 1 -- 172.21.15.63:0/1464603515 <== mon.2 v2:172.21.15.63:3301/0 5 ==== osd_map(215..215 src has 1..215) ==== 4346+0+0 (secure 0 0 0) 0x7f59340e3850 con 0x7f5940137320 2024-04-26T04:22:56.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.920+0000 7f593ffff640 1 --2- 172.21.15.63:0/1464603515 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f592c035ec0 0x7f592c038380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:56.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.920+0000 7f593ffff640 10 monclient: get_auth_request con 0x7f592c035ec0 auth_method 0 2024-04-26T04:22:56.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.920+0000 7f593f7fe640 1 -- 172.21.15.63:0/1464603515 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f594012d320 msgr2=0x7f594012d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:22:56.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.920+0000 7f593f7fe640 1 --2- 172.21.15.63:0/1464603515 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f594012d320 0x7f594012d700 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f5930004c00 tx=0x7f593000cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:22:56.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.920+0000 7f5927fff640 1 -- 172.21.15.63:0/1464603515 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5930017070 con 0x7f594012d320 2024-04-26T04:22:56.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.920+0000 7f5927fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:56.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.920+0000 7f5927fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:22:56.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.920+0000 7f5927fff640 10 monclient: dump: 2024-04-26T04:22:56.937 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:56.937 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:56.937 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:56.937 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:56.937 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:56.937 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:56.937 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:56.937 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:56.937 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:56.937 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:56.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.924+0000 7f5927fff640 1 -- 172.21.15.63:0/1464603515 <== mon.0 v2:172.21.15.63:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5930012400 con 0x7f594012d320 2024-04-26T04:22:56.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.924+0000 7f5927fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:22:56.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.924+0000 7f5927fff640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:22:56.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.924+0000 7f5927fff640 1 -- 172.21.15.63:0/1464603515 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f594012d320 msgr2=0x7f594012d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:56.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.924+0000 7f5927fff640 1 --2- 172.21.15.63:0/1464603515 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f594012d320 0x7f594012d700 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f5930004c00 tx=0x7f593000cf80 comp rx=0 tx=0).stop 2024-04-26T04:22:56.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.924+0000 7f5927fff640 1 -- 172.21.15.63:0/1464603515 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f594012d320 msgr2=0x7f594012d700 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:22:56.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.924+0000 7f5927fff640 1 --2- 172.21.15.63:0/1464603515 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f594012d320 0x7f594012d700 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:56.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.924+0000 7f593ffff640 1 --2- 172.21.15.63:0/1464603515 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f592c035ec0 0x7f592c038380 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f594012fb80 tx=0x7f5938003800 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:22:56.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.928+0000 7f5945de2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T04:22:56.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.928+0000 7f5945de2640 1 --2- 172.21.15.63:0/1464603515 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f594012dc40 0x7f594012a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:56.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.928+0000 7f5945de2640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:56.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.928+0000 7f5945de2640 1 -- 172.21.15.63:0/1464603515 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f59400504d0 con 0x7f594012dc40 2024-04-26T04:22:56.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.928+0000 7f593f7fe640 1 --2- 172.21.15.63:0/1464603515 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f594012dc40 0x7f594012a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:56.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.928+0000 7f593f7fe640 10 monclient: get_auth_request con 0x7f594012dc40 auth_method 0 2024-04-26T04:22:56.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.928+0000 7f593f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:56.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.928+0000 7f593f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:56.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.928+0000 7f593f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:56.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.928+0000 7f593f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:22:56.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.928+0000 7f593f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:22:56.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.928+0000 7f593f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:22:56.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.928+0000 7f593f7fe640 10 monclient(hunting): handle_auth_done global_id 5292 payload 824 2024-04-26T04:22:56.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.928+0000 7f593f7fe640 1 -- 172.21.15.63:0/1464603515 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5940068430 con 0x7f594012dc40 2024-04-26T04:22:56.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.932+0000 7f593f7fe640 1 -- 172.21.15.63:0/1464603515 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f594012dc40 msgr2=0x7f594012a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:22:56.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.932+0000 7f593f7fe640 1 --2- 172.21.15.63:0/1464603515 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f594012dc40 0x7f594012a260 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f5930014000 tx=0x7f5930019f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:22:56.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.932+0000 7f5927fff640 1 -- 172.21.15.63:0/1464603515 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5930017040 con 0x7f594012dc40 2024-04-26T04:22:56.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.932+0000 7f5927fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:56.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.932+0000 7f5927fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:22:56.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.932+0000 7f5927fff640 10 monclient: dump: 2024-04-26T04:22:56.940 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:56.940 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:56.940 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:56.941 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:56.941 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:56.941 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:56.941 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:56.941 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:56.941 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:56.941 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:56.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.932+0000 7f5927fff640 1 -- 172.21.15.63:0/1464603515 <== mon.0 v2:172.21.15.63:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4882 (secure 0 0 0) 0x7f5930013830 con 0x7f594012dc40 2024-04-26T04:22:56.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.932+0000 7f5927fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T04:22:56.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.932+0000 7f5927fff640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:22:56.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.932+0000 7f5927fff640 1 -- 172.21.15.63:0/1464603515 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f594012dc40 msgr2=0x7f594012a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:56.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.932+0000 7f5927fff640 1 --2- 172.21.15.63:0/1464603515 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f594012dc40 0x7f594012a260 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f5930014000 tx=0x7f5930019f00 comp rx=0 tx=0).stop 2024-04-26T04:22:56.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.932+0000 7f5927fff640 1 -- 172.21.15.63:0/1464603515 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f594012dc40 msgr2=0x7f594012a260 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:22:56.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.932+0000 7f5927fff640 1 --2- 172.21.15.63:0/1464603515 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f594012dc40 0x7f594012a260 secure :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f5930014000 tx=0x7f5930019f00 comp rx=0 tx=0).stop 2024-04-26T04:22:56.942 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-26T04:22:56.942 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a", 2024-04-26T04:22:56.942 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 0, 2024-04-26T04:22:56.942 INFO:teuthology.orchestra.run.smithi063.stdout: "state": "leader", 2024-04-26T04:22:56.942 INFO:teuthology.orchestra.run.smithi063.stdout: "election_epoch": 24, 2024-04-26T04:22:56.942 INFO:teuthology.orchestra.run.smithi063.stdout: "uptime": 268085, 2024-04-26T04:22:56.942 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum": [ 2024-04-26T04:22:56.942 INFO:teuthology.orchestra.run.smithi063.stdout: 0, 2024-04-26T04:22:56.942 INFO:teuthology.orchestra.run.smithi063.stdout: 2 2024-04-26T04:22:56.942 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:22:56.942 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_age": 0, 2024-04-26T04:22:56.943 INFO:teuthology.orchestra.run.smithi063.stdout: "features": { 2024-04-26T04:22:56.943 INFO:teuthology.orchestra.run.smithi063.stdout: "required_con": "2450521708007866372", 2024-04-26T04:22:56.943 INFO:teuthology.orchestra.run.smithi063.stdout: "required_mon": [ 2024-04-26T04:22:56.943 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:22:56.943 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:22:56.943 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:22:56.943 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:22:56.943 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:22:56.943 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:22:56.943 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:22:56.943 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:22:56.943 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:22:56.943 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:22:56.943 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:22:56.944 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:22:56.944 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_con": "4540701547738038271", 2024-04-26T04:22:56.944 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_mon": [ 2024-04-26T04:22:56.944 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:22:56.944 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:22:56.944 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:22:56.944 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:22:56.944 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:22:56.944 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:22:56.944 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:22:56.944 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:22:56.944 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:22:56.944 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:22:56.944 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:22:56.944 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:22:56.944 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:56.945 INFO:teuthology.orchestra.run.smithi063.stdout: "outside_quorum": [], 2024-04-26T04:22:56.945 INFO:teuthology.orchestra.run.smithi063.stdout: "extra_probe_peers": [], 2024-04-26T04:22:56.945 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_provider": [], 2024-04-26T04:22:56.945 INFO:teuthology.orchestra.run.smithi063.stdout: "monmap": { 2024-04-26T04:22:56.945 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 1, 2024-04-26T04:22:56.945 INFO:teuthology.orchestra.run.smithi063.stdout: "fsid": "cd1c6a0c-4e4b-4014-89c0-2e535479ef39", 2024-04-26T04:22:56.945 INFO:teuthology.orchestra.run.smithi063.stdout: "modified": "2024-04-26T04:18:03.702339Z", 2024-04-26T04:22:56.945 INFO:teuthology.orchestra.run.smithi063.stdout: "created": "2024-04-26T04:18:03.702339Z", 2024-04-26T04:22:56.945 INFO:teuthology.orchestra.run.smithi063.stdout: "min_mon_release": 19, 2024-04-26T04:22:56.945 INFO:teuthology.orchestra.run.smithi063.stdout: "min_mon_release_name": "squid", 2024-04-26T04:22:56.945 INFO:teuthology.orchestra.run.smithi063.stdout: "election_strategy": 1, 2024-04-26T04:22:56.945 INFO:teuthology.orchestra.run.smithi063.stdout: "disallowed_leaders: ": "", 2024-04-26T04:22:56.945 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": false, 2024-04-26T04:22:56.945 INFO:teuthology.orchestra.run.smithi063.stdout: "tiebreaker_mon": "", 2024-04-26T04:22:56.946 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_ranks: ": "", 2024-04-26T04:22:56.946 INFO:teuthology.orchestra.run.smithi063.stdout: "features": { 2024-04-26T04:22:56.946 INFO:teuthology.orchestra.run.smithi063.stdout: "persistent": [ 2024-04-26T04:22:56.946 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:22:56.946 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:22:56.946 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:22:56.946 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:22:56.946 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:22:56.946 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:22:56.946 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:22:56.946 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:22:56.946 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:22:56.946 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:22:56.947 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:22:56.947 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:22:56.947 INFO:teuthology.orchestra.run.smithi063.stdout: "optional": [] 2024-04-26T04:22:56.947 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:56.947 INFO:teuthology.orchestra.run.smithi063.stdout: "mons": [ 2024-04-26T04:22:56.947 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:56.947 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 0, 2024-04-26T04:22:56.947 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a", 2024-04-26T04:22:56.947 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:22:56.947 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:22:56.947 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:56.947 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:22:56.947 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:3300", 2024-04-26T04:22:56.948 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:22:56.948 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:56.948 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:56.948 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:22:56.948 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6789", 2024-04-26T04:22:56.948 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:22:56.948 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:22:56.948 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:22:56.948 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:56.948 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6789/0", 2024-04-26T04:22:56.948 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6789/0", 2024-04-26T04:22:56.948 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:22:56.948 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:22:56.948 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:22:56.948 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:56.948 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:56.949 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 1, 2024-04-26T04:22:56.949 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b", 2024-04-26T04:22:56.949 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:22:56.949 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:22:56.949 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:56.949 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:22:56.949 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:3300", 2024-04-26T04:22:56.949 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:22:56.949 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:56.949 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:56.949 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:22:56.949 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:6789", 2024-04-26T04:22:56.949 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:22:56.949 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:22:56.949 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:22:56.949 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:56.950 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:6789/0", 2024-04-26T04:22:56.950 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.186:6789/0", 2024-04-26T04:22:56.950 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:22:56.950 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:22:56.950 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:22:56.950 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:56.950 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:56.950 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 2, 2024-04-26T04:22:56.950 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "c", 2024-04-26T04:22:56.950 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:22:56.950 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:22:56.950 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:56.951 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:22:56.951 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:3301", 2024-04-26T04:22:56.951 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:22:56.951 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:56.951 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:56.951 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:22:56.951 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6790", 2024-04-26T04:22:56.951 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:22:56.951 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:22:56.951 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:22:56.951 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:56.951 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6790/0", 2024-04-26T04:22:56.951 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6790/0", 2024-04-26T04:22:56.951 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:22:56.951 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:22:56.951 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:22:56.952 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:22:56.952 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:22:56.952 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:56.952 INFO:teuthology.orchestra.run.smithi063.stdout: "feature_map": { 2024-04-26T04:22:56.952 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": [ 2024-04-26T04:22:56.952 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:56.952 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:22:56.952 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:22:56.952 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 1 2024-04-26T04:22:56.952 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:22:56.952 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:22:56.952 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": [ 2024-04-26T04:22:56.952 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:56.952 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:22:56.952 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:22:56.952 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 3 2024-04-26T04:22:56.953 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:22:56.953 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:22:56.953 INFO:teuthology.orchestra.run.smithi063.stdout: "client": [ 2024-04-26T04:22:56.953 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:56.953 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:22:56.953 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:22:56.953 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 4 2024-04-26T04:22:56.953 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:22:56.953 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:22:56.953 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": [ 2024-04-26T04:22:56.953 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:56.953 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:22:56.953 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:22:56.953 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 1 2024-04-26T04:22:56.953 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:22:56.954 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:22:56.954 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:56.954 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": false 2024-04-26T04:22:56.954 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-26T04:22:56.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.932+0000 7f5945de2640 1 -- 172.21.15.63:0/1464603515 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f592c035ec0 msgr2=0x7f592c038380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:56.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.932+0000 7f5945de2640 1 --2- 172.21.15.63:0/1464603515 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f592c035ec0 0x7f592c038380 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f594012fb80 tx=0x7f5938003800 comp rx=0 tx=0).stop 2024-04-26T04:22:56.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.932+0000 7f5945de2640 10 monclient: shutdown 2024-04-26T04:22:56.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.932+0000 7f5945de2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:22:56.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.932+0000 7f5945de2640 1 -- 172.21.15.63:0/1464603515 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f5940137320 msgr2=0x7f5940135710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:56.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.932+0000 7f5945de2640 1 --2- 172.21.15.63:0/1464603515 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f5940137320 0x7f5940135710 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f593401a810 tx=0x7f593401a9a0 comp rx=0 tx=0).stop 2024-04-26T04:22:56.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.932+0000 7f593ffff640 1 -- 172.21.15.63:0/1464603515 reap_dead start 2024-04-26T04:22:56.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.932+0000 7f593ffff640 1 -- 172.21.15.63:0/1464603515 reap_dead start 2024-04-26T04:22:56.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.932+0000 7f5945de2640 1 -- 172.21.15.63:0/1464603515 shutdown_connections 2024-04-26T04:22:56.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.932+0000 7f5945de2640 1 -- 172.21.15.63:0/1464603515 >> 172.21.15.63:0/1464603515 conn(0x7f59400a1e70 msgr2=0x7f59401365c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:22:56.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.932+0000 7f5945de2640 1 -- 172.21.15.63:0/1464603515 shutdown_connections 2024-04-26T04:22:56.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:56.932+0000 7f5945de2640 1 -- 172.21.15.63:0/1464603515 wait complete. 2024-04-26T04:22:56.957 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-26T04:22:56.957 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-04-26T04:22:56.957 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T04:22:57.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.072+0000 7f03628e9640 10 monclient: get_monmap_and_config 2024-04-26T04:22:57.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.072+0000 7f03628e9640 10 monclient: build_initial_monmap 2024-04-26T04:22:57.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.072+0000 7f03628e9640 10 monclient: monmap: 2024-04-26T04:22:57.078 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:22:57.078 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:57.078 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:22:57.077781+0000 2024-04-26T04:22:57.078 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:22:57.077781+0000 2024-04-26T04:22:57.078 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:22:57.078 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:57.078 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:22:57.078 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:22:57.078 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:22:57.079 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:22:57.079 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:22:57.079 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:57.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.076+0000 7f03628e9640 1 Processor -- start 2024-04-26T04:22:57.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.076+0000 7f03628e9640 1 -- start start 2024-04-26T04:22:57.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.076+0000 7f03628e9640 10 monclient: init 2024-04-26T04:22:57.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.076+0000 7f03628e9640 10 monclient: _reopen_session rank -1 2024-04-26T04:22:57.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.076+0000 7f03628e9640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-04-26T04:22:57.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.076+0000 7f03628e9640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f035c0a0bb0 0x7f035c09eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:57.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.076+0000 7f03628e9640 10 monclient(hunting): picked mon.noname-b con 0x7f035c0a0bb0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:22:57.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.076+0000 7f03628e9640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f035c09d7b0 0x7f035c09db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:57.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.076+0000 7f03628e9640 10 monclient(hunting): picked mon.noname-c con 0x7f035c09d7b0 addr v2:172.21.15.186:3300/0 2024-04-26T04:22:57.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.076+0000 7f03628e9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f035c09e0d0 addr v1:172.21.15.63:6789/0 2024-04-26T04:22:57.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.076+0000 7f03628e9640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f035c0a0f90 con 0x7f035c09e0d0 2024-04-26T04:22:57.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.076+0000 7f03628e9640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:57.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.076+0000 7f03628e9640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f035c099240 con 0x7f035c09d7b0 2024-04-26T04:22:57.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.076+0000 7f03628e9640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:57.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.076+0000 7f03628e9640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f035c09b820 con 0x7f035c0a0bb0 2024-04-26T04:22:57.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.076+0000 7f03628e9640 10 monclient(hunting): _renew_subs 2024-04-26T04:22:57.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.076+0000 7f03628e9640 10 monclient(hunting): authenticate will time out at 1208.785358s 2024-04-26T04:22:57.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.076+0000 7f035bfff640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f035c0a0bb0 0x7f035c09eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:57.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.076+0000 7f035bfff640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f035c0a0bb0 0x7f035c09eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:42374/0 (socket says 172.21.15.63:42374) 2024-04-26T04:22:57.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.076+0000 7f035bfff640 1 -- 172.21.15.63:0/753594816 learned_addr learned my addr 172.21.15.63:0/753594816 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:22:57.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.076+0000 7f035bfff640 10 monclient(hunting): get_auth_request con 0x7f035c0a0bb0 auth_method 0 2024-04-26T04:22:57.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.076+0000 7f035bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:57.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.076+0000 7f035bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:57.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.076+0000 7f035bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:57.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.076+0000 7f035bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:22:57.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.076+0000 7f035b7fe640 1 -- 172.21.15.63:0/753594816 >> v2:172.21.15.186:3300/0 conn(0x7f035c09d7b0 msgr2=0x7f035c09db90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.186:3300/0 2024-04-26T04:22:57.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.076+0000 7f035bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:22:57.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.076+0000 7f035bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:22:57.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.076+0000 7f035b7fe640 1 --2- 172.21.15.63:0/753594816 >> v2:172.21.15.186:3300/0 conn(0x7f035c09d7b0 0x7f035c09db90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:22:57.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.076+0000 7f035bfff640 10 monclient(hunting): handle_auth_done global_id 24197 payload 274 2024-04-26T04:22:57.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.076+0000 7f035bfff640 1 -- 172.21.15.63:0/753594816 >> v2:172.21.15.186:3300/0 conn(0x7f035c09d7b0 msgr2=0x7f035c09db90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T04:22:57.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.076+0000 7f035bfff640 1 --2- 172.21.15.63:0/753594816 >> v2:172.21.15.186:3300/0 conn(0x7f035c09d7b0 0x7f035c09db90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:57.083 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.076+0000 7f035bfff640 1 -- 172.21.15.63:0/753594816 >> v1:172.21.15.63:6789/0 conn(0x7f035c09e0d0 legacy=0x7f035c13a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:22:57.083 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.076+0000 7f035bfff640 10 monclient: _finish_hunting 0 2024-04-26T04:22:57.083 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.076+0000 7f035bfff640 1 monclient: found mon.noname-b 2024-04-26T04:22:57.083 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.076+0000 7f035bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:22:57.083 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.076+0000 7f035bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.63:3301/0 2024-04-26T04:22:57.083 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.076+0000 7f035bfff640 1 -- 172.21.15.63:0/753594816 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f035c0994e0 con 0x7f035c0a0bb0 2024-04-26T04:22:57.084 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.080+0000 7f035bfff640 1 -- 172.21.15.63:0/753594816 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f035c0a0bb0 msgr2=0x7f035c09eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:22:57.084 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.080+0000 7f035bfff640 1 --2- 172.21.15.63:0/753594816 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f035c0a0bb0 0x7f035c09eb60 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f034c009870 tx=0x7f034c01cc50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cdbb8501ed47dc19 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:22:57.084 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.080+0000 7f035affd640 1 -- 172.21.15.63:0/753594816 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f034c0b0070 con 0x7f035c0a0bb0 2024-04-26T04:22:57.084 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.080+0000 7f035affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:57.084 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.080+0000 7f035affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T04:22:57.084 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.080+0000 7f035affd640 10 monclient: dump: 2024-04-26T04:22:57.084 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:57.084 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:57.084 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:57.084 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:57.085 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:57.085 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:57.085 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:57.085 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:57.085 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:57.085 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:57.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.080+0000 7f035affd640 10 monclient: _finish_auth 0 2024-04-26T04:22:57.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.080+0000 7f035affd640 10 monclient: _check_auth_tickets 2024-04-26T04:22:57.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.080+0000 7f035affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:22:57.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.080+0000 7f035affd640 1 -- 172.21.15.63:0/753594816 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f034c0ab600 con 0x7f035c0a0bb0 2024-04-26T04:22:57.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.080+0000 7f035affd640 10 monclient: handle_config config(0 keys) 2024-04-26T04:22:57.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.080+0000 7f035affd640 1 -- 172.21.15.63:0/753594816 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f034c0ab9d0 con 0x7f035c0a0bb0 2024-04-26T04:22:57.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.080+0000 7f03628e9640 5 monclient: authenticate success, global_id 24197 2024-04-26T04:22:57.085 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.080+0000 7f03628e9640 10 monclient: get_monmap_and_config success 2024-04-26T04:22:57.086 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.080+0000 7f03628e9640 4 set_mon_vals no callback set 2024-04-26T04:22:57.086 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.080+0000 7f035affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:57.086 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.080+0000 7f035affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:22:57.086 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.080+0000 7f035affd640 10 monclient: dump: 2024-04-26T04:22:57.086 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:57.086 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:57.086 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:57.086 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:57.086 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:57.086 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:57.086 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:57.086 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:57.086 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:57.086 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:57.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.080+0000 7f03628e9640 10 monclient: shutdown 2024-04-26T04:22:57.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.080+0000 7f03628e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:22:57.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.080+0000 7f03628e9640 1 -- 172.21.15.63:0/753594816 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f035c0a0bb0 msgr2=0x7f035c09eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:57.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.080+0000 7f03628e9640 1 --2- 172.21.15.63:0/753594816 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f035c0a0bb0 0x7f035c09eb60 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f034c009870 tx=0x7f034c01cc50 comp rx=0 tx=0).stop 2024-04-26T04:22:57.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.080+0000 7f03628e9640 1 -- 172.21.15.63:0/753594816 shutdown_connections 2024-04-26T04:22:57.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.080+0000 7f03628e9640 1 --2- 172.21.15.63:0/753594816 >> v2:172.21.15.186:3300/0 conn(0x7f035c09d7b0 0x7f035c09db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:57.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.080+0000 7f03628e9640 1 --2- 172.21.15.63:0/753594816 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f035c0a0bb0 0x7f035c09eb60 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:57.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.080+0000 7f03628e9640 1 -- 172.21.15.63:0/753594816 >> 172.21.15.63:0/753594816 conn(0x7f035c0a1e70 msgr2=0x7f035c0a2280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:22:57.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.080+0000 7f03628e9640 1 -- 172.21.15.63:0/753594816 shutdown_connections 2024-04-26T04:22:57.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.080+0000 7f03628e9640 1 -- 172.21.15.63:0/753594816 wait complete. 2024-04-26T04:22:57.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.080+0000 7f03628e9640 10 monclient: build_initial_monmap 2024-04-26T04:22:57.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.080+0000 7f03628e9640 10 monclient: monmap: 2024-04-26T04:22:57.092 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:22:57.093 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:22:57.093 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:22:57.093 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:22:57.093 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:22:57.093 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:57.093 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:22:57.093 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:22:57.093 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:22:57.093 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:57.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.080+0000 7f03628e9640 1 Processor -- start 2024-04-26T04:22:57.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.080+0000 7f03628e9640 1 -- start start 2024-04-26T04:22:57.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.080+0000 7f03628e9640 10 monclient: init 2024-04-26T04:22:57.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.084+0000 7f03628e9640 10 monclient: _reopen_session rank -1 2024-04-26T04:22:57.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.084+0000 7f03628e9640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T04:22:57.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.084+0000 7f03628e9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f035c09d7b0 0x7f035c136540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:57.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.084+0000 7f03628e9640 10 monclient(hunting): picked mon.noname-b con 0x7f035c09d7b0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:22:57.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.084+0000 7f03628e9640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f035c09e0d0 0x7f035c136a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:57.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.084+0000 7f03628e9640 10 monclient(hunting): picked mon.noname-a con 0x7f035c09e0d0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:22:57.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.084+0000 7f03628e9640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f035c0a0bb0 0x7f035c13ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:57.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.084+0000 7f03628e9640 10 monclient(hunting): picked mon.noname-c con 0x7f035c0a0bb0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:22:57.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.084+0000 7f03628e9640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:57.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.084+0000 7f03628e9640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f035c13c3e0 con 0x7f035c09e0d0 2024-04-26T04:22:57.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.084+0000 7f03628e9640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:57.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.084+0000 7f03628e9640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f035c13c260 con 0x7f035c09d7b0 2024-04-26T04:22:57.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.084+0000 7f03628e9640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:57.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.084+0000 7f03628e9640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f035c13c560 con 0x7f035c0a0bb0 2024-04-26T04:22:57.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.084+0000 7f03628e9640 10 monclient(hunting): _renew_subs 2024-04-26T04:22:57.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.084+0000 7f03628e9640 10 monclient(hunting): authenticate will time out at 1208.792407s 2024-04-26T04:22:57.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.084+0000 7f0360e86640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f035c0a0bb0 0x7f035c13ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:57.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.084+0000 7f035bfff640 1 -- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f035c09d7b0 msgr2=0x7f035c136540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.186:3300/0 2024-04-26T04:22:57.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.084+0000 7f035b7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f035c09e0d0 0x7f035c136a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:57.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.084+0000 7f035bfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f035c09d7b0 0x7f035c136540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:22:57.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.084+0000 7f0360e86640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f035c0a0bb0 0x7f035c13ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:42378/0 (socket says 172.21.15.63:42378) 2024-04-26T04:22:57.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.084+0000 7f0360e86640 1 -- 172.21.15.63:0/392812352 learned_addr learned my addr 172.21.15.63:0/392812352 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:22:57.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.084+0000 7f0360e86640 10 monclient(hunting): get_auth_request con 0x7f035c0a0bb0 auth_method 0 2024-04-26T04:22:57.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.084+0000 7f0360e86640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:57.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.084+0000 7f0360e86640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:57.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.084+0000 7f0360e86640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:57.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.084+0000 7f035b7fe640 10 monclient(hunting): get_auth_request con 0x7f035c09e0d0 auth_method 0 2024-04-26T04:22:57.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.084+0000 7f035b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:57.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.084+0000 7f035b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:57.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.084+0000 7f035b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:57.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.084+0000 7f0360e86640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:22:57.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.084+0000 7f0360e86640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:22:57.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.084+0000 7f0360e86640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:22:57.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.084+0000 7f035b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:22:57.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.084+0000 7f035b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:22:57.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.084+0000 7f035b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:22:57.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.084+0000 7f0360e86640 10 monclient(hunting): handle_auth_done global_id 24200 payload 824 2024-04-26T04:22:57.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.084+0000 7f0360e86640 1 -- 172.21.15.63:0/392812352 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f035c09d7b0 msgr2=0x7f035c136540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T04:22:57.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.084+0000 7f0360e86640 1 --2- 172.21.15.63:0/392812352 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f035c09d7b0 0x7f035c136540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:57.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.084+0000 7f0360e86640 1 -- 172.21.15.63:0/392812352 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f035c09e0d0 msgr2=0x7f035c136a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:57.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.084+0000 7f0360e86640 1 --2- 172.21.15.63:0/392812352 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f035c09e0d0 0x7f035c136a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:57.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.084+0000 7f0360e86640 10 monclient: _finish_hunting 0 2024-04-26T04:22:57.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.084+0000 7f0360e86640 1 monclient: found mon.noname-c 2024-04-26T04:22:57.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.084+0000 7f0360e86640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:22:57.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.084+0000 7f0360e86640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.63:3301/0 2024-04-26T04:22:57.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.084+0000 7f0360e86640 1 -- 172.21.15.63:0/392812352 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f035c09ad50 con 0x7f035c0a0bb0 2024-04-26T04:22:57.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.084+0000 7f0360e86640 1 -- 172.21.15.63:0/392812352 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f035c0a0bb0 msgr2=0x7f035c13ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:22:57.098 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.084+0000 7f0360e86640 1 --2- 172.21.15.63:0/392812352 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f035c0a0bb0 0x7f035c13ca60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f035400c780 tx=0x7f035400cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:22:57.098 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.084+0000 7f0358ff9640 1 -- 172.21.15.63:0/392812352 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f035400ac60 con 0x7f035c0a0bb0 2024-04-26T04:22:57.098 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.084+0000 7f0358ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:57.098 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.084+0000 7f0358ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:22:57.098 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.084+0000 7f0358ff9640 10 monclient: dump: 2024-04-26T04:22:57.098 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:57.098 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:57.098 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:57.098 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:57.098 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:57.098 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:57.098 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:57.098 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:57.098 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:57.099 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:57.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.084+0000 7f0358ff9640 10 monclient: _finish_auth 0 2024-04-26T04:22:57.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.084+0000 7f0358ff9640 10 monclient: _check_auth_tickets 2024-04-26T04:22:57.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.084+0000 7f0358ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:22:57.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.084+0000 7f0358ff9640 1 -- 172.21.15.63:0/392812352 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f035400ae00 con 0x7f035c0a0bb0 2024-04-26T04:22:57.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.084+0000 7f03628e9640 5 monclient: authenticate success, global_id 24200 2024-04-26T04:22:57.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.084+0000 7f03628e9640 10 monclient: _renew_subs 2024-04-26T04:22:57.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.084+0000 7f03628e9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:22:57.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.084+0000 7f03628e9640 1 -- 172.21.15.63:0/392812352 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f035c09ba60 con 0x7f035c0a0bb0 2024-04-26T04:22:57.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.084+0000 7f0358ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T04:22:57.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.084+0000 7f0358ff9640 1 -- 172.21.15.63:0/392812352 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0354004b60 con 0x7f035c0a0bb0 2024-04-26T04:22:57.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.084+0000 7f0358ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:57.100 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.084+0000 7f0358ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:22:57.100 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.084+0000 7f0358ff9640 10 monclient: dump: 2024-04-26T04:22:57.100 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:57.100 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:57.100 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:57.100 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:57.100 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:57.100 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:57.100 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:57.100 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:57.100 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:57.100 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:57.100 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.084+0000 7f03628e9640 10 monclient: _renew_subs 2024-04-26T04:22:57.100 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.084+0000 7f03628e9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:22:57.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.084+0000 7f035a7fc640 4 set_mon_vals no callback set 2024-04-26T04:22:57.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.084+0000 7f03628e9640 1 -- 172.21.15.63:0/392812352 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f035c13b1f0 con 0x7f035c0a0bb0 2024-04-26T04:22:57.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.088+0000 7f03628e9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:22:57.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.088+0000 7f03628e9640 1 --2- 172.21.15.63:0/392812352 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f035c1cf050 0x7f035c12ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:57.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.088+0000 7f03628e9640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:57.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.088+0000 7f03628e9640 1 -- 172.21.15.63:0/392812352 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f035c0504d0 con 0x7f035c1cf050 2024-04-26T04:22:57.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.088+0000 7f035bfff640 1 --2- 172.21.15.63:0/392812352 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f035c1cf050 0x7f035c12ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:57.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.088+0000 7f035bfff640 10 monclient: get_auth_request con 0x7f035c1cf050 auth_method 0 2024-04-26T04:22:57.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.088+0000 7f035bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:57.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.088+0000 7f035bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:57.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.088+0000 7f035bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:57.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.088+0000 7f0358ff9640 1 -- 172.21.15.63:0/392812352 <== mon.2 v2:172.21.15.63:3301/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f0354005030 con 0x7f035c0a0bb0 2024-04-26T04:22:57.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.088+0000 7f035bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:22:57.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.088+0000 7f035bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:22:57.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.088+0000 7f035bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:22:57.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.088+0000 7f035bfff640 10 monclient(hunting): handle_auth_done global_id 24203 payload 824 2024-04-26T04:22:57.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.088+0000 7f035bfff640 1 -- 172.21.15.63:0/392812352 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f035c062e10 con 0x7f035c1cf050 2024-04-26T04:22:57.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.088+0000 7f0358ff9640 1 --2- 172.21.15.63:0/392812352 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f0348035ec0 0x7f0348038380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:57.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.088+0000 7f0358ff9640 1 -- 172.21.15.63:0/392812352 <== mon.2 v2:172.21.15.63:3301/0 5 ==== osd_map(216..216 src has 1..216) ==== 4738+0+0 (secure 0 0 0) 0x7f0354048fc0 con 0x7f035c0a0bb0 2024-04-26T04:22:57.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.088+0000 7f035b7fe640 1 --2- 172.21.15.63:0/392812352 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f0348035ec0 0x7f0348038380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:57.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.088+0000 7f035b7fe640 10 monclient: get_auth_request con 0x7f0348035ec0 auth_method 0 2024-04-26T04:22:57.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.088+0000 7f035bfff640 1 -- 172.21.15.63:0/392812352 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f035c1cf050 msgr2=0x7f035c12ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:22:57.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.088+0000 7f035bfff640 1 --2- 172.21.15.63:0/392812352 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f035c1cf050 0x7f035c12ec30 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f034c0ad000 tx=0x7f034c018f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:22:57.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.088+0000 7f0358ff9640 1 -- 172.21.15.63:0/392812352 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f034c0bb020 con 0x7f035c1cf050 2024-04-26T04:22:57.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.088+0000 7f0358ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:57.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.088+0000 7f0358ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:22:57.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.088+0000 7f0358ff9640 10 monclient: dump: 2024-04-26T04:22:57.105 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:57.105 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:57.105 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:57.105 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:57.105 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:57.105 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:57.106 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:57.106 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:57.106 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:57.106 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:57.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.088+0000 7f0358ff9640 1 -- 172.21.15.63:0/392812352 <== mon.2 v2:172.21.15.63:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f034c0ab9b0 con 0x7f035c1cf050 2024-04-26T04:22:57.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.088+0000 7f0358ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:22:57.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.088+0000 7f0358ff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:22:57.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.088+0000 7f0358ff9640 1 -- 172.21.15.63:0/392812352 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f035c1cf050 msgr2=0x7f035c12ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:57.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.088+0000 7f0358ff9640 1 --2- 172.21.15.63:0/392812352 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f035c1cf050 0x7f035c12ec30 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f034c0ad000 tx=0x7f034c018f90 comp rx=0 tx=0).stop 2024-04-26T04:22:57.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.088+0000 7f0358ff9640 1 -- 172.21.15.63:0/392812352 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f035c1cf050 msgr2=0x7f035c12ec30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:22:57.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.088+0000 7f0358ff9640 1 --2- 172.21.15.63:0/392812352 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f035c1cf050 0x7f035c12ec30 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:57.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.092+0000 7f035b7fe640 1 --2- 172.21.15.63:0/392812352 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f0348035ec0 0x7f0348038380 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f03500093d0 tx=0x7f0350009360 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:22:57.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.096+0000 7f03628e9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T04:22:57.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.096+0000 7f03628e9640 1 --2- 172.21.15.63:0/392812352 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f035c13cfa0 0x7f035c13d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:22:57.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.096+0000 7f03628e9640 10 monclient(hunting): start opening mon connection 2024-04-26T04:22:57.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.096+0000 7f03628e9640 1 -- 172.21.15.63:0/392812352 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f035c13d8c0 con 0x7f035c13cfa0 2024-04-26T04:22:57.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.096+0000 7f035bfff640 1 --2- 172.21.15.63:0/392812352 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f035c13cfa0 0x7f035c13d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:22:57.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.096+0000 7f035bfff640 10 monclient: get_auth_request con 0x7f035c13cfa0 auth_method 0 2024-04-26T04:22:57.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.096+0000 7f035bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:22:57.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.096+0000 7f035bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:22:57.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.096+0000 7f035bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:22:57.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.096+0000 7f035bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:22:57.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.096+0000 7f035bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:22:57.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.096+0000 7f035bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:22:57.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.096+0000 7f035bfff640 10 monclient(hunting): handle_auth_done global_id 24206 payload 824 2024-04-26T04:22:57.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.096+0000 7f035bfff640 1 -- 172.21.15.63:0/392812352 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f035c068430 con 0x7f035c13cfa0 2024-04-26T04:22:57.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.100+0000 7f035bfff640 1 -- 172.21.15.63:0/392812352 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f035c13cfa0 msgr2=0x7f035c13d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:22:57.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.100+0000 7f035bfff640 1 --2- 172.21.15.63:0/392812352 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f035c13cfa0 0x7f035c13d380 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f034c01ca90 tx=0x7f034c009240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:22:57.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.100+0000 7f0358ff9640 1 -- 172.21.15.63:0/392812352 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f034c0bb050 con 0x7f035c13cfa0 2024-04-26T04:22:57.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.100+0000 7f0358ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:22:57.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.100+0000 7f0358ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:22:57.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.100+0000 7f0358ff9640 10 monclient: dump: 2024-04-26T04:22:57.108 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:22:57.108 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:22:57.108 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:57.108 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:22:57.108 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:22:57.108 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:22:57.109 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:22:57.109 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:22:57.109 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:22:57.109 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:22:57.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.100+0000 7f0358ff9640 1 -- 172.21.15.63:0/392812352 <== mon.2 v2:172.21.15.63:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4709 (secure 0 0 0) 0x7f034c003ce0 con 0x7f035c13cfa0 2024-04-26T04:22:57.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.100+0000 7f0358ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T04:22:57.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.100+0000 7f0358ff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:22:57.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.100+0000 7f0358ff9640 1 -- 172.21.15.63:0/392812352 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f035c13cfa0 msgr2=0x7f035c13d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:57.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.100+0000 7f0358ff9640 1 --2- 172.21.15.63:0/392812352 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f035c13cfa0 0x7f035c13d380 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f034c01ca90 tx=0x7f034c009240 comp rx=0 tx=0).stop 2024-04-26T04:22:57.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.100+0000 7f0358ff9640 1 -- 172.21.15.63:0/392812352 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f035c13cfa0 msgr2=0x7f035c13d380 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:22:57.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.100+0000 7f0358ff9640 1 --2- 172.21.15.63:0/392812352 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f035c13cfa0 0x7f035c13d380 secure :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f034c01ca90 tx=0x7f034c009240 comp rx=0 tx=0).stop 2024-04-26T04:22:57.110 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-26T04:22:57.110 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "c", 2024-04-26T04:22:57.110 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 2, 2024-04-26T04:22:57.110 INFO:teuthology.orchestra.run.smithi063.stdout: "state": "peon", 2024-04-26T04:22:57.110 INFO:teuthology.orchestra.run.smithi063.stdout: "election_epoch": 24, 2024-04-26T04:22:57.110 INFO:teuthology.orchestra.run.smithi063.stdout: "uptime": 24699, 2024-04-26T04:22:57.110 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum": [ 2024-04-26T04:22:57.110 INFO:teuthology.orchestra.run.smithi063.stdout: 0, 2024-04-26T04:22:57.110 INFO:teuthology.orchestra.run.smithi063.stdout: 2 2024-04-26T04:22:57.110 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:22:57.110 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_age": 1, 2024-04-26T04:22:57.110 INFO:teuthology.orchestra.run.smithi063.stdout: "features": { 2024-04-26T04:22:57.111 INFO:teuthology.orchestra.run.smithi063.stdout: "required_con": "2450521708007866372", 2024-04-26T04:22:57.111 INFO:teuthology.orchestra.run.smithi063.stdout: "required_mon": [ 2024-04-26T04:22:57.111 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:22:57.111 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:22:57.111 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:22:57.111 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:22:57.111 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:22:57.111 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:22:57.111 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:22:57.111 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:22:57.111 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:22:57.111 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:22:57.112 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:22:57.112 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:22:57.112 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_con": "4540701547738038271", 2024-04-26T04:22:57.112 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_mon": [ 2024-04-26T04:22:57.112 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:22:57.112 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:22:57.112 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:22:57.112 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:22:57.112 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:22:57.112 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:22:57.112 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:22:57.112 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:22:57.113 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:22:57.113 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:22:57.113 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:22:57.113 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:22:57.113 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:57.113 INFO:teuthology.orchestra.run.smithi063.stdout: "outside_quorum": [], 2024-04-26T04:22:57.114 INFO:teuthology.orchestra.run.smithi063.stdout: "extra_probe_peers": [], 2024-04-26T04:22:57.114 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_provider": [], 2024-04-26T04:22:57.114 INFO:teuthology.orchestra.run.smithi063.stdout: "monmap": { 2024-04-26T04:22:57.114 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 1, 2024-04-26T04:22:57.114 INFO:teuthology.orchestra.run.smithi063.stdout: "fsid": "cd1c6a0c-4e4b-4014-89c0-2e535479ef39", 2024-04-26T04:22:57.114 INFO:teuthology.orchestra.run.smithi063.stdout: "modified": "2024-04-26T04:18:03.702339Z", 2024-04-26T04:22:57.115 INFO:teuthology.orchestra.run.smithi063.stdout: "created": "2024-04-26T04:18:03.702339Z", 2024-04-26T04:22:57.115 INFO:teuthology.orchestra.run.smithi063.stdout: "min_mon_release": 19, 2024-04-26T04:22:57.115 INFO:teuthology.orchestra.run.smithi063.stdout: "min_mon_release_name": "squid", 2024-04-26T04:22:57.115 INFO:teuthology.orchestra.run.smithi063.stdout: "election_strategy": 1, 2024-04-26T04:22:57.115 INFO:teuthology.orchestra.run.smithi063.stdout: "disallowed_leaders: ": "", 2024-04-26T04:22:57.116 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": false, 2024-04-26T04:22:57.116 INFO:teuthology.orchestra.run.smithi063.stdout: "tiebreaker_mon": "", 2024-04-26T04:22:57.116 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_ranks: ": "", 2024-04-26T04:22:57.116 INFO:teuthology.orchestra.run.smithi063.stdout: "features": { 2024-04-26T04:22:57.116 INFO:teuthology.orchestra.run.smithi063.stdout: "persistent": [ 2024-04-26T04:22:57.116 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:22:57.117 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:22:57.117 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:22:57.117 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:22:57.117 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:22:57.117 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:22:57.117 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:22:57.117 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:22:57.118 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:22:57.118 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:22:57.118 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:22:57.118 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:22:57.118 INFO:teuthology.orchestra.run.smithi063.stdout: "optional": [] 2024-04-26T04:22:57.119 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:57.119 INFO:teuthology.orchestra.run.smithi063.stdout: "mons": [ 2024-04-26T04:22:57.119 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:57.119 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 0, 2024-04-26T04:22:57.119 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a", 2024-04-26T04:22:57.119 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:22:57.119 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:22:57.120 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:57.120 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:22:57.120 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:3300", 2024-04-26T04:22:57.120 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:22:57.120 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:57.120 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:57.121 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:22:57.121 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6789", 2024-04-26T04:22:57.121 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:22:57.121 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:22:57.121 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:22:57.121 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:57.122 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6789/0", 2024-04-26T04:22:57.122 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6789/0", 2024-04-26T04:22:57.122 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:22:57.122 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:22:57.122 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:22:57.122 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:57.123 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:57.123 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 1, 2024-04-26T04:22:57.123 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b", 2024-04-26T04:22:57.123 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:22:57.123 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:22:57.123 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:57.124 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:22:57.124 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:3300", 2024-04-26T04:22:57.124 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:22:57.124 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:57.124 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:57.125 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:22:57.125 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:6789", 2024-04-26T04:22:57.125 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:22:57.125 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:22:57.125 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:22:57.126 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:57.126 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:6789/0", 2024-04-26T04:22:57.126 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.186:6789/0", 2024-04-26T04:22:57.126 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:22:57.126 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:22:57.127 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:22:57.127 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:57.127 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:57.127 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 2, 2024-04-26T04:22:57.127 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "c", 2024-04-26T04:22:57.127 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:22:57.127 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:22:57.128 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:57.128 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:22:57.128 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:3301", 2024-04-26T04:22:57.128 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:22:57.128 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:57.128 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:57.129 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:22:57.129 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6790", 2024-04-26T04:22:57.129 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:22:57.129 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:22:57.129 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:22:57.129 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:57.130 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6790/0", 2024-04-26T04:22:57.130 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6790/0", 2024-04-26T04:22:57.130 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:22:57.130 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:22:57.130 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:22:57.131 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:22:57.131 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:22:57.131 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:57.131 INFO:teuthology.orchestra.run.smithi063.stdout: "feature_map": { 2024-04-26T04:22:57.131 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": [ 2024-04-26T04:22:57.131 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:57.132 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:22:57.132 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:22:57.132 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 1 2024-04-26T04:22:57.132 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:22:57.132 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:22:57.132 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": [ 2024-04-26T04:22:57.132 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:57.133 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:22:57.133 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:22:57.133 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 3 2024-04-26T04:22:57.133 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:22:57.133 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:22:57.133 INFO:teuthology.orchestra.run.smithi063.stdout: "client": [ 2024-04-26T04:22:57.134 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:22:57.134 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:22:57.134 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:22:57.134 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 5 2024-04-26T04:22:57.134 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:22:57.134 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:22:57.135 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:22:57.135 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": false 2024-04-26T04:22:57.135 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-26T04:22:57.136 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.100+0000 7f03628e9640 1 -- 172.21.15.63:0/392812352 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f0348035ec0 msgr2=0x7f0348038380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:57.136 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.100+0000 7f03628e9640 1 --2- 172.21.15.63:0/392812352 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f0348035ec0 0x7f0348038380 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f03500093d0 tx=0x7f0350009360 comp rx=0 tx=0).stop 2024-04-26T04:22:57.136 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.100+0000 7f03628e9640 10 monclient: shutdown 2024-04-26T04:22:57.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.100+0000 7f03628e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:22:57.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.100+0000 7f03628e9640 1 -- 172.21.15.63:0/392812352 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f035c0a0bb0 msgr2=0x7f035c13ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:22:57.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.100+0000 7f03628e9640 1 --2- 172.21.15.63:0/392812352 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f035c0a0bb0 0x7f035c13ca60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f035400c780 tx=0x7f035400cc40 comp rx=0 tx=0).stop 2024-04-26T04:22:57.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.100+0000 7f0360e86640 1 -- 172.21.15.63:0/392812352 reap_dead start 2024-04-26T04:22:57.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.100+0000 7f03628e9640 1 -- 172.21.15.63:0/392812352 shutdown_connections 2024-04-26T04:22:57.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.100+0000 7f03628e9640 1 --2- 172.21.15.63:0/392812352 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f035c0a0bb0 0x7f035c13ca60 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:22:57.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.100+0000 7f03628e9640 1 -- 172.21.15.63:0/392812352 >> 172.21.15.63:0/392812352 conn(0x7f035c0a1e70 msgr2=0x7f035c09c5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:22:57.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.100+0000 7f03628e9640 1 -- 172.21.15.63:0/392812352 shutdown_connections 2024-04-26T04:22:57.137 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:22:57.100+0000 7f03628e9640 1 -- 172.21.15.63:0/392812352 wait complete. 2024-04-26T04:22:57.137 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-26T04:22:57.137 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-04-26T04:22:58.219 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:22:58.220 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30535]: starting. 2024-04-26T04:22:58.220 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30535]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-30500 2024-04-26T04:22:58.220 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30535]: created object 0... 2024-04-26T04:22:58.220 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30535]: created object 25... 2024-04-26T04:22:58.220 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30535]: created object 49... 2024-04-26T04:22:58.220 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30535]: finishing. 2024-04-26T04:22:58.221 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30535]: shutting down. 2024-04-26T04:24:27.138 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-04-26T04:24:27.139 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-04-26T04:24:27.139 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-26T04:24:27.139 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-04-26T04:24:27.143 INFO:tasks.ceph.mon.b:Started 2024-04-26T04:24:27.143 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-26T04:24:27.143 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T04:24:27.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.271+0000 7fcc1c62d640 10 monclient: get_monmap_and_config 2024-04-26T04:24:27.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.271+0000 7fcc1c62d640 10 monclient: build_initial_monmap 2024-04-26T04:24:27.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.271+0000 7fcc1c62d640 10 monclient: monmap: 2024-04-26T04:24:27.276 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:24:27.276 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:27.276 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:24:27.275718+0000 2024-04-26T04:24:27.276 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:24:27.275718+0000 2024-04-26T04:24:27.276 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:24:27.276 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:27.276 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:24:27.276 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:24:27.276 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:24:27.276 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:24:27.276 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:24:27.276 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:27.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.271+0000 7fcc1c62d640 1 Processor -- start 2024-04-26T04:24:27.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.271+0000 7fcc1c62d640 1 -- start start 2024-04-26T04:24:27.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.271+0000 7fcc1c62d640 10 monclient: init 2024-04-26T04:24:27.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.271+0000 7fcc1c62d640 10 monclient: _reopen_session rank -1 2024-04-26T04:24:27.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.271+0000 7fcc1c62d640 10 monclient: _add_conns ranks=[1,0,3,2,4] 2024-04-26T04:24:27.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.271+0000 7fcc1c62d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcc140a2ce0 addr v1:172.21.15.186:6789/0 2024-04-26T04:24:27.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.271+0000 7fcc1c62d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcc140a37d0 addr v1:172.21.15.63:6789/0 2024-04-26T04:24:27.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.271+0000 7fcc1c62d640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7fcc1413d9f0 0x7fcc1413fde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:27.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.271+0000 7fcc1c62d640 10 monclient(hunting): picked mon.noname-a con 0x7fcc1413d9f0 addr v2:172.21.15.63:3300/0 2024-04-26T04:24:27.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.271+0000 7fcc1c62d640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc1409e160 con 0x7fcc140a37d0 2024-04-26T04:24:27.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.271+0000 7fcc1c62d640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc1409f0e0 con 0x7fcc140a2ce0 2024-04-26T04:24:27.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.271+0000 7fcc1c62d640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:27.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.271+0000 7fcc1c62d640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7fcc1409eca0 con 0x7fcc1413d9f0 2024-04-26T04:24:27.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.271+0000 7fcc1c62d640 10 monclient(hunting): _renew_subs 2024-04-26T04:24:27.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.271+0000 7fcc1c62d640 10 monclient(hunting): authenticate will time out at 1298.983455s 2024-04-26T04:24:27.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.271+0000 7fcc19bc8640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7fcc140a37d0 0x7fcc1413c2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:51148/0 (socket says 172.21.15.63:51148) 2024-04-26T04:24:27.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.271+0000 7fcc19bc8640 1 -- 172.21.15.63:0/3833434064 learned_addr learned my addr 172.21.15.63:0/3833434064 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-04-26T04:24:27.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.271+0000 7fcc1abca640 1 --2- 172.21.15.63:0/3833434064 >> v2:172.21.15.63:3300/0 conn(0x7fcc1413d9f0 0x7fcc1413fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:27.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.271+0000 7fcc1abca640 10 monclient(hunting): get_auth_request con 0x7fcc1413d9f0 auth_method 0 2024-04-26T04:24:27.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.271+0000 7fcc1abca640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:27.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.271+0000 7fcc1abca640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:27.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.271+0000 7fcc1abca640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:27.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.271+0000 7fcc1abca640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:27.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.271+0000 7fcc1abca640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:27.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.271+0000 7fcc1abca640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:24:27.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.275+0000 7fcc1abca640 10 monclient(hunting): handle_auth_done global_id 5298 payload 274 2024-04-26T04:24:27.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.275+0000 7fcc1abca640 1 -- 172.21.15.63:0/3833434064 >> v1:172.21.15.186:6789/0 conn(0x7fcc140a2ce0 legacy=0x7fcc140a30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:24:27.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.275+0000 7fcc1abca640 1 -- 172.21.15.63:0/3833434064 >> v1:172.21.15.63:6789/0 conn(0x7fcc140a37d0 legacy=0x7fcc1413c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:24:27.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.275+0000 7fcc1abca640 10 monclient: _finish_hunting 0 2024-04-26T04:24:27.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.275+0000 7fcc1abca640 1 monclient: found mon.noname-a 2024-04-26T04:24:27.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.275+0000 7fcc1abca640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:24:27.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.275+0000 7fcc1abca640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:24:27.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.275+0000 7fcc1abca640 1 -- 172.21.15.63:0/3833434064 --> v2:172.21.15.63:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc140a0c30 con 0x7fcc1413d9f0 2024-04-26T04:24:27.285 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.279+0000 7fcc1abca640 1 -- 172.21.15.63:0/3833434064 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcc1413d9f0 msgr2=0x7fcc1413fde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:24:27.285 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.279+0000 7fcc1abca640 1 --2- 172.21.15.63:0/3833434064 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcc1413d9f0 0x7fcc1413fde0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c009d90 tx=0x7fcc0c01c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e99160ab688175cb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:27.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc193c7640 1 -- 172.21.15.63:0/3833434064 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc0c0ac070 con 0x7fcc1413d9f0 2024-04-26T04:24:27.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc193c7640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:27.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc193c7640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:24:27.286 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc193c7640 10 monclient: dump: 2024-04-26T04:24:27.286 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:27.287 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:27.287 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:27.287 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:27.287 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:27.287 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:27.287 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:27.287 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:27.287 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:27.287 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:27.287 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc193c7640 10 monclient: _finish_auth 0 2024-04-26T04:24:27.287 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc193c7640 10 monclient: _check_auth_tickets 2024-04-26T04:24:27.287 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc193c7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:24:27.287 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc193c7640 1 -- 172.21.15.63:0/3833434064 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc0c0a6760 con 0x7fcc1413d9f0 2024-04-26T04:24:27.287 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc193c7640 10 monclient: handle_config config(0 keys) 2024-04-26T04:24:27.287 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc193c7640 1 -- 172.21.15.63:0/3833434064 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc0c0a6b30 con 0x7fcc1413d9f0 2024-04-26T04:24:27.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc1c62d640 5 monclient: authenticate success, global_id 5298 2024-04-26T04:24:27.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc1c62d640 10 monclient: get_monmap_and_config success 2024-04-26T04:24:27.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc1c62d640 4 set_mon_vals no callback set 2024-04-26T04:24:27.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc193c7640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:27.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc193c7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:24:27.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc193c7640 10 monclient: dump: 2024-04-26T04:24:27.288 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:27.288 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:27.288 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:27.288 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:27.288 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:27.288 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:27.288 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:27.288 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:27.288 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:27.289 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:27.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc1c62d640 10 monclient: shutdown 2024-04-26T04:24:27.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc1c62d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:24:27.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc1c62d640 1 -- 172.21.15.63:0/3833434064 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcc1413d9f0 msgr2=0x7fcc1413fde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:27.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc1c62d640 1 --2- 172.21.15.63:0/3833434064 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcc1413d9f0 0x7fcc1413fde0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c009d90 tx=0x7fcc0c01c7b0 comp rx=0 tx=0).stop 2024-04-26T04:24:27.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc1c62d640 1 -- 172.21.15.63:0/3833434064 shutdown_connections 2024-04-26T04:24:27.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc1c62d640 1 --2- 172.21.15.63:0/3833434064 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcc1413d9f0 0x7fcc1413fde0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:27.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc1c62d640 1 -- 172.21.15.63:0/3833434064 >> 172.21.15.63:0/3833434064 conn(0x7fcc140a78d0 msgr2=0x7fcc140a3dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:24:27.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc1c62d640 1 -- 172.21.15.63:0/3833434064 shutdown_connections 2024-04-26T04:24:27.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc1c62d640 1 -- 172.21.15.63:0/3833434064 wait complete. 2024-04-26T04:24:27.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc1c62d640 10 monclient: build_initial_monmap 2024-04-26T04:24:27.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc1c62d640 10 monclient: monmap: 2024-04-26T04:24:27.294 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:24:27.294 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:24:27.294 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:24:27.294 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:24:27.294 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:24:27.294 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:27.294 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:24:27.294 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:24:27.294 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:24:27.294 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:27.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc1c62d640 1 Processor -- start 2024-04-26T04:24:27.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc1c62d640 1 -- start start 2024-04-26T04:24:27.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc1c62d640 10 monclient: init 2024-04-26T04:24:27.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc1c62d640 10 monclient: _reopen_session rank -1 2024-04-26T04:24:27.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc1c62d640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T04:24:27.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc1c62d640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcc140a2ce0 0x7fcc141350c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:27.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc1c62d640 10 monclient(hunting): picked mon.noname-c con 0x7fcc140a2ce0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:24:27.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc1c62d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcc140a37d0 0x7fcc14135600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:27.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc1c62d640 10 monclient(hunting): picked mon.noname-a con 0x7fcc140a37d0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:24:27.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc1c62d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcc1413d9f0 0x7fcc1413b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:27.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc1c62d640 10 monclient(hunting): picked mon.noname-b con 0x7fcc1413d9f0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:24:27.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc1c62d640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:27.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc1c62d640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fcc14142cc0 con 0x7fcc140a37d0 2024-04-26T04:24:27.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc1c62d640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:27.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc1c62d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcc14142b40 con 0x7fcc1413d9f0 2024-04-26T04:24:27.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc1c62d640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:27.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc1c62d640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7fcc14142e40 con 0x7fcc140a2ce0 2024-04-26T04:24:27.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc1c62d640 10 monclient(hunting): _renew_subs 2024-04-26T04:24:27.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc1c62d640 10 monclient(hunting): authenticate will time out at 1298.995113s 2024-04-26T04:24:27.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc19bc8640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcc140a37d0 0x7fcc14135600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:27.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc19bc8640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcc140a37d0 0x7fcc14135600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:47684/0 (socket says 172.21.15.63:47684) 2024-04-26T04:24:27.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc19bc8640 1 -- 172.21.15.63:0/1568687599 learned_addr learned my addr 172.21.15.63:0/1568687599 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:24:27.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc19bc8640 10 monclient(hunting): get_auth_request con 0x7fcc140a37d0 auth_method 0 2024-04-26T04:24:27.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc19bc8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:27.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc19bc8640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:27.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc19bc8640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:27.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc1a3c9640 1 --2- 172.21.15.63:0/1568687599 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcc140a2ce0 0x7fcc141350c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:27.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc19bc8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:27.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc19bc8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:27.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc19bc8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:24:27.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc1a3c9640 10 monclient(hunting): get_auth_request con 0x7fcc140a2ce0 auth_method 0 2024-04-26T04:24:27.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc1a3c9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:27.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc1a3c9640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:27.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc1a3c9640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:27.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc1abca640 1 --2- 172.21.15.63:0/1568687599 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcc1413d9f0 0x7fcc1413b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:27.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc1abca640 10 monclient(hunting): get_auth_request con 0x7fcc1413d9f0 auth_method 0 2024-04-26T04:24:27.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc1abca640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:27.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc1abca640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:27.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc1abca640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:27.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc1a3c9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:27.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc1a3c9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:27.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc1a3c9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:24:27.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc19bc8640 10 monclient(hunting): handle_auth_done global_id 5301 payload 824 2024-04-26T04:24:27.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc19bc8640 1 -- 172.21.15.63:0/1568687599 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcc140a2ce0 msgr2=0x7fcc141350c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:27.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc19bc8640 1 --2- 172.21.15.63:0/1568687599 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcc140a2ce0 0x7fcc141350c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:27.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc19bc8640 1 -- 172.21.15.63:0/1568687599 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcc1413d9f0 msgr2=0x7fcc1413b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:27.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc19bc8640 1 --2- 172.21.15.63:0/1568687599 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcc1413d9f0 0x7fcc1413b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:27.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc19bc8640 10 monclient: _finish_hunting 0 2024-04-26T04:24:27.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc19bc8640 1 monclient: found mon.noname-a 2024-04-26T04:24:27.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc19bc8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:24:27.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc19bc8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:24:27.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.283+0000 7fcc19bc8640 1 -- 172.21.15.63:0/1568687599 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc1406e270 con 0x7fcc140a37d0 2024-04-26T04:24:27.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.287+0000 7fcc19bc8640 1 -- 172.21.15.63:0/1568687599 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcc140a37d0 msgr2=0x7fcc14135600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:24:27.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.287+0000 7fcc19bc8640 1 --2- 172.21.15.63:0/1568687599 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcc140a37d0 0x7fcc14135600 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fcc0400c470 tx=0x7fcc0400c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:27.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.287+0000 7fcc0affd640 1 -- 172.21.15.63:0/1568687599 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc04007070 con 0x7fcc140a37d0 2024-04-26T04:24:27.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.287+0000 7fcc0affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:27.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.287+0000 7fcc0affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:24:27.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.287+0000 7fcc0affd640 10 monclient: dump: 2024-04-26T04:24:27.299 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:27.299 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:27.299 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:27.299 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:27.299 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:27.299 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:27.299 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:27.299 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:27.299 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:27.299 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:27.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.287+0000 7fcc0affd640 10 monclient: _finish_auth 0 2024-04-26T04:24:27.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.287+0000 7fcc0affd640 10 monclient: _check_auth_tickets 2024-04-26T04:24:27.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.287+0000 7fcc0affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:24:27.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.287+0000 7fcc1c62d640 5 monclient: authenticate success, global_id 5301 2024-04-26T04:24:27.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.287+0000 7fcc0affd640 1 -- 172.21.15.63:0/1568687599 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc04007210 con 0x7fcc140a37d0 2024-04-26T04:24:27.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.287+0000 7fcc0affd640 10 monclient: handle_config config(0 keys) 2024-04-26T04:24:27.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.287+0000 7fcc0affd640 1 -- 172.21.15.63:0/1568687599 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc04007600 con 0x7fcc140a37d0 2024-04-26T04:24:27.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.287+0000 7fcc0affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:27.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.287+0000 7fcc0affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:24:27.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.287+0000 7fcc0affd640 10 monclient: dump: 2024-04-26T04:24:27.300 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:27.300 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:27.300 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:27.300 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:27.300 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:27.300 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:27.300 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:27.300 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:27.301 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:27.301 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:27.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.287+0000 7fcc18bc6640 4 set_mon_vals no callback set 2024-04-26T04:24:27.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.287+0000 7fcc1c62d640 10 monclient: _renew_subs 2024-04-26T04:24:27.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.287+0000 7fcc1c62d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:24:27.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.287+0000 7fcc1c62d640 1 -- 172.21.15.63:0/1568687599 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc1406fa00 con 0x7fcc140a37d0 2024-04-26T04:24:27.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.287+0000 7fcc1c62d640 10 monclient: _renew_subs 2024-04-26T04:24:27.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.287+0000 7fcc1c62d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:24:27.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.287+0000 7fcc1c62d640 1 -- 172.21.15.63:0/1568687599 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc1406f440 con 0x7fcc140a37d0 2024-04-26T04:24:27.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.287+0000 7fcc1c62d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:24:27.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.287+0000 7fcc1c62d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:24:27.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.287+0000 7fcc1c62d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:24:27.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.287+0000 7fcc1c62d640 1 -- 172.21.15.63:0/1568687599 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc14062e10 con 0x7fcc140a37d0 2024-04-26T04:24:27.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.287+0000 7fcc0affd640 1 -- 172.21.15.63:0/1568687599 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fcc0401b400 con 0x7fcc140a37d0 2024-04-26T04:24:27.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.287+0000 7fcc0affd640 1 --2- 172.21.15.63:0/1568687599 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fcbf8035ef0 0x7fcbf80383b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:27.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.291+0000 7fcc0affd640 1 -- 172.21.15.63:0/1568687599 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(219..219 src has 1..219) ==== 4738+0+0 (secure 0 0 0) 0x7fcc04048ba0 con 0x7fcc140a37d0 2024-04-26T04:24:27.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.291+0000 7fcc1a3c9640 1 --2- 172.21.15.63:0/1568687599 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fcbf8035ef0 0x7fcbf80383b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:27.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.291+0000 7fcc1a3c9640 10 monclient: get_auth_request con 0x7fcbf8035ef0 auth_method 0 2024-04-26T04:24:27.302 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.291+0000 7fcc1a3c9640 1 --2- 172.21.15.63:0/1568687599 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fcbf8035ef0 0x7fcbf80383b0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fcc10007fd0 tx=0x7fcc10007e40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:27.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.295+0000 7fcc0affd640 1 -- 172.21.15.63:0/1568687599 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fcc0401b980 con 0x7fcc140a37d0 2024-04-26T04:24:27.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.295+0000 7fcc0affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:24:27.303 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.295+0000 7fcc0affd640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:24:27.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.515+0000 7fcc1c62d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T04:24:27.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.515+0000 7fcc1c62d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T04:24:27.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.515+0000 7fcc1c62d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:24:27.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.515+0000 7fcc1c62d640 1 -- 172.21.15.63:0/1568687599 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fcc1413d7b0 con 0x7fcc140a37d0 2024-04-26T04:24:27.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.515+0000 7fcc0affd640 1 -- 172.21.15.63:0/1568687599 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7fcc04012400 con 0x7fcc140a37d0 2024-04-26T04:24:27.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.515+0000 7fcc0affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T04:24:27.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.515+0000 7fcc0affd640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:24:27.521 INFO:teuthology.orchestra.run.smithi063.stdout:{"election_epoch":24,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":91,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cd1c6a0c-4e4b-4014-89c0-2e535479ef39","modified":"2024-04-26T04:18:03.702339Z","created":"2024-04-26T04:18:03.702339Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3300","nonce":0},{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:3300","nonce":0},{"type":"v1","addr":"172.21.15.186:6789","nonce":0}]},"addr":"172.21.15.186:6789/0","public_addr":"172.21.15.186:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3301","nonce":0},{"type":"v1","addr":"172.21.15.63:6790","nonce":0}]},"addr":"172.21.15.63:6790/0","public_addr":"172.21.15.63:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T04:24:27.523 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.519+0000 7fcc1c62d640 1 -- 172.21.15.63:0/1568687599 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fcbf8035ef0 msgr2=0x7fcbf80383b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:27.523 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.519+0000 7fcc1c62d640 1 --2- 172.21.15.63:0/1568687599 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fcbf8035ef0 0x7fcbf80383b0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fcc10007fd0 tx=0x7fcc10007e40 comp rx=0 tx=0).stop 2024-04-26T04:24:27.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.519+0000 7fcc1c62d640 10 monclient: shutdown 2024-04-26T04:24:27.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.519+0000 7fcc1c62d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:24:27.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.519+0000 7fcc1c62d640 1 -- 172.21.15.63:0/1568687599 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcc140a37d0 msgr2=0x7fcc14135600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:27.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.519+0000 7fcc1c62d640 1 --2- 172.21.15.63:0/1568687599 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcc140a37d0 0x7fcc14135600 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fcc0400c470 tx=0x7fcc0400c930 comp rx=0 tx=0).stop 2024-04-26T04:24:27.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.519+0000 7fcc1c62d640 1 -- 172.21.15.63:0/1568687599 shutdown_connections 2024-04-26T04:24:27.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.519+0000 7fcc1c62d640 1 --2- 172.21.15.63:0/1568687599 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fcbf8035ef0 0x7fcbf80383b0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:27.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.519+0000 7fcc1c62d640 1 --2- 172.21.15.63:0/1568687599 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcc1413d9f0 0x7fcc1413b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:27.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.519+0000 7fcc1c62d640 1 --2- 172.21.15.63:0/1568687599 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcc140a37d0 0x7fcc14135600 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:27.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.519+0000 7fcc1c62d640 1 --2- 172.21.15.63:0/1568687599 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcc140a2ce0 0x7fcc141350c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:27.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.519+0000 7fcc1c62d640 1 -- 172.21.15.63:0/1568687599 >> 172.21.15.63:0/1568687599 conn(0x7fcc140a78d0 msgr2=0x7fcc1413f510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:24:27.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.519+0000 7fcc1c62d640 1 -- 172.21.15.63:0/1568687599 shutdown_connections 2024-04-26T04:24:27.524 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:27.519+0000 7fcc1c62d640 1 -- 172.21.15.63:0/1568687599 wait complete. 2024-04-26T04:24:30.541 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T04:24:30.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.667+0000 7fcf8df79640 10 monclient: get_monmap_and_config 2024-04-26T04:24:30.672 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.667+0000 7fcf8df79640 10 monclient: build_initial_monmap 2024-04-26T04:24:30.672 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.667+0000 7fcf8df79640 10 monclient: monmap: 2024-04-26T04:24:30.672 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:24:30.672 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:30.672 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:24:30.672072+0000 2024-04-26T04:24:30.672 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:24:30.672072+0000 2024-04-26T04:24:30.672 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:24:30.672 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:30.672 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:24:30.672 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:24:30.672 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:24:30.672 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:24:30.673 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:24:30.673 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:30.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.667+0000 7fcf8df79640 1 Processor -- start 2024-04-26T04:24:30.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.667+0000 7fcf8df79640 1 -- start start 2024-04-26T04:24:30.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.667+0000 7fcf8df79640 10 monclient: init 2024-04-26T04:24:30.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.667+0000 7fcf8df79640 10 monclient: _reopen_session rank -1 2024-04-26T04:24:30.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.667+0000 7fcf8df79640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-04-26T04:24:30.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.667+0000 7fcf8df79640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fcf880a2cd0 0x7fcf880a30b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:30.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.667+0000 7fcf8df79640 10 monclient(hunting): picked mon.noname-c con 0x7fcf880a2cd0 addr v2:172.21.15.186:3300/0 2024-04-26T04:24:30.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.667+0000 7fcf8df79640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcf880a47d0 addr v1:172.21.15.186:6789/0 2024-04-26T04:24:30.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.667+0000 7fcf8df79640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7fcf880a4c00 0x7fcf881373f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:30.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.667+0000 7fcf8df79640 10 monclient(hunting): picked mon.noname-a con 0x7fcf880a4c00 addr v2:172.21.15.63:3300/0 2024-04-26T04:24:30.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.667+0000 7fcf8df79640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf880a0fb0 con 0x7fcf880a47d0 2024-04-26T04:24:30.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.667+0000 7fcf8df79640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:30.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.667+0000 7fcf8df79640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7fcf880a35f0 con 0x7fcf880a4c00 2024-04-26T04:24:30.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.667+0000 7fcf8df79640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:30.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.667+0000 7fcf8df79640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fcf880a5380 con 0x7fcf880a2cd0 2024-04-26T04:24:30.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.667+0000 7fcf8df79640 10 monclient(hunting): _renew_subs 2024-04-26T04:24:30.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.667+0000 7fcf8df79640 10 monclient(hunting): authenticate will time out at 1302.379943s 2024-04-26T04:24:30.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.671+0000 7fcf87fff640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7fcf880a4c00 0x7fcf881373f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:30.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.671+0000 7fcf87fff640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7fcf880a4c00 0x7fcf881373f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:33176/0 (socket says 172.21.15.63:33176) 2024-04-26T04:24:30.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.671+0000 7fcf87fff640 1 -- 172.21.15.63:0/3541243809 learned_addr learned my addr 172.21.15.63:0/3541243809 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:24:30.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.671+0000 7fcf87fff640 10 monclient(hunting): get_auth_request con 0x7fcf880a4c00 auth_method 0 2024-04-26T04:24:30.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.671+0000 7fcf87fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:30.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.671+0000 7fcf87fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:30.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.671+0000 7fcf87fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:30.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.671+0000 7fcf87fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:30.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.671+0000 7fcf87fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:30.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.671+0000 7fcf87fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:24:30.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.671+0000 7fcf87fff640 10 monclient(hunting): handle_auth_done global_id 5304 payload 274 2024-04-26T04:24:30.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.671+0000 7fcf877fe640 1 --2- 172.21.15.63:0/3541243809 >> v2:172.21.15.186:3300/0 conn(0x7fcf880a2cd0 0x7fcf880a30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:30.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.671+0000 7fcf87fff640 1 -- 172.21.15.63:0/3541243809 >> v2:172.21.15.186:3300/0 conn(0x7fcf880a2cd0 msgr2=0x7fcf880a30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:24:30.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.671+0000 7fcf87fff640 1 --2- 172.21.15.63:0/3541243809 >> v2:172.21.15.186:3300/0 conn(0x7fcf880a2cd0 0x7fcf880a30b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:30.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.671+0000 7fcf87fff640 1 -- 172.21.15.63:0/3541243809 >> v1:172.21.15.186:6789/0 conn(0x7fcf880a47d0 legacy=0x7fcf88136ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:24:30.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.671+0000 7fcf87fff640 10 monclient: _finish_hunting 0 2024-04-26T04:24:30.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.671+0000 7fcf87fff640 1 monclient: found mon.noname-a 2024-04-26T04:24:30.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.671+0000 7fcf87fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:24:30.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.671+0000 7fcf87fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:24:30.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.671+0000 7fcf87fff640 1 -- 172.21.15.63:0/3541243809 --> v2:172.21.15.63:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf880a1290 con 0x7fcf880a4c00 2024-04-26T04:24:30.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.671+0000 7fcf877fe640 10 monclient: get_auth_request con 0x7fcf880a2cd0 auth_method 0 2024-04-26T04:24:30.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.671+0000 7fcf877fe640 1 --2- 172.21.15.63:0/3541243809 >> v2:172.21.15.186:3300/0 conn(0x7fcf880a2cd0 0x7fcf880a30b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T04:24:30.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf87fff640 1 -- 172.21.15.63:0/3541243809 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcf880a4c00 msgr2=0x7fcf881373f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:24:30.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf87fff640 1 --2- 172.21.15.63:0/3541243809 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcf880a4c00 0x7fcf881373f0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fcf8806c760 tx=0x7fcf8001c4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19fdf4ec77bfb82 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:30.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf867fc640 1 -- 172.21.15.63:0/3541243809 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf800b0070 con 0x7fcf880a4c00 2024-04-26T04:24:30.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:30.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf867fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:24:30.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf867fc640 10 monclient: dump: 2024-04-26T04:24:30.679 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:30.679 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:30.679 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:30.679 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:30.679 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:30.679 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:30.679 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:30.680 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:30.680 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:30.680 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:30.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf867fc640 10 monclient: _finish_auth 0 2024-04-26T04:24:30.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf867fc640 10 monclient: _check_auth_tickets 2024-04-26T04:24:30.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:24:30.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf867fc640 1 -- 172.21.15.63:0/3541243809 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf800ab600 con 0x7fcf880a4c00 2024-04-26T04:24:30.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf867fc640 10 monclient: handle_config config(0 keys) 2024-04-26T04:24:30.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf867fc640 1 -- 172.21.15.63:0/3541243809 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf800ab9d0 con 0x7fcf880a4c00 2024-04-26T04:24:30.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf8df79640 5 monclient: authenticate success, global_id 5304 2024-04-26T04:24:30.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf8df79640 10 monclient: get_monmap_and_config success 2024-04-26T04:24:30.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf8df79640 4 set_mon_vals no callback set 2024-04-26T04:24:30.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:30.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf867fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:24:30.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf867fc640 10 monclient: dump: 2024-04-26T04:24:30.681 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:30.681 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:30.681 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:30.681 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:30.681 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:30.681 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:30.681 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:30.681 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:30.681 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:30.681 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:30.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf8df79640 10 monclient: shutdown 2024-04-26T04:24:30.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf8df79640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:24:30.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf8df79640 1 -- 172.21.15.63:0/3541243809 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcf880a4c00 msgr2=0x7fcf881373f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:30.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf8df79640 1 --2- 172.21.15.63:0/3541243809 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcf880a4c00 0x7fcf881373f0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fcf8806c760 tx=0x7fcf8001c4c0 comp rx=0 tx=0).stop 2024-04-26T04:24:30.682 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf8df79640 1 -- 172.21.15.63:0/3541243809 shutdown_connections 2024-04-26T04:24:30.682 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf8df79640 1 --2- 172.21.15.63:0/3541243809 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcf880a4c00 0x7fcf881373f0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:30.682 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf8df79640 1 --2- 172.21.15.63:0/3541243809 >> v2:172.21.15.186:3300/0 conn(0x7fcf880a2cd0 0x7fcf880a30b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:30.682 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf8df79640 1 -- 172.21.15.63:0/3541243809 >> 172.21.15.63:0/3541243809 conn(0x7fcf880a5f40 msgr2=0x7fcf880a6320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:24:30.682 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf8df79640 1 -- 172.21.15.63:0/3541243809 shutdown_connections 2024-04-26T04:24:30.682 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf8df79640 1 -- 172.21.15.63:0/3541243809 wait complete. 2024-04-26T04:24:30.687 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf8df79640 10 monclient: build_initial_monmap 2024-04-26T04:24:30.687 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf8df79640 10 monclient: monmap: 2024-04-26T04:24:30.687 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:24:30.687 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:24:30.687 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:24:30.687 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:24:30.687 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:24:30.687 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:30.687 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:24:30.687 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:24:30.687 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:24:30.687 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:30.687 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf8df79640 1 Processor -- start 2024-04-26T04:24:30.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf8df79640 1 -- start start 2024-04-26T04:24:30.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf8df79640 10 monclient: init 2024-04-26T04:24:30.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf8df79640 10 monclient: _reopen_session rank -1 2024-04-26T04:24:30.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf8df79640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T04:24:30.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf8df79640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf880a2cd0 0x7fcf881cef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:30.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf8df79640 10 monclient(hunting): picked mon.noname-b con 0x7fcf880a2cd0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:24:30.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf8df79640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcf880a47d0 0x7fcf88135ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:30.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf8df79640 10 monclient(hunting): picked mon.noname-c con 0x7fcf880a47d0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:24:30.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf8df79640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcf880a4c00 0x7fcf8813bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:30.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf8df79640 10 monclient(hunting): picked mon.noname-a con 0x7fcf880a4c00 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:24:30.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf8df79640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:30.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf8df79640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fcf88141a50 con 0x7fcf880a4c00 2024-04-26T04:24:30.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf8df79640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:30.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf8df79640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcf88137b40 con 0x7fcf880a2cd0 2024-04-26T04:24:30.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf8df79640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:30.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf8df79640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7fcf88141bd0 con 0x7fcf880a47d0 2024-04-26T04:24:30.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf8df79640 10 monclient(hunting): _renew_subs 2024-04-26T04:24:30.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf8df79640 10 monclient(hunting): authenticate will time out at 1302.387007s 2024-04-26T04:24:30.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf86ffd640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcf880a47d0 0x7fcf88135ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:30.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf87fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcf880a4c00 0x7fcf8813bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:30.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf86ffd640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcf880a47d0 0x7fcf88135ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:40032/0 (socket says 172.21.15.63:40032) 2024-04-26T04:24:30.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf86ffd640 1 -- 172.21.15.63:0/1486353316 learned_addr learned my addr 172.21.15.63:0/1486353316 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:24:30.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf86ffd640 10 monclient(hunting): get_auth_request con 0x7fcf880a47d0 auth_method 0 2024-04-26T04:24:30.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf86ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:30.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf86ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:30.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:30.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf87fff640 10 monclient(hunting): get_auth_request con 0x7fcf880a4c00 auth_method 0 2024-04-26T04:24:30.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf87fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:30.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf87fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:30.690 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf87fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:30.690 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf86ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:30.690 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf86ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:30.690 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf86ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:24:30.690 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf87fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:30.690 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf87fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:30.690 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf87fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:24:30.690 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf877fe640 1 --2- 172.21.15.63:0/1486353316 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf880a2cd0 0x7fcf881cef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:30.690 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf877fe640 10 monclient(hunting): get_auth_request con 0x7fcf880a2cd0 auth_method 0 2024-04-26T04:24:30.690 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:30.690 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf877fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:30.690 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf877fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:30.690 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf87fff640 10 monclient(hunting): handle_auth_done global_id 5307 payload 824 2024-04-26T04:24:30.690 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf87fff640 1 -- 172.21.15.63:0/1486353316 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcf880a47d0 msgr2=0x7fcf88135ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:30.690 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf87fff640 1 --2- 172.21.15.63:0/1486353316 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcf880a47d0 0x7fcf88135ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:30.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf87fff640 1 -- 172.21.15.63:0/1486353316 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf880a2cd0 msgr2=0x7fcf881cef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:30.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf87fff640 1 --2- 172.21.15.63:0/1486353316 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf880a2cd0 0x7fcf881cef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:30.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf87fff640 10 monclient: _finish_hunting 0 2024-04-26T04:24:30.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf87fff640 1 monclient: found mon.noname-a 2024-04-26T04:24:30.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf87fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:24:30.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf87fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:24:30.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf87fff640 1 -- 172.21.15.63:0/1486353316 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf8813d880 con 0x7fcf880a4c00 2024-04-26T04:24:30.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.675+0000 7fcf86ffd640 1 --2- 172.21.15.63:0/1486353316 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcf880a47d0 0x7fcf88135ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T04:24:30.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.679+0000 7fcf87fff640 1 -- 172.21.15.63:0/1486353316 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcf880a4c00 msgr2=0x7fcf8813bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:24:30.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.679+0000 7fcf87fff640 1 --2- 172.21.15.63:0/1486353316 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcf880a4c00 0x7fcf8813bbe0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fcf800ad000 tx=0x7fcf80019260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:30.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.679+0000 7fcf7ffff640 1 -- 172.21.15.63:0/1486353316 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf800b0070 con 0x7fcf880a4c00 2024-04-26T04:24:30.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.679+0000 7fcf7ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:30.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.679+0000 7fcf7ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:24:30.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.679+0000 7fcf7ffff640 10 monclient: dump: 2024-04-26T04:24:30.692 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:30.692 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:30.692 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:30.692 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:30.692 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:30.692 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:30.692 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:30.692 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:30.692 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:30.692 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:30.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.679+0000 7fcf7ffff640 10 monclient: _finish_auth 0 2024-04-26T04:24:30.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.679+0000 7fcf7ffff640 10 monclient: _check_auth_tickets 2024-04-26T04:24:30.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.679+0000 7fcf7ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:24:30.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.679+0000 7fcf7ffff640 1 -- 172.21.15.63:0/1486353316 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf800b1040 con 0x7fcf880a4c00 2024-04-26T04:24:30.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.679+0000 7fcf7ffff640 10 monclient: handle_config config(0 keys) 2024-04-26T04:24:30.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.679+0000 7fcf7ffff640 1 -- 172.21.15.63:0/1486353316 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf800aa9e0 con 0x7fcf880a4c00 2024-04-26T04:24:30.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.679+0000 7fcf7ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:30.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.679+0000 7fcf7ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:24:30.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.679+0000 7fcf7ffff640 10 monclient: dump: 2024-04-26T04:24:30.693 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:30.693 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:30.693 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:30.693 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:30.693 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:30.693 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:30.693 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:30.693 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:30.693 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:30.693 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:30.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.679+0000 7fcf8df79640 5 monclient: authenticate success, global_id 5307 2024-04-26T04:24:30.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.679+0000 7fcf85ffb640 4 set_mon_vals no callback set 2024-04-26T04:24:30.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.679+0000 7fcf8df79640 10 monclient: _renew_subs 2024-04-26T04:24:30.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.679+0000 7fcf8df79640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:24:30.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.679+0000 7fcf8df79640 1 -- 172.21.15.63:0/1486353316 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf880a37d0 con 0x7fcf880a4c00 2024-04-26T04:24:30.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.679+0000 7fcf8df79640 10 monclient: _renew_subs 2024-04-26T04:24:30.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.679+0000 7fcf8df79640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:24:30.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.679+0000 7fcf8df79640 1 -- 172.21.15.63:0/1486353316 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf88120630 con 0x7fcf880a4c00 2024-04-26T04:24:30.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.679+0000 7fcf7ffff640 1 -- 172.21.15.63:0/1486353316 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fcf800aab80 con 0x7fcf880a4c00 2024-04-26T04:24:30.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.679+0000 7fcf8df79640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:24:30.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.679+0000 7fcf8df79640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:24:30.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.679+0000 7fcf8df79640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:24:30.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.679+0000 7fcf7ffff640 1 --2- 172.21.15.63:0/1486353316 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fcf68035ec0 0x7fcf68038380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:30.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.679+0000 7fcf8df79640 1 -- 172.21.15.63:0/1486353316 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf88062e10 con 0x7fcf880a4c00 2024-04-26T04:24:30.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.679+0000 7fcf7ffff640 1 -- 172.21.15.63:0/1486353316 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(219..219 src has 1..219) ==== 4738+0+0 (secure 0 0 0) 0x7fcf800e2c50 con 0x7fcf880a4c00 2024-04-26T04:24:30.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.679+0000 7fcf877fe640 1 --2- 172.21.15.63:0/1486353316 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fcf68035ec0 0x7fcf68038380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:30.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.679+0000 7fcf877fe640 10 monclient: get_auth_request con 0x7fcf68035ec0 auth_method 0 2024-04-26T04:24:30.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.683+0000 7fcf877fe640 1 --2- 172.21.15.63:0/1486353316 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fcf68035ec0 0x7fcf68038380 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fcf7000a990 tx=0x7fcf70008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:30.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.687+0000 7fcf7ffff640 1 -- 172.21.15.63:0/1486353316 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fcf800160b0 con 0x7fcf880a4c00 2024-04-26T04:24:30.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.687+0000 7fcf7ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:24:30.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.687+0000 7fcf7ffff640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:24:30.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.907+0000 7fcf8df79640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T04:24:30.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.907+0000 7fcf8df79640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T04:24:30.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.907+0000 7fcf8df79640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:24:30.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.907+0000 7fcf8df79640 1 -- 172.21.15.63:0/1486353316 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fcf8811fa90 con 0x7fcf880a4c00 2024-04-26T04:24:30.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.907+0000 7fcf7ffff640 1 -- 172.21.15.63:0/1486353316 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7fcf800a95a0 con 0x7fcf880a4c00 2024-04-26T04:24:30.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.907+0000 7fcf7ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T04:24:30.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.907+0000 7fcf7ffff640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:24:30.913 INFO:teuthology.orchestra.run.smithi063.stdout:{"election_epoch":24,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":94,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cd1c6a0c-4e4b-4014-89c0-2e535479ef39","modified":"2024-04-26T04:18:03.702339Z","created":"2024-04-26T04:18:03.702339Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3300","nonce":0},{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:3300","nonce":0},{"type":"v1","addr":"172.21.15.186:6789","nonce":0}]},"addr":"172.21.15.186:6789/0","public_addr":"172.21.15.186:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3301","nonce":0},{"type":"v1","addr":"172.21.15.63:6790","nonce":0}]},"addr":"172.21.15.63:6790/0","public_addr":"172.21.15.63:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T04:24:30.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.911+0000 7fcf8df79640 1 -- 172.21.15.63:0/1486353316 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fcf68035ec0 msgr2=0x7fcf68038380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:30.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.911+0000 7fcf8df79640 1 --2- 172.21.15.63:0/1486353316 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fcf68035ec0 0x7fcf68038380 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fcf7000a990 tx=0x7fcf70008040 comp rx=0 tx=0).stop 2024-04-26T04:24:30.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.911+0000 7fcf8df79640 10 monclient: shutdown 2024-04-26T04:24:30.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.911+0000 7fcf8df79640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:24:30.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.911+0000 7fcf8df79640 1 -- 172.21.15.63:0/1486353316 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcf880a4c00 msgr2=0x7fcf8813bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:30.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.911+0000 7fcf8df79640 1 --2- 172.21.15.63:0/1486353316 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcf880a4c00 0x7fcf8813bbe0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fcf800ad000 tx=0x7fcf80019260 comp rx=0 tx=0).stop 2024-04-26T04:24:30.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.911+0000 7fcf8df79640 1 -- 172.21.15.63:0/1486353316 shutdown_connections 2024-04-26T04:24:30.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.911+0000 7fcf8df79640 1 --2- 172.21.15.63:0/1486353316 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fcf68035ec0 0x7fcf68038380 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:30.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.911+0000 7fcf8df79640 1 --2- 172.21.15.63:0/1486353316 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcf880a4c00 0x7fcf8813bbe0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:30.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.911+0000 7fcf8df79640 1 --2- 172.21.15.63:0/1486353316 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcf880a47d0 0x7fcf88135ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:30.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.911+0000 7fcf8df79640 1 --2- 172.21.15.63:0/1486353316 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf880a2cd0 0x7fcf881cef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:30.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.911+0000 7fcf8df79640 1 -- 172.21.15.63:0/1486353316 >> 172.21.15.63:0/1486353316 conn(0x7fcf880a5f40 msgr2=0x7fcf880702e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:24:30.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.911+0000 7fcf8df79640 1 -- 172.21.15.63:0/1486353316 shutdown_connections 2024-04-26T04:24:30.916 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:30.911+0000 7fcf8df79640 1 -- 172.21.15.63:0/1486353316 wait complete. 2024-04-26T04:24:33.932 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T04:24:34.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.063+0000 7f842263a640 10 monclient: get_monmap_and_config 2024-04-26T04:24:34.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.063+0000 7f842263a640 10 monclient: build_initial_monmap 2024-04-26T04:24:34.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.063+0000 7f842263a640 10 monclient: monmap: 2024-04-26T04:24:34.067 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:24:34.067 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:34.067 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:24:34.066951+0000 2024-04-26T04:24:34.067 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:24:34.066951+0000 2024-04-26T04:24:34.067 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:24:34.067 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:34.067 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:24:34.067 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:24:34.067 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:24:34.067 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:24:34.068 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:24:34.068 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:34.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.063+0000 7f842263a640 1 Processor -- start 2024-04-26T04:24:34.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.063+0000 7f842263a640 1 -- start start 2024-04-26T04:24:34.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.063+0000 7f842263a640 10 monclient: init 2024-04-26T04:24:34.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.063+0000 7f842263a640 10 monclient: _reopen_session rank -1 2024-04-26T04:24:34.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.063+0000 7f842263a640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-04-26T04:24:34.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.063+0000 7f842263a640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f841c0a0bb0 0x7f841c09eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:34.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.063+0000 7f842263a640 10 monclient(hunting): picked mon.noname-c con 0x7f841c0a0bb0 addr v2:172.21.15.186:3300/0 2024-04-26T04:24:34.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.063+0000 7f842263a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f841c09d7b0 addr v1:172.21.15.186:6789/0 2024-04-26T04:24:34.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.063+0000 7f842263a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f841c09e2d0 addr v1:172.21.15.63:6789/0 2024-04-26T04:24:34.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.063+0000 7f842263a640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f841c0a0f90 con 0x7f841c09e2d0 2024-04-26T04:24:34.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.063+0000 7f842263a640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f841c099240 con 0x7f841c09d7b0 2024-04-26T04:24:34.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.063+0000 7f842263a640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:34.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.063+0000 7f842263a640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f841c09b820 con 0x7f841c0a0bb0 2024-04-26T04:24:34.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.063+0000 7f842263a640 10 monclient(hunting): _renew_subs 2024-04-26T04:24:34.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.063+0000 7f842263a640 10 monclient(hunting): authenticate will time out at 1305.775113s 2024-04-26T04:24:34.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.063+0000 7f8420bd7640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f841c09e2d0 0x7f841c13b340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:55390/0 (socket says 172.21.15.63:55390) 2024-04-26T04:24:34.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.063+0000 7f8420bd7640 1 -- 172.21.15.63:0/3969231651 learned_addr learned my addr 172.21.15.63:0/3969231651 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-04-26T04:24:34.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.063+0000 7f841bfff640 1 --2- 172.21.15.63:0/3969231651 >> v2:172.21.15.186:3300/0 conn(0x7f841c0a0bb0 0x7f841c09eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:34.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.063+0000 7f841bfff640 10 monclient(hunting): get_auth_request con 0x7f841c0a0bb0 auth_method 0 2024-04-26T04:24:34.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.063+0000 7f841bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:34.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.063+0000 7f841bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:34.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.063+0000 7f841bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:34.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.063+0000 7f841bfff640 1 -- 172.21.15.63:0/3969231651 >> v2:172.21.15.186:3300/0 conn(0x7f841c0a0bb0 msgr2=0x7f841c09eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T04:24:34.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.063+0000 7f841bfff640 1 -- 172.21.15.63:0/3969231651 >> v2:172.21.15.186:3300/0 conn(0x7f841c0a0bb0 msgr2=0x7f841c09eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:24:34.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.063+0000 7f841bfff640 1 --2- 172.21.15.63:0/3969231651 >> v2:172.21.15.186:3300/0 conn(0x7f841c0a0bb0 0x7f841c09eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:24:34.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.063+0000 7f841bfff640 1 --2- 172.21.15.63:0/3969231651 >> v2:172.21.15.186:3300/0 conn(0x7f841c0a0bb0 0x7f841c09eb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:24:34.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f8420bd7640 1 -- 172.21.15.63:0/3969231651 >> v1:172.21.15.63:6789/0 conn(0x7f841c09e2d0 legacy=0x7f841c13b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:24:34.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f841b7fe640 1 -- 172.21.15.63:0/3969231651 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1218979529 0 0) 0x7f84140017e0 con 0x7f841c09e2d0 2024-04-26T04:24:34.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f841b7fe640 10 monclient(hunting): _negotiate 2024-04-26T04:24:34.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f841b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:34.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f841b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:34.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f841b7fe640 10 monclient(hunting): my global_id is 5310 2024-04-26T04:24:34.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f841b7fe640 1 -- 172.21.15.63:0/3969231651 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8408003530 con 0x7f841c09e2d0 2024-04-26T04:24:34.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f8413fff640 1 -- 172.21.15.63:0/3969231651 >> v1:172.21.15.186:6789/0 conn(0x7f841c09d7b0 legacy=0x7f841c09dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:24:34.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f841b7fe640 1 -- 172.21.15.63:0/3969231651 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2912807684 0 0) 0x7f8408003530 con 0x7f841c09e2d0 2024-04-26T04:24:34.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f841b7fe640 1 -- 172.21.15.63:0/3969231651 >> v2:172.21.15.186:3300/0 conn(0x7f841c0a0bb0 msgr2=0x7f841c09eb60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T04:24:34.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f841b7fe640 1 --2- 172.21.15.63:0/3969231651 >> v2:172.21.15.186:3300/0 conn(0x7f841c0a0bb0 0x7f841c09eb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:34.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f841b7fe640 1 -- 172.21.15.63:0/3969231651 >> v1:172.21.15.186:6789/0 conn(0x7f841c09d7b0 legacy=0x7f841c09dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:34.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f841b7fe640 10 monclient: _finish_hunting 0 2024-04-26T04:24:34.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f841b7fe640 1 monclient: found mon.noname-a-legacy 2024-04-26T04:24:34.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f841b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:24:34.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f841b7fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.63:6789/0 2024-04-26T04:24:34.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f841b7fe640 1 -- 172.21.15.63:0/3969231651 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f841c0994e0 con 0x7f841c09e2d0 2024-04-26T04:24:34.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f841b7fe640 10 monclient: _finish_auth 0 2024-04-26T04:24:34.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f841b7fe640 10 monclient: _check_auth_tickets 2024-04-26T04:24:34.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f841b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:24:34.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f841b7fe640 1 -- 172.21.15.63:0/3969231651 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2612289768 0 0) 0x7f8414001e80 con 0x7f841c09e2d0 2024-04-26T04:24:34.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f841b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:34.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f841b7fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-26T04:24:34.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f841b7fe640 10 monclient: dump: 2024-04-26T04:24:34.076 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:34.076 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:34.076 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:34.076 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:34.076 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:34.076 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:34.076 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:34.076 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:34.076 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:34.076 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:34.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f841b7fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] but i'm connected to v1:172.21.15.63:6789/0, reconnecting 2024-04-26T04:24:34.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f841b7fe640 10 monclient: _reopen_session rank -1 2024-04-26T04:24:34.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f841b7fe640 1 -- 172.21.15.63:0/3969231651 >> v1:172.21.15.63:6789/0 conn(0x7f841c09e2d0 legacy=0x7f841c13b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:34.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f841b7fe640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T04:24:34.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f841b7fe640 1 --2- 172.21.15.63:0/3969231651 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8408003af0 0x7f8408003f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:34.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f841b7fe640 10 monclient(hunting): picked mon.b con 0x7f8408003af0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:24:34.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f841b7fe640 1 --2- 172.21.15.63:0/3969231651 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f8408004810 0x7f8408004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:34.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f841b7fe640 10 monclient(hunting): picked mon.c con 0x7f8408004810 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:24:34.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f841b7fe640 1 --2- 172.21.15.63:0/3969231651 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f84080099c0 0x7f840800bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:34.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f841b7fe640 10 monclient(hunting): picked mon.a con 0x7f84080099c0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:24:34.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f841b7fe640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:34.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f841b7fe640 1 -- 172.21.15.63:0/3969231651 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f841c09b820 con 0x7f84080099c0 2024-04-26T04:24:34.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f841b7fe640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:34.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f841b7fe640 1 -- 172.21.15.63:0/3969231651 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f840800c3a0 con 0x7f8408003af0 2024-04-26T04:24:34.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f841b7fe640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:34.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f841b7fe640 1 -- 172.21.15.63:0/3969231651 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f840800c5a0 con 0x7f8408004810 2024-04-26T04:24:34.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f841b7fe640 10 monclient(hunting): _renew_subs 2024-04-26T04:24:34.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f841b7fe640 10 monclient(hunting): _finish_auth 0 2024-04-26T04:24:34.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f8413fff640 1 --2- 172.21.15.63:0/3969231651 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f8408004810 0x7f8408004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:34.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f8420bd7640 1 --2- 172.21.15.63:0/3969231651 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f84080099c0 0x7f840800bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:34.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f8420bd7640 10 monclient(hunting): get_auth_request con 0x7f84080099c0 auth_method 0 2024-04-26T04:24:34.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f8420bd7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:34.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f8420bd7640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:34.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f8420bd7640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:24:34.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f8413fff640 10 monclient(hunting): get_auth_request con 0x7f8408004810 auth_method 0 2024-04-26T04:24:34.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f8413fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:34.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f8413fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:34.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f8413fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:24:34.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f841bfff640 1 --2- 172.21.15.63:0/3969231651 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8408003af0 0x7f8408003f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:34.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f8420bd7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:34.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f8420bd7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:34.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f8420bd7640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T04:24:34.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f8413fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:34.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f8413fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:34.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f8413fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T04:24:34.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f841bfff640 10 monclient(hunting): get_auth_request con 0x7f8408003af0 auth_method 0 2024-04-26T04:24:34.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f841bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:34.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f841bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:34.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f841bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:24:34.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f8420bd7640 10 monclient(hunting): handle_auth_done global_id 5310 payload 293 2024-04-26T04:24:34.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f8420bd7640 1 -- 172.21.15.63:0/3969231651 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f8408004810 msgr2=0x7f8408004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:24:34.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f8420bd7640 1 --2- 172.21.15.63:0/3969231651 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f8408004810 0x7f8408004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:34.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f8420bd7640 1 -- 172.21.15.63:0/3969231651 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8408003af0 msgr2=0x7f8408003f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:24:34.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f8420bd7640 1 --2- 172.21.15.63:0/3969231651 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8408003af0 0x7f8408003f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:34.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f8420bd7640 10 monclient: _finish_hunting 0 2024-04-26T04:24:34.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f8420bd7640 1 monclient: found mon.a 2024-04-26T04:24:34.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f8420bd7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:24:34.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f8420bd7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:24:34.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f8420bd7640 1 -- 172.21.15.63:0/3969231651 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8408003530 con 0x7f84080099c0 2024-04-26T04:24:34.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f8420bd7640 10 monclient: _finish_auth 0 2024-04-26T04:24:34.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f8420bd7640 10 monclient: _check_auth_tickets 2024-04-26T04:24:34.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f8420bd7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:24:34.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f842263a640 5 monclient: authenticate success, global_id 5310 2024-04-26T04:24:34.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f842263a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T04:24:34.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f8413fff640 1 --2- 172.21.15.63:0/3969231651 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f8408004810 0x7f8408004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T04:24:34.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.067+0000 7f841bfff640 1 --2- 172.21.15.63:0/3969231651 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8408003af0 0x7f8408003f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T04:24:34.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.071+0000 7f8420bd7640 1 -- 172.21.15.63:0/3969231651 reap_dead start 2024-04-26T04:24:34.086 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.071+0000 7f8420bd7640 1 -- 172.21.15.63:0/3969231651 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f84080099c0 msgr2=0x7f840800bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:24:34.086 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.071+0000 7f8420bd7640 1 --2- 172.21.15.63:0/3969231651 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f84080099c0 0x7f840800bdb0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f84140020a0 tx=0x7f841401fa70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26019b89cf0efc87 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:34.086 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.071+0000 7f841b7fe640 1 -- 172.21.15.63:0/3969231651 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8404001680 con 0x7f84080099c0 2024-04-26T04:24:34.086 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.071+0000 7f841b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:34.086 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.071+0000 7f841b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:24:34.086 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.071+0000 7f841b7fe640 10 monclient: dump: 2024-04-26T04:24:34.087 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:34.087 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:34.087 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:34.087 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:34.087 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:34.087 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:34.087 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:34.087 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:34.087 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:34.087 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:34.087 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.071+0000 7f841b7fe640 1 -- 172.21.15.63:0/3969231651 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84140ac070 con 0x7f84080099c0 2024-04-26T04:24:34.087 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.071+0000 7f841b7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T04:24:34.087 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.071+0000 7f842263a640 10 monclient: get_monmap_and_config success 2024-04-26T04:24:34.087 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.071+0000 7f842263a640 4 set_mon_vals no callback set 2024-04-26T04:24:34.087 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.071+0000 7f841b7fe640 1 -- 172.21.15.63:0/3969231651 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f841400a930 con 0x7f84080099c0 2024-04-26T04:24:34.088 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.071+0000 7f841b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:34.088 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.071+0000 7f841b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:24:34.088 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.071+0000 7f841b7fe640 10 monclient: dump: 2024-04-26T04:24:34.088 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:34.088 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:34.088 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:34.088 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:34.088 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:34.088 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:34.088 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:34.088 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:34.088 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:34.088 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:34.088 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.071+0000 7f842263a640 10 monclient: shutdown 2024-04-26T04:24:34.088 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.071+0000 7f842263a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:24:34.089 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.071+0000 7f842263a640 1 -- 172.21.15.63:0/3969231651 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f84080099c0 msgr2=0x7f840800bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:34.089 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.071+0000 7f842263a640 1 --2- 172.21.15.63:0/3969231651 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f84080099c0 0x7f840800bdb0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f84140020a0 tx=0x7f841401fa70 comp rx=0 tx=0).stop 2024-04-26T04:24:34.089 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.071+0000 7f842263a640 1 -- 172.21.15.63:0/3969231651 shutdown_connections 2024-04-26T04:24:34.089 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.075+0000 7f842263a640 1 --2- 172.21.15.63:0/3969231651 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f84080099c0 0x7f840800bdb0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:34.089 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.075+0000 7f842263a640 1 -- 172.21.15.63:0/3969231651 >> 172.21.15.63:0/3969231651 conn(0x7f841c0a1e70 msgr2=0x7f841c0a2280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:24:34.089 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.075+0000 7f842263a640 1 -- 172.21.15.63:0/3969231651 shutdown_connections 2024-04-26T04:24:34.089 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.075+0000 7f842263a640 1 -- 172.21.15.63:0/3969231651 wait complete. 2024-04-26T04:24:34.089 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.075+0000 7f842263a640 10 monclient: build_initial_monmap 2024-04-26T04:24:34.089 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.075+0000 7f842263a640 10 monclient: monmap: 2024-04-26T04:24:34.089 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:24:34.089 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:24:34.089 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:24:34.089 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:24:34.089 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:24:34.089 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:34.090 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:24:34.090 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:24:34.090 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:24:34.090 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:34.090 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.075+0000 7f842263a640 1 Processor -- start 2024-04-26T04:24:34.090 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.075+0000 7f842263a640 1 -- start start 2024-04-26T04:24:34.090 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.075+0000 7f842263a640 10 monclient: init 2024-04-26T04:24:34.090 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.075+0000 7f842263a640 10 monclient: _reopen_session rank -1 2024-04-26T04:24:34.090 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.075+0000 7f842263a640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T04:24:34.090 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.075+0000 7f842263a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f84080099c0 0x7f841c1cee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:34.090 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.075+0000 7f842263a640 10 monclient(hunting): picked mon.noname-a con 0x7f84080099c0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:24:34.090 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.075+0000 7f842263a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f841c097ed0 0x7f841c098310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:34.090 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.075+0000 7f842263a640 10 monclient(hunting): picked mon.noname-b con 0x7f841c097ed0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:24:34.090 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.075+0000 7f842263a640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f841c090d30 0x7f841c0911b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:34.091 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.075+0000 7f842263a640 10 monclient(hunting): picked mon.noname-c con 0x7f841c090d30 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:24:34.091 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.075+0000 7f842263a640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:34.091 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.075+0000 7f842263a640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f841c09eef0 con 0x7f84080099c0 2024-04-26T04:24:34.091 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.075+0000 7f842263a640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:34.091 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.075+0000 7f842263a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f841c09ed70 con 0x7f841c097ed0 2024-04-26T04:24:34.091 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.075+0000 7f842263a640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:34.091 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.075+0000 7f842263a640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f841c09dbc0 con 0x7f841c090d30 2024-04-26T04:24:34.091 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.075+0000 7f842263a640 10 monclient(hunting): _renew_subs 2024-04-26T04:24:34.091 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.075+0000 7f842263a640 10 monclient(hunting): authenticate will time out at 1305.785667s 2024-04-26T04:24:34.091 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.075+0000 7f841bfff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f84080099c0 0x7f841c1cee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:34.091 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.075+0000 7f8420bd7640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f841c090d30 0x7f841c0911b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:34.091 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.075+0000 7f841bfff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f84080099c0 0x7f841c1cee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:33212/0 (socket says 172.21.15.63:33212) 2024-04-26T04:24:34.091 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.075+0000 7f8420bd7640 10 monclient(hunting): get_auth_request con 0x7f841c090d30 auth_method 0 2024-04-26T04:24:34.091 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.075+0000 7f841bfff640 1 -- 172.21.15.63:0/2692434931 learned_addr learned my addr 172.21.15.63:0/2692434931 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:24:34.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.075+0000 7f8420bd7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:34.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.075+0000 7f8420bd7640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:34.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.075+0000 7f8420bd7640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:34.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.075+0000 7f841bfff640 10 monclient(hunting): get_auth_request con 0x7f84080099c0 auth_method 0 2024-04-26T04:24:34.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.075+0000 7f841bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:34.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.075+0000 7f841bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:34.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.075+0000 7f841bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:34.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.075+0000 7f8420bd7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:34.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.075+0000 7f8420bd7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:34.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.075+0000 7f8420bd7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:24:34.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.075+0000 7f841bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:34.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.075+0000 7f841bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:34.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.075+0000 7f841bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:24:34.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.075+0000 7f8413fff640 1 --2- 172.21.15.63:0/2692434931 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f841c097ed0 0x7f841c098310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:34.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.075+0000 7f841bfff640 10 monclient(hunting): handle_auth_done global_id 5313 payload 824 2024-04-26T04:24:34.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.075+0000 7f841bfff640 1 -- 172.21.15.63:0/2692434931 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f841c090d30 msgr2=0x7f841c0911b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:34.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.075+0000 7f841bfff640 1 --2- 172.21.15.63:0/2692434931 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f841c090d30 0x7f841c0911b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:34.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.075+0000 7f841bfff640 1 -- 172.21.15.63:0/2692434931 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f841c097ed0 msgr2=0x7f841c098310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:34.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.075+0000 7f841bfff640 1 --2- 172.21.15.63:0/2692434931 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f841c097ed0 0x7f841c098310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:34.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.075+0000 7f841bfff640 10 monclient: _finish_hunting 0 2024-04-26T04:24:34.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.075+0000 7f841bfff640 1 monclient: found mon.noname-a 2024-04-26T04:24:34.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.075+0000 7f841bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:24:34.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.075+0000 7f841bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:24:34.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.075+0000 7f841bfff640 1 -- 172.21.15.63:0/2692434931 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f841c092b30 con 0x7f84080099c0 2024-04-26T04:24:34.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.075+0000 7f8413fff640 10 monclient: get_auth_request con 0x7f841c097ed0 auth_method 0 2024-04-26T04:24:34.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.075+0000 7f8413fff640 1 --2- 172.21.15.63:0/2692434931 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f841c097ed0 0x7f841c098310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T04:24:34.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.075+0000 7f8420bd7640 1 --2- 172.21.15.63:0/2692434931 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f841c090d30 0x7f841c0911b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T04:24:34.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.079+0000 7f841bfff640 1 -- 172.21.15.63:0/2692434931 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f84080099c0 msgr2=0x7f841c1cee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:24:34.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.079+0000 7f841bfff640 1 --2- 172.21.15.63:0/2692434931 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f84080099c0 0x7f841c1cee10 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f840c010860 tx=0x7f840c010d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:34.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.079+0000 7f84197fa640 1 -- 172.21.15.63:0/2692434931 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f840c01a020 con 0x7f84080099c0 2024-04-26T04:24:34.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.079+0000 7f84197fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:34.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.079+0000 7f84197fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:24:34.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.079+0000 7f84197fa640 10 monclient: dump: 2024-04-26T04:24:34.094 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:34.094 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:34.094 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:34.094 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:34.094 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:34.094 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:34.094 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:34.094 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:34.094 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:34.095 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:34.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.079+0000 7f84197fa640 10 monclient: _finish_auth 0 2024-04-26T04:24:34.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.079+0000 7f84197fa640 10 monclient: _check_auth_tickets 2024-04-26T04:24:34.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.079+0000 7f84197fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:24:34.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.079+0000 7f84197fa640 1 -- 172.21.15.63:0/2692434931 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f840c00cd50 con 0x7f84080099c0 2024-04-26T04:24:34.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.079+0000 7f842263a640 5 monclient: authenticate success, global_id 5313 2024-04-26T04:24:34.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.079+0000 7f842263a640 10 monclient: _renew_subs 2024-04-26T04:24:34.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.079+0000 7f842263a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:24:34.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.079+0000 7f842263a640 1 -- 172.21.15.63:0/2692434931 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f841c0a40b0 con 0x7f84080099c0 2024-04-26T04:24:34.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.079+0000 7f84197fa640 10 monclient: handle_config config(0 keys) 2024-04-26T04:24:34.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.079+0000 7f84197fa640 1 -- 172.21.15.63:0/2692434931 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f840c00f040 con 0x7f84080099c0 2024-04-26T04:24:34.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.079+0000 7f84197fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:34.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.079+0000 7f84197fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:24:34.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.079+0000 7f84197fa640 10 monclient: dump: 2024-04-26T04:24:34.095 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:34.096 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:34.096 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:34.096 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:34.096 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:34.096 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:34.096 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:34.096 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:34.096 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:34.096 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:34.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.079+0000 7f841affd640 4 set_mon_vals no callback set 2024-04-26T04:24:34.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.079+0000 7f842263a640 10 monclient: _renew_subs 2024-04-26T04:24:34.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.079+0000 7f842263a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:24:34.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.079+0000 7f842263a640 1 -- 172.21.15.63:0/2692434931 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f841c0940e0 con 0x7f84080099c0 2024-04-26T04:24:34.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.079+0000 7f842263a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:24:34.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.079+0000 7f842263a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:24:34.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.079+0000 7f842263a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:24:34.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.079+0000 7f84197fa640 1 -- 172.21.15.63:0/2692434931 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f840c01d020 con 0x7f84080099c0 2024-04-26T04:24:34.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.079+0000 7f842263a640 1 -- 172.21.15.63:0/2692434931 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f841c062e10 con 0x7f84080099c0 2024-04-26T04:24:34.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.079+0000 7f84197fa640 1 --2- 172.21.15.63:0/2692434931 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f83fc035ec0 0x7f83fc038380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:34.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.079+0000 7f84197fa640 1 -- 172.21.15.63:0/2692434931 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(219..219 src has 1..219) ==== 4738+0+0 (secure 0 0 0) 0x7f840c049480 con 0x7f84080099c0 2024-04-26T04:24:34.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.079+0000 7f8413fff640 1 --2- 172.21.15.63:0/2692434931 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f83fc035ec0 0x7f83fc038380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:34.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.079+0000 7f8413fff640 10 monclient: get_auth_request con 0x7f83fc035ec0 auth_method 0 2024-04-26T04:24:34.098 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.083+0000 7f8413fff640 1 --2- 172.21.15.63:0/2692434931 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f83fc035ec0 0x7f83fc038380 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f841c08c230 tx=0x7f8404009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:34.098 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.083+0000 7f84197fa640 1 -- 172.21.15.63:0/2692434931 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f840c024020 con 0x7f84080099c0 2024-04-26T04:24:34.098 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.083+0000 7f84197fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:24:34.098 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.083+0000 7f84197fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:24:34.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.303+0000 7f842263a640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T04:24:34.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.303+0000 7f842263a640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T04:24:34.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.303+0000 7f842263a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:24:34.309 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.303+0000 7f842263a640 1 -- 172.21.15.63:0/2692434931 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f841c068430 con 0x7f84080099c0 2024-04-26T04:24:34.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.307+0000 7f84197fa640 1 -- 172.21.15.63:0/2692434931 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f840c010e50 con 0x7f84080099c0 2024-04-26T04:24:34.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.307+0000 7f84197fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T04:24:34.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.307+0000 7f84197fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:24:34.310 INFO:teuthology.orchestra.run.smithi063.stdout:{"election_epoch":24,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":98,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cd1c6a0c-4e4b-4014-89c0-2e535479ef39","modified":"2024-04-26T04:18:03.702339Z","created":"2024-04-26T04:18:03.702339Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3300","nonce":0},{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:3300","nonce":0},{"type":"v1","addr":"172.21.15.186:6789","nonce":0}]},"addr":"172.21.15.186:6789/0","public_addr":"172.21.15.186:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3301","nonce":0},{"type":"v1","addr":"172.21.15.63:6790","nonce":0}]},"addr":"172.21.15.63:6790/0","public_addr":"172.21.15.63:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T04:24:34.312 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.307+0000 7f842263a640 1 -- 172.21.15.63:0/2692434931 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f83fc035ec0 msgr2=0x7f83fc038380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:34.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.307+0000 7f842263a640 1 --2- 172.21.15.63:0/2692434931 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f83fc035ec0 0x7f83fc038380 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f841c08c230 tx=0x7f8404009040 comp rx=0 tx=0).stop 2024-04-26T04:24:34.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.307+0000 7f842263a640 10 monclient: shutdown 2024-04-26T04:24:34.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.307+0000 7f842263a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:24:34.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.307+0000 7f842263a640 1 -- 172.21.15.63:0/2692434931 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f84080099c0 msgr2=0x7f841c1cee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:34.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.307+0000 7f842263a640 1 --2- 172.21.15.63:0/2692434931 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f84080099c0 0x7f841c1cee10 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f840c010860 tx=0x7f840c010d20 comp rx=0 tx=0).stop 2024-04-26T04:24:34.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.307+0000 7f842263a640 1 -- 172.21.15.63:0/2692434931 shutdown_connections 2024-04-26T04:24:34.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.307+0000 7f842263a640 1 --2- 172.21.15.63:0/2692434931 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f83fc035ec0 0x7f83fc038380 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:34.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.307+0000 7f842263a640 1 --2- 172.21.15.63:0/2692434931 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f841c090d30 0x7f841c0911b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:34.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.307+0000 7f842263a640 1 --2- 172.21.15.63:0/2692434931 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f841c097ed0 0x7f841c098310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:34.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.307+0000 7f842263a640 1 --2- 172.21.15.63:0/2692434931 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f84080099c0 0x7f841c1cee10 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:34.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.307+0000 7f842263a640 1 -- 172.21.15.63:0/2692434931 >> 172.21.15.63:0/2692434931 conn(0x7f841c0a1e70 msgr2=0x7f841c099fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:24:34.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.307+0000 7f842263a640 1 -- 172.21.15.63:0/2692434931 shutdown_connections 2024-04-26T04:24:34.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:34.307+0000 7f842263a640 1 -- 172.21.15.63:0/2692434931 wait complete. 2024-04-26T04:24:36.839 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:24:36.839 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[30537]: starting. 2024-04-26T04:24:36.839 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[30537]: listing objects. 2024-04-26T04:24:36.839 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[30537]: listed object 0... 2024-04-26T04:24:36.840 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[30537]: listed object 25... 2024-04-26T04:24:36.840 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[30537]: saw 50 objects 2024-04-26T04:24:36.840 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[30537]: shutting down. 2024-04-26T04:24:37.330 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T04:24:37.476 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.459+0000 7f2692c1a640 10 monclient: get_monmap_and_config 2024-04-26T04:24:37.476 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.459+0000 7f2692c1a640 10 monclient: build_initial_monmap 2024-04-26T04:24:37.476 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.459+0000 7f2692c1a640 10 monclient: monmap: 2024-04-26T04:24:37.476 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:24:37.476 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:37.476 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:24:37.463573+0000 2024-04-26T04:24:37.476 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:24:37.463573+0000 2024-04-26T04:24:37.476 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:24:37.476 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:37.477 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:24:37.477 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:24:37.477 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:24:37.477 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:24:37.477 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:24:37.477 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:37.477 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.459+0000 7f2692c1a640 1 Processor -- start 2024-04-26T04:24:37.477 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.459+0000 7f2692c1a640 1 -- start start 2024-04-26T04:24:37.477 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.459+0000 7f2692c1a640 10 monclient: init 2024-04-26T04:24:37.477 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.459+0000 7f2692c1a640 10 monclient: _reopen_session rank -1 2024-04-26T04:24:37.477 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.459+0000 7f2692c1a640 10 monclient: _add_conns ranks=[3,2,0,1,4] 2024-04-26T04:24:37.477 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.459+0000 7f2692c1a640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f268c09fe60 0x7f268c1372f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:37.477 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.459+0000 7f2692c1a640 10 monclient(hunting): picked mon.noname-a con 0x7f268c09fe60 addr v2:172.21.15.63:3300/0 2024-04-26T04:24:37.477 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.459+0000 7f2692c1a640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f268c0a0240 0x7f268c0a2a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:37.478 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.459+0000 7f2692c1a640 10 monclient(hunting): picked mon.noname-b con 0x7f268c0a0240 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:24:37.478 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.459+0000 7f2692c1a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f268c0a2f70 addr v1:172.21.15.63:6789/0 2024-04-26T04:24:37.478 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.459+0000 7f2692c1a640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f268c137830 con 0x7f268c0a2f70 2024-04-26T04:24:37.478 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.459+0000 7f2692c1a640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:37.478 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.459+0000 7f2692c1a640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7f268c0a3730 con 0x7f268c09fe60 2024-04-26T04:24:37.478 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.459+0000 7f2692c1a640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:37.478 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.459+0000 7f2692c1a640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f268c0a0c60 con 0x7f268c0a0240 2024-04-26T04:24:37.478 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.459+0000 7f2692c1a640 10 monclient(hunting): _renew_subs 2024-04-26T04:24:37.478 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.459+0000 7f2692c1a640 10 monclient(hunting): authenticate will time out at 1309.171227s 2024-04-26T04:24:37.478 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.459+0000 7f268bfff640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f268c0a0240 0x7f268c0a2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:37.478 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.459+0000 7f26909b6640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f268c09fe60 0x7f268c1372f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:37.478 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.459+0000 7f26911b7640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f268c0a2f70 0x7f268c0a3eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:55392/0 (socket says 172.21.15.63:55392) 2024-04-26T04:24:37.478 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.459+0000 7f26911b7640 1 -- 172.21.15.63:0/1555608359 learned_addr learned my addr 172.21.15.63:0/1555608359 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-04-26T04:24:37.478 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.459+0000 7f268bfff640 10 monclient(hunting): get_auth_request con 0x7f268c0a0240 auth_method 0 2024-04-26T04:24:37.479 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.459+0000 7f268bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:37.479 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.459+0000 7f268bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:37.479 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.459+0000 7f268bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:37.479 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.459+0000 7f26909b6640 10 monclient(hunting): get_auth_request con 0x7f268c09fe60 auth_method 0 2024-04-26T04:24:37.479 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.459+0000 7f26909b6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:37.479 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.459+0000 7f26909b6640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:37.479 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.459+0000 7f26909b6640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:37.479 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.459+0000 7f268bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:37.479 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.459+0000 7f268bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:37.479 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.459+0000 7f268bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:24:37.479 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.459+0000 7f26909b6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:37.479 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.459+0000 7f26909b6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:37.479 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.459+0000 7f26909b6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:24:37.479 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.459+0000 7f268bfff640 10 monclient(hunting): handle_auth_done global_id 24218 payload 274 2024-04-26T04:24:37.479 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.459+0000 7f268bfff640 1 -- 172.21.15.63:0/1555608359 >> v2:172.21.15.63:3300/0 conn(0x7f268c09fe60 msgr2=0x7f268c1372f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:24:37.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.459+0000 7f268bfff640 1 --2- 172.21.15.63:0/1555608359 >> v2:172.21.15.63:3300/0 conn(0x7f268c09fe60 0x7f268c1372f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:37.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.459+0000 7f268bfff640 1 -- 172.21.15.63:0/1555608359 >> v1:172.21.15.63:6789/0 conn(0x7f268c0a2f70 legacy=0x7f268c0a3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:24:37.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.459+0000 7f268bfff640 10 monclient: _finish_hunting 0 2024-04-26T04:24:37.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.459+0000 7f268bfff640 1 monclient: found mon.noname-b 2024-04-26T04:24:37.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.459+0000 7f268bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:24:37.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.459+0000 7f268bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.63:3301/0 2024-04-26T04:24:37.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.459+0000 7f268bfff640 1 -- 172.21.15.63:0/1555608359 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f268c0a0f40 con 0x7f268c0a0240 2024-04-26T04:24:37.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.463+0000 7f268bfff640 1 -- 172.21.15.63:0/1555608359 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f268c0a0240 msgr2=0x7f268c0a2a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:24:37.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.463+0000 7f268bfff640 1 --2- 172.21.15.63:0/1555608359 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f268c0a0240 0x7f268c0a2a30 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f26800097c0 tx=0x7f268001c2f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c7a3790f5c37089a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:37.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f268b7fe640 1 -- 172.21.15.63:0/1555608359 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26800b0070 con 0x7f268c0a0240 2024-04-26T04:24:37.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f268b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:37.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f268b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T04:24:37.480 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f268b7fe640 10 monclient: dump: 2024-04-26T04:24:37.480 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:37.480 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:37.481 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:37.481 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:37.481 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:37.481 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:37.481 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:37.481 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:37.481 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:37.481 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:37.481 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f268b7fe640 10 monclient: _finish_auth 0 2024-04-26T04:24:37.481 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f268b7fe640 10 monclient: _check_auth_tickets 2024-04-26T04:24:37.481 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f268b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:24:37.481 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f268b7fe640 1 -- 172.21.15.63:0/1555608359 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26800ab600 con 0x7f268c0a0240 2024-04-26T04:24:37.481 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f268b7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T04:24:37.481 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f268b7fe640 1 -- 172.21.15.63:0/1555608359 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26800ab9d0 con 0x7f268c0a0240 2024-04-26T04:24:37.482 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f2692c1a640 5 monclient: authenticate success, global_id 24218 2024-04-26T04:24:37.482 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f2692c1a640 10 monclient: get_monmap_and_config success 2024-04-26T04:24:37.482 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f2692c1a640 4 set_mon_vals no callback set 2024-04-26T04:24:37.482 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f268b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:37.482 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f268b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:24:37.482 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f268b7fe640 10 monclient: dump: 2024-04-26T04:24:37.482 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:37.482 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:37.482 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:37.482 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:37.482 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:37.482 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:37.482 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:37.482 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:37.483 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:37.483 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:37.483 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f2692c1a640 10 monclient: shutdown 2024-04-26T04:24:37.483 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f2692c1a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:24:37.483 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f2692c1a640 1 -- 172.21.15.63:0/1555608359 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f268c0a0240 msgr2=0x7f268c0a2a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:37.483 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f2692c1a640 1 --2- 172.21.15.63:0/1555608359 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f268c0a0240 0x7f268c0a2a30 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f26800097c0 tx=0x7f268001c2f0 comp rx=0 tx=0).stop 2024-04-26T04:24:37.483 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f2692c1a640 1 -- 172.21.15.63:0/1555608359 shutdown_connections 2024-04-26T04:24:37.483 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f2692c1a640 1 --2- 172.21.15.63:0/1555608359 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f268c0a0240 0x7f268c0a2a30 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:37.483 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f2692c1a640 1 --2- 172.21.15.63:0/1555608359 >> v2:172.21.15.63:3300/0 conn(0x7f268c09fe60 0x7f268c1372f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:37.483 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f2692c1a640 1 -- 172.21.15.63:0/1555608359 >> 172.21.15.63:0/1555608359 conn(0x7f268c0a5fb0 msgr2=0x7f268c0a63c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:24:37.483 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f2692c1a640 1 -- 172.21.15.63:0/1555608359 shutdown_connections 2024-04-26T04:24:37.483 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f2692c1a640 1 -- 172.21.15.63:0/1555608359 wait complete. 2024-04-26T04:24:37.483 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f2692c1a640 10 monclient: build_initial_monmap 2024-04-26T04:24:37.483 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f2692c1a640 10 monclient: monmap: 2024-04-26T04:24:37.484 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:24:37.484 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:24:37.484 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:24:37.484 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:24:37.484 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:24:37.484 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:37.484 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:24:37.484 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:24:37.484 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:24:37.484 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:37.484 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f2692c1a640 1 Processor -- start 2024-04-26T04:24:37.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f2692c1a640 1 -- start start 2024-04-26T04:24:37.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f2692c1a640 10 monclient: init 2024-04-26T04:24:37.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f2692c1a640 10 monclient: _reopen_session rank -1 2024-04-26T04:24:37.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f2692c1a640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T04:24:37.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f2692c1a640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f268c09fe60 0x7f268c1cf440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:37.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f2692c1a640 10 monclient(hunting): picked mon.noname-c con 0x7f268c09fe60 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:24:37.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f2692c1a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f268c0a0240 0x7f268c13df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:37.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f2692c1a640 10 monclient(hunting): picked mon.noname-b con 0x7f268c0a0240 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:24:37.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f2692c1a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f268c0a2f70 0x7f268c13e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:37.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f2692c1a640 10 monclient(hunting): picked mon.noname-a con 0x7f268c0a2f70 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:24:37.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f2692c1a640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:37.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f2692c1a640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f268c141d10 con 0x7f268c0a2f70 2024-04-26T04:24:37.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f2692c1a640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:37.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f2692c1a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f268c141b90 con 0x7f268c0a0240 2024-04-26T04:24:37.486 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f2692c1a640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:37.486 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f2692c1a640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f268c141e90 con 0x7f268c09fe60 2024-04-26T04:24:37.486 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f2692c1a640 10 monclient(hunting): _renew_subs 2024-04-26T04:24:37.486 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f2692c1a640 10 monclient(hunting): authenticate will time out at 1309.178488s 2024-04-26T04:24:37.486 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f26909b6640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f268c09fe60 0x7f268c1cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:37.486 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f26911b7640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f268c0a2f70 0x7f268c13e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:37.486 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f26909b6640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f268c09fe60 0x7f268c1cf440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:40088/0 (socket says 172.21.15.63:40088) 2024-04-26T04:24:37.486 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f26909b6640 1 -- 172.21.15.63:0/95919399 learned_addr learned my addr 172.21.15.63:0/95919399 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:24:37.486 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f26909b6640 10 monclient(hunting): get_auth_request con 0x7f268c09fe60 auth_method 0 2024-04-26T04:24:37.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f26909b6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:37.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f26909b6640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:37.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f26909b6640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:37.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f26911b7640 10 monclient(hunting): get_auth_request con 0x7f268c0a2f70 auth_method 0 2024-04-26T04:24:37.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f26911b7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:37.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f26911b7640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:37.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f26911b7640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:37.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f26909b6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:37.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f26909b6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:37.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f26909b6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:24:37.488 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f26911b7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:37.488 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f26911b7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:37.488 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f26911b7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:24:37.488 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f268bfff640 1 --2- 172.21.15.63:0/95919399 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f268c0a0240 0x7f268c13df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:37.488 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f268bfff640 10 monclient(hunting): get_auth_request con 0x7f268c0a0240 auth_method 0 2024-04-26T04:24:37.488 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f268bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:37.488 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f268bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:37.488 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f268bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:37.488 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f26909b6640 10 monclient(hunting): handle_auth_done global_id 24221 payload 824 2024-04-26T04:24:37.488 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f26909b6640 1 -- 172.21.15.63:0/95919399 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f268c0a0240 msgr2=0x7f268c13df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:37.488 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f26909b6640 1 --2- 172.21.15.63:0/95919399 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f268c0a0240 0x7f268c13df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:37.488 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f26909b6640 1 -- 172.21.15.63:0/95919399 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f268c0a2f70 msgr2=0x7f268c13e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:37.488 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f26909b6640 1 --2- 172.21.15.63:0/95919399 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f268c0a2f70 0x7f268c13e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:37.488 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f26909b6640 10 monclient: _finish_hunting 0 2024-04-26T04:24:37.489 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f26909b6640 1 monclient: found mon.noname-c 2024-04-26T04:24:37.489 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f26909b6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:24:37.489 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f26909b6640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.63:3301/0 2024-04-26T04:24:37.489 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f26909b6640 1 -- 172.21.15.63:0/95919399 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f268c13c700 con 0x7f268c09fe60 2024-04-26T04:24:37.489 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.467+0000 7f26911b7640 1 --2- 172.21.15.63:0/95919399 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f268c0a2f70 0x7f268c13e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T04:24:37.489 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.471+0000 7f26909b6640 1 -- 172.21.15.63:0/95919399 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f268c09fe60 msgr2=0x7f268c1cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:24:37.489 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.471+0000 7f26909b6640 1 --2- 172.21.15.63:0/95919399 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f268c09fe60 0x7f268c1cf440 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f267c00e930 tx=0x7f267c00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:37.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.471+0000 7f26897fa640 1 -- 172.21.15.63:0/95919399 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f267c017070 con 0x7f268c09fe60 2024-04-26T04:24:37.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.471+0000 7f26897fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:37.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.471+0000 7f26897fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:24:37.491 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.471+0000 7f26897fa640 10 monclient: dump: 2024-04-26T04:24:37.491 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:37.491 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:37.491 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:37.491 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:37.491 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:37.491 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:37.491 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:37.492 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:37.492 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:37.492 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:37.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.471+0000 7f26897fa640 10 monclient: _finish_auth 0 2024-04-26T04:24:37.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.471+0000 7f26897fa640 10 monclient: _check_auth_tickets 2024-04-26T04:24:37.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.471+0000 7f26897fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:24:37.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.471+0000 7f26897fa640 1 -- 172.21.15.63:0/95919399 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f267c00cc10 con 0x7f268c09fe60 2024-04-26T04:24:37.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.471+0000 7f26897fa640 10 monclient: handle_config config(0 keys) 2024-04-26T04:24:37.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.471+0000 7f26897fa640 1 -- 172.21.15.63:0/95919399 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f267c013470 con 0x7f268c09fe60 2024-04-26T04:24:37.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.471+0000 7f26897fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:37.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.471+0000 7f26897fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:24:37.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.471+0000 7f26897fa640 10 monclient: dump: 2024-04-26T04:24:37.492 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:37.492 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:37.493 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:37.493 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:37.493 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:37.493 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:37.493 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:37.493 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:37.493 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:37.493 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:37.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.471+0000 7f2692c1a640 5 monclient: authenticate success, global_id 24221 2024-04-26T04:24:37.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.471+0000 7f2692c1a640 10 monclient: _renew_subs 2024-04-26T04:24:37.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.471+0000 7f268affd640 4 set_mon_vals no callback set 2024-04-26T04:24:37.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.471+0000 7f2692c1a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:24:37.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.471+0000 7f2692c1a640 1 -- 172.21.15.63:0/95919399 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f268c13afd0 con 0x7f268c09fe60 2024-04-26T04:24:37.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.471+0000 7f2692c1a640 10 monclient: _renew_subs 2024-04-26T04:24:37.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.471+0000 7f2692c1a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:24:37.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.471+0000 7f2692c1a640 1 -- 172.21.15.63:0/95919399 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f268c09f9d0 con 0x7f268c09fe60 2024-04-26T04:24:37.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.475+0000 7f2692c1a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:24:37.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.475+0000 7f2692c1a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:24:37.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.475+0000 7f2692c1a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:24:37.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.475+0000 7f26897fa640 1 -- 172.21.15.63:0/95919399 <== mon.2 v2:172.21.15.63:3301/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f267c01a070 con 0x7f268c09fe60 2024-04-26T04:24:37.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.475+0000 7f2692c1a640 1 -- 172.21.15.63:0/95919399 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f268c062e10 con 0x7f268c09fe60 2024-04-26T04:24:37.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.475+0000 7f26897fa640 1 --2- 172.21.15.63:0/95919399 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f2678035ec0 0x7f2678038380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:37.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.475+0000 7f26897fa640 1 -- 172.21.15.63:0/95919399 <== mon.2 v2:172.21.15.63:3301/0 5 ==== osd_map(219..219 src has 1..219) ==== 4738+0+0 (secure 0 0 0) 0x7f267c048fb0 con 0x7f268c09fe60 2024-04-26T04:24:37.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.475+0000 7f268bfff640 1 --2- 172.21.15.63:0/95919399 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f2678035ec0 0x7f2678038380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:37.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.475+0000 7f268bfff640 10 monclient: get_auth_request con 0x7f2678035ec0 auth_method 0 2024-04-26T04:24:37.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.475+0000 7f268bfff640 1 --2- 172.21.15.63:0/95919399 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f2678035ec0 0x7f2678038380 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f26800ab000 tx=0x7f26800a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:37.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.479+0000 7f26897fa640 1 -- 172.21.15.63:0/95919399 <== mon.2 v2:172.21.15.63:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f267c013e10 con 0x7f268c09fe60 2024-04-26T04:24:37.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.479+0000 7f26897fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:24:37.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.479+0000 7f26897fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:24:37.702 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.699+0000 7f2692c1a640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T04:24:37.702 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.699+0000 7f2692c1a640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T04:24:37.702 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.699+0000 7f2692c1a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:24:37.702 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.699+0000 7f2692c1a640 1 -- 172.21.15.63:0/95919399 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f268c13e990 con 0x7f268c09fe60 2024-04-26T04:24:37.702 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.699+0000 7f26897fa640 1 -- 172.21.15.63:0/95919399 <== mon.2 v2:172.21.15.63:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f267c0479d0 con 0x7f268c09fe60 2024-04-26T04:24:37.703 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.699+0000 7f26897fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T04:24:37.703 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.699+0000 7f26897fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:24:37.703 INFO:teuthology.orchestra.run.smithi063.stdout:{"election_epoch":26,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cd1c6a0c-4e4b-4014-89c0-2e535479ef39","modified":"2024-04-26T04:18:03.702339Z","created":"2024-04-26T04:18:03.702339Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3300","nonce":0},{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:3300","nonce":0},{"type":"v1","addr":"172.21.15.186:6789","nonce":0}]},"addr":"172.21.15.186:6789/0","public_addr":"172.21.15.186:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3301","nonce":0},{"type":"v1","addr":"172.21.15.63:6790","nonce":0}]},"addr":"172.21.15.63:6790/0","public_addr":"172.21.15.63:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T04:24:37.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.699+0000 7f2692c1a640 1 -- 172.21.15.63:0/95919399 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f2678035ec0 msgr2=0x7f2678038380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:37.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.699+0000 7f2692c1a640 1 --2- 172.21.15.63:0/95919399 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f2678035ec0 0x7f2678038380 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f26800ab000 tx=0x7f26800a6000 comp rx=0 tx=0).stop 2024-04-26T04:24:37.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.699+0000 7f2692c1a640 10 monclient: shutdown 2024-04-26T04:24:37.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.699+0000 7f2692c1a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:24:37.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.699+0000 7f2692c1a640 1 -- 172.21.15.63:0/95919399 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f268c09fe60 msgr2=0x7f268c1cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:37.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.699+0000 7f2692c1a640 1 --2- 172.21.15.63:0/95919399 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f268c09fe60 0x7f268c1cf440 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f267c00e930 tx=0x7f267c00edf0 comp rx=0 tx=0).stop 2024-04-26T04:24:37.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.699+0000 7f2692c1a640 1 -- 172.21.15.63:0/95919399 shutdown_connections 2024-04-26T04:24:37.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.699+0000 7f2692c1a640 1 --2- 172.21.15.63:0/95919399 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f2678035ec0 0x7f2678038380 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:37.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.699+0000 7f2692c1a640 1 --2- 172.21.15.63:0/95919399 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f268c0a2f70 0x7f268c13e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:37.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.699+0000 7f2692c1a640 1 --2- 172.21.15.63:0/95919399 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f268c0a0240 0x7f268c13df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:37.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.699+0000 7f2692c1a640 1 --2- 172.21.15.63:0/95919399 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f268c09fe60 0x7f268c1cf440 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:37.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.699+0000 7f2692c1a640 1 -- 172.21.15.63:0/95919399 >> 172.21.15.63:0/95919399 conn(0x7f268c0a5fb0 msgr2=0x7f268c135ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:24:37.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.699+0000 7f2692c1a640 1 -- 172.21.15.63:0/95919399 shutdown_connections 2024-04-26T04:24:37.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.699+0000 7f2692c1a640 1 -- 172.21.15.63:0/95919399 wait complete. 2024-04-26T04:24:37.720 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-26T04:24:37.720 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-26T04:24:37.720 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T04:24:37.815 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:24:37.815 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[30536]: starting. 2024-04-26T04:24:37.815 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[30536]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-30500 2024-04-26T04:24:37.815 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[30536]: shutting down. 2024-04-26T04:24:37.818 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:24:37.819 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:24:37.819 INFO:teuthology.orchestra.run.smithi186.stdout:******* SUCCESS ********** 2024-04-26T04:24:37.821 DEBUG:teuthology.orchestra.run.smithi186:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-26T04:24:37.844 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.839+0000 7f29fd3a4640 10 monclient: get_monmap_and_config 2024-04-26T04:24:37.845 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.839+0000 7f29fd3a4640 10 monclient: build_initial_monmap 2024-04-26T04:24:37.845 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.839+0000 7f29fd3a4640 10 monclient: monmap: 2024-04-26T04:24:37.845 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:24:37.845 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:37.845 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:24:37.844803+0000 2024-04-26T04:24:37.845 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:24:37.844803+0000 2024-04-26T04:24:37.845 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:24:37.845 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:37.845 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:24:37.845 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:24:37.845 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:24:37.845 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:24:37.845 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:24:37.845 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:37.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.839+0000 7f29fd3a4640 1 Processor -- start 2024-04-26T04:24:37.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.839+0000 7f29fd3a4640 1 -- start start 2024-04-26T04:24:37.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.839+0000 7f29fd3a4640 10 monclient: init 2024-04-26T04:24:37.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.843+0000 7f29fd3a4640 10 monclient: _reopen_session rank -1 2024-04-26T04:24:37.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.843+0000 7f29fd3a4640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-04-26T04:24:37.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.843+0000 7f29fd3a4640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f29f809c6a0 0x7f29f80996b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:37.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.843+0000 7f29fd3a4640 10 monclient(hunting): picked mon.noname-b con 0x7f29f809c6a0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:24:37.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.843+0000 7f29fd3a4640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f29f809ca80 0x7f29f809ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:37.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.843+0000 7f29fd3a4640 10 monclient(hunting): picked mon.noname-c con 0x7f29f809ca80 addr v2:172.21.15.186:3300/0 2024-04-26T04:24:37.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.843+0000 7f29fd3a4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f29f8098090 addr v1:172.21.15.63:6789/0 2024-04-26T04:24:37.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.843+0000 7f29fd3a4640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29f809d9f0 con 0x7f29f8098090 2024-04-26T04:24:37.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.843+0000 7f29fd3a4640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:37.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.843+0000 7f29fd3a4640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f29f809c1d0 con 0x7f29f809ca80 2024-04-26T04:24:37.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.843+0000 7f29fd3a4640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:37.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.843+0000 7f29fd3a4640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f29f80992d0 con 0x7f29f809c6a0 2024-04-26T04:24:37.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.843+0000 7f29fd3a4640 10 monclient(hunting): _renew_subs 2024-04-26T04:24:37.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.843+0000 7f29fd3a4640 10 monclient(hunting): authenticate will time out at 1309.552330s 2024-04-26T04:24:37.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.843+0000 7f29f77fe640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f29f8098090 0x7f29f80984a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:55406/0 (socket says 172.21.15.63:55406) 2024-04-26T04:24:37.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.843+0000 7f29f77fe640 1 -- 172.21.15.63:0/3820141468 learned_addr learned my addr 172.21.15.63:0/3820141468 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-04-26T04:24:37.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.843+0000 7f29f6ffd640 1 --2- 172.21.15.63:0/3820141468 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f29f809c6a0 0x7f29f80996b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:37.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.843+0000 7f29f6ffd640 10 monclient(hunting): get_auth_request con 0x7f29f809c6a0 auth_method 0 2024-04-26T04:24:37.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.843+0000 7f29f6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:37.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.843+0000 7f29f6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:37.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.843+0000 7f29f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:37.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.843+0000 7f29f6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:37.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.843+0000 7f29f6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:37.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.843+0000 7f29f6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:24:37.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.843+0000 7f29f6ffd640 10 monclient(hunting): handle_auth_done global_id 24224 payload 274 2024-04-26T04:24:37.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.843+0000 7f29f6ffd640 1 -- 172.21.15.63:0/3820141468 >> v2:172.21.15.186:3300/0 conn(0x7f29f809ca80 msgr2=0x7f29f809ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:24:37.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.843+0000 7f29f6ffd640 1 --2- 172.21.15.63:0/3820141468 >> v2:172.21.15.186:3300/0 conn(0x7f29f809ca80 0x7f29f809ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:37.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.843+0000 7f29f6ffd640 1 -- 172.21.15.63:0/3820141468 >> v1:172.21.15.63:6789/0 conn(0x7f29f8098090 legacy=0x7f29f80984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:24:37.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.843+0000 7f29f6ffd640 10 monclient: _finish_hunting 0 2024-04-26T04:24:37.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.843+0000 7f29f6ffd640 1 monclient: found mon.noname-b 2024-04-26T04:24:37.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.843+0000 7f29f6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:24:37.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.843+0000 7f29f6ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.63:3301/0 2024-04-26T04:24:37.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.843+0000 7f29f6ffd640 1 -- 172.21.15.63:0/3820141468 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29f809dc30 con 0x7f29f809c6a0 2024-04-26T04:24:37.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29f6ffd640 1 -- 172.21.15.63:0/3820141468 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f29f809c6a0 msgr2=0x7f29f80996b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:24:37.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29f6ffd640 1 --2- 172.21.15.63:0/3820141468 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f29f809c6a0 0x7f29f80996b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f29f806bde0 tx=0x7f29e001c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7dd73095fe1296af server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:37.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29f5ffb640 1 -- 172.21.15.63:0/3820141468 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29e00b0070 con 0x7f29f809c6a0 2024-04-26T04:24:37.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29f5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:37.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29f5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T04:24:37.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29f5ffb640 10 monclient: dump: 2024-04-26T04:24:37.855 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:37.855 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:37.855 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:37.855 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:37.855 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:37.855 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:37.855 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:37.855 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:37.855 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:37.855 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:37.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29f5ffb640 10 monclient: _finish_auth 0 2024-04-26T04:24:37.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29f5ffb640 10 monclient: _check_auth_tickets 2024-04-26T04:24:37.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29f5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:24:37.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29f5ffb640 1 -- 172.21.15.63:0/3820141468 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29e00ab600 con 0x7f29f809c6a0 2024-04-26T04:24:37.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29f5ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T04:24:37.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29f5ffb640 1 -- 172.21.15.63:0/3820141468 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29e00ab9d0 con 0x7f29f809c6a0 2024-04-26T04:24:37.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29fd3a4640 5 monclient: authenticate success, global_id 24224 2024-04-26T04:24:37.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29fd3a4640 10 monclient: get_monmap_and_config success 2024-04-26T04:24:37.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29fd3a4640 4 set_mon_vals no callback set 2024-04-26T04:24:37.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29f5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:37.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29f5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:24:37.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29f5ffb640 10 monclient: dump: 2024-04-26T04:24:37.856 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:37.856 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:37.857 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:37.857 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:37.857 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:37.857 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:37.857 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:37.857 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:37.857 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:37.857 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:37.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29fd3a4640 10 monclient: shutdown 2024-04-26T04:24:37.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29fd3a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:24:37.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29fd3a4640 1 -- 172.21.15.63:0/3820141468 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f29f809c6a0 msgr2=0x7f29f80996b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:37.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29fd3a4640 1 --2- 172.21.15.63:0/3820141468 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f29f809c6a0 0x7f29f80996b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f29f806bde0 tx=0x7f29e001c2d0 comp rx=0 tx=0).stop 2024-04-26T04:24:37.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29fd3a4640 1 -- 172.21.15.63:0/3820141468 shutdown_connections 2024-04-26T04:24:37.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29fd3a4640 1 --2- 172.21.15.63:0/3820141468 >> v2:172.21.15.186:3300/0 conn(0x7f29f809ca80 0x7f29f809ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:37.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29fd3a4640 1 --2- 172.21.15.63:0/3820141468 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f29f809c6a0 0x7f29f80996b0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:37.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29fd3a4640 1 -- 172.21.15.63:0/3820141468 >> 172.21.15.63:0/3820141468 conn(0x7f29f80a1e70 msgr2=0x7f29f80a2280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:24:37.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29fd3a4640 1 -- 172.21.15.63:0/3820141468 shutdown_connections 2024-04-26T04:24:37.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29fd3a4640 1 -- 172.21.15.63:0/3820141468 wait complete. 2024-04-26T04:24:37.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29fd3a4640 10 monclient: build_initial_monmap 2024-04-26T04:24:37.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29fd3a4640 10 monclient: monmap: 2024-04-26T04:24:37.858 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:24:37.858 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:24:37.858 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:24:37.858 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:24:37.858 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:24:37.858 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:37.858 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:24:37.858 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:24:37.859 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:24:37.859 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:37.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29fd3a4640 1 Processor -- start 2024-04-26T04:24:37.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29fd3a4640 1 -- start start 2024-04-26T04:24:37.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29fd3a4640 10 monclient: init 2024-04-26T04:24:37.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29fd3a4640 10 monclient: _reopen_session rank -1 2024-04-26T04:24:37.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29fd3a4640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T04:24:37.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29fd3a4640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f29f8098090 0x7f29f813d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:37.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29fd3a4640 10 monclient(hunting): picked mon.noname-c con 0x7f29f8098090 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:24:37.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29fd3a4640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29f809c6a0 0x7f29f8071070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:37.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29fd3a4640 10 monclient(hunting): picked mon.noname-a con 0x7f29f809c6a0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:24:37.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29fd3a4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f29f809ca80 0x7f29f80715b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:37.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29fd3a4640 10 monclient(hunting): picked mon.noname-b con 0x7f29f809ca80 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:24:37.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29fd3a4640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:37.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29fd3a4640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f29f80a1820 con 0x7f29f809c6a0 2024-04-26T04:24:37.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29fd3a4640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:37.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29fd3a4640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f29f80a16a0 con 0x7f29f809ca80 2024-04-26T04:24:37.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29fd3a4640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:37.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29fd3a4640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f29f813b760 con 0x7f29f8098090 2024-04-26T04:24:37.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29f6ffd640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f29f8098090 0x7f29f813d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:37.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29f67fc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29f809c6a0 0x7f29f8071070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:37.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29f6ffd640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f29f8098090 0x7f29f813d150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:40108/0 (socket says 172.21.15.63:40108) 2024-04-26T04:24:37.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29f6ffd640 1 -- 172.21.15.63:0/3584283482 learned_addr learned my addr 172.21.15.63:0/3584283482 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:24:37.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29fd3a4640 10 monclient(hunting): _renew_subs 2024-04-26T04:24:37.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29fd3a4640 10 monclient(hunting): authenticate will time out at 1309.559292s 2024-04-26T04:24:37.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29f6ffd640 10 monclient(hunting): get_auth_request con 0x7f29f8098090 auth_method 0 2024-04-26T04:24:37.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29f6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:37.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29f6ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:37.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:37.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29f67fc640 10 monclient(hunting): get_auth_request con 0x7f29f809c6a0 auth_method 0 2024-04-26T04:24:37.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29f67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:37.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29f67fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:37.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:37.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29f6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:37.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29f6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:37.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29f6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:24:37.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29f67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:37.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29f67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:37.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29f67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:24:37.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29f77fe640 1 --2- 172.21.15.63:0/3584283482 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f29f809ca80 0x7f29f80715b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:37.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29f6ffd640 10 monclient(hunting): handle_auth_done global_id 24227 payload 824 2024-04-26T04:24:37.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29f6ffd640 1 -- 172.21.15.63:0/3584283482 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f29f809ca80 msgr2=0x7f29f80715b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:37.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29f6ffd640 1 --2- 172.21.15.63:0/3584283482 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f29f809ca80 0x7f29f80715b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:37.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29f6ffd640 1 -- 172.21.15.63:0/3584283482 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29f809c6a0 msgr2=0x7f29f8071070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:37.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29f6ffd640 1 --2- 172.21.15.63:0/3584283482 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29f809c6a0 0x7f29f8071070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:37.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29f6ffd640 10 monclient: _finish_hunting 0 2024-04-26T04:24:37.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29f6ffd640 1 monclient: found mon.noname-c 2024-04-26T04:24:37.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29f6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:24:37.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29f6ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.63:3301/0 2024-04-26T04:24:37.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29f6ffd640 1 -- 172.21.15.63:0/3584283482 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29f812fce0 con 0x7f29f8098090 2024-04-26T04:24:37.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29f77fe640 10 monclient: get_auth_request con 0x7f29f809ca80 auth_method 0 2024-04-26T04:24:37.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.847+0000 7f29f77fe640 1 --2- 172.21.15.63:0/3584283482 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f29f809ca80 0x7f29f80715b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T04:24:37.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.851+0000 7f29f6ffd640 1 -- 172.21.15.63:0/3584283482 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f29f8098090 msgr2=0x7f29f813d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:24:37.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.851+0000 7f29f6ffd640 1 --2- 172.21.15.63:0/3584283482 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f29f8098090 0x7f29f813d150 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f29e00ae040 tx=0x7f29e001ab10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:37.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.851+0000 7f29ef7fe640 1 -- 172.21.15.63:0/3584283482 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29e00b0050 con 0x7f29f8098090 2024-04-26T04:24:37.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.851+0000 7f29ef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:37.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.851+0000 7f29ef7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:24:37.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.851+0000 7f29ef7fe640 10 monclient: dump: 2024-04-26T04:24:37.867 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:37.867 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:37.867 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:37.867 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:37.867 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:37.867 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:37.867 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:37.867 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:37.868 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:37.868 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:37.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.851+0000 7f29ef7fe640 10 monclient: _finish_auth 0 2024-04-26T04:24:37.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.851+0000 7f29ef7fe640 10 monclient: _check_auth_tickets 2024-04-26T04:24:37.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.851+0000 7f29ef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:24:37.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.851+0000 7f29ef7fe640 1 -- 172.21.15.63:0/3584283482 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29e00bc070 con 0x7f29f8098090 2024-04-26T04:24:37.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.851+0000 7f29fd3a4640 5 monclient: authenticate success, global_id 24227 2024-04-26T04:24:37.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.851+0000 7f29fd3a4640 10 monclient: _renew_subs 2024-04-26T04:24:37.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.851+0000 7f29fd3a4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:24:37.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.851+0000 7f29fd3a4640 1 -- 172.21.15.63:0/3584283482 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29f8072420 con 0x7f29f8098090 2024-04-26T04:24:37.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.851+0000 7f29ef7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T04:24:37.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.851+0000 7f29ef7fe640 1 -- 172.21.15.63:0/3584283482 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29e00b5a80 con 0x7f29f8098090 2024-04-26T04:24:37.869 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.851+0000 7f29ef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:37.869 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.851+0000 7f29ef7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:24:37.869 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.851+0000 7f29ef7fe640 10 monclient: dump: 2024-04-26T04:24:37.869 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:37.869 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:37.869 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:37.869 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:37.869 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:37.869 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:37.869 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:37.869 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:37.869 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:37.869 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:37.869 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.851+0000 7f29f57fa640 4 set_mon_vals no callback set 2024-04-26T04:24:37.869 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.851+0000 7f29fd3a4640 10 monclient: _renew_subs 2024-04-26T04:24:37.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.851+0000 7f29fd3a4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:24:37.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.851+0000 7f29fd3a4640 1 -- 172.21.15.63:0/3584283482 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f29f813cc30 con 0x7f29f8098090 2024-04-26T04:24:37.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.851+0000 7f29fd3a4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:24:37.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.851+0000 7f29fd3a4640 1 --2- 172.21.15.63:0/3584283482 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29f8071af0 0x7f29f806db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:37.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.851+0000 7f29fd3a4640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:37.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.851+0000 7f29fd3a4640 1 -- 172.21.15.63:0/3584283482 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f29f80504d0 con 0x7f29f8071af0 2024-04-26T04:24:37.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.851+0000 7f29f67fc640 1 --2- 172.21.15.63:0/3584283482 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29f8071af0 0x7f29f806db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:37.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.851+0000 7f29ef7fe640 1 -- 172.21.15.63:0/3584283482 <== mon.2 v2:172.21.15.63:3301/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f29e00b5c20 con 0x7f29f8098090 2024-04-26T04:24:37.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.851+0000 7f29f67fc640 10 monclient: get_auth_request con 0x7f29f8071af0 auth_method 0 2024-04-26T04:24:37.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.851+0000 7f29f67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:37.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.851+0000 7f29f67fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:37.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.851+0000 7f29f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:37.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.851+0000 7f29f67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:37.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.851+0000 7f29f67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:37.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.851+0000 7f29f67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:24:37.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.851+0000 7f29ef7fe640 1 --2- 172.21.15.63:0/3584283482 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f29d8035f10 0x7f29d80383d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:37.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.851+0000 7f29ef7fe640 1 -- 172.21.15.63:0/3584283482 <== mon.2 v2:172.21.15.63:3301/0 5 ==== osd_map(220..220 src has 1..220) ==== 4346+0+0 (secure 0 0 0) 0x7f29e00e3660 con 0x7f29f8098090 2024-04-26T04:24:37.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.851+0000 7f29f67fc640 10 monclient(hunting): handle_auth_done global_id 5325 payload 824 2024-04-26T04:24:37.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.851+0000 7f29f67fc640 1 -- 172.21.15.63:0/3584283482 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f29f8071ed0 con 0x7f29f8071af0 2024-04-26T04:24:37.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.851+0000 7f29f77fe640 1 --2- 172.21.15.63:0/3584283482 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f29d8035f10 0x7f29d80383d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:37.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.851+0000 7f29f77fe640 10 monclient: get_auth_request con 0x7f29d8035f10 auth_method 0 2024-04-26T04:24:37.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.855+0000 7f29f77fe640 1 -- 172.21.15.63:0/3584283482 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f29d8035f10 msgr2=0x7f29d80383d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-04-26T04:24:37.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.855+0000 7f29f77fe640 1 -- 172.21.15.63:0/3584283482 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f29d8035f10 msgr2=0x7f29d80383d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T04:24:37.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.855+0000 7f29f77fe640 1 --2- 172.21.15.63:0/3584283482 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f29d8035f10 0x7f29d80383d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7f29f812f370 tx=0x7f29f0009040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:24:37.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.855+0000 7f29f77fe640 1 --2- 172.21.15.63:0/3584283482 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f29d8035f10 0x7f29d80383d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:24:37.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.855+0000 7f29f67fc640 1 -- 172.21.15.63:0/3584283482 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29f8071af0 msgr2=0x7f29f806db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:24:37.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.855+0000 7f29f67fc640 1 --2- 172.21.15.63:0/3584283482 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29f8071af0 0x7f29f806db00 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f29f809d220 tx=0x7f29e800ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:37.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.855+0000 7f29ef7fe640 1 -- 172.21.15.63:0/3584283482 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29e8017070 con 0x7f29f8071af0 2024-04-26T04:24:37.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.855+0000 7f29ef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:37.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.855+0000 7f29ef7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:24:37.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.855+0000 7f29ef7fe640 10 monclient: dump: 2024-04-26T04:24:37.872 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:37.872 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:37.872 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:37.872 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:37.872 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:37.872 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:37.872 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:37.872 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:37.872 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:37.872 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:37.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.855+0000 7f29ef7fe640 1 -- 172.21.15.63:0/3584283482 <== mon.0 v2:172.21.15.63:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f29e800c980 con 0x7f29f8071af0 2024-04-26T04:24:37.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.855+0000 7f29ef7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:24:37.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.855+0000 7f29ef7fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:24:37.873 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.855+0000 7f29ef7fe640 1 -- 172.21.15.63:0/3584283482 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29f8071af0 msgr2=0x7f29f806db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:37.873 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.855+0000 7f29ef7fe640 1 --2- 172.21.15.63:0/3584283482 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29f8071af0 0x7f29f806db00 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f29f809d220 tx=0x7f29e800ec00 comp rx=0 tx=0).stop 2024-04-26T04:24:37.873 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.855+0000 7f29ef7fe640 1 -- 172.21.15.63:0/3584283482 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29f8071af0 msgr2=0x7f29f806db00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:24:37.873 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.855+0000 7f29ef7fe640 1 --2- 172.21.15.63:0/3584283482 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29f8071af0 0x7f29f806db00 secure :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f29f809d220 tx=0x7f29e800ec00 comp rx=0 tx=0).stop 2024-04-26T04:24:37.873 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.859+0000 7f29fd3a4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T04:24:37.873 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.859+0000 7f29fd3a4640 1 --2- 172.21.15.63:0/3584283482 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29f806e040 0x7f29f806e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:37.873 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.859+0000 7f29fd3a4640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:37.873 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.859+0000 7f29fd3a4640 1 -- 172.21.15.63:0/3584283482 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f29f812f680 con 0x7f29f806e040 2024-04-26T04:24:37.873 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.859+0000 7f29f67fc640 1 --2- 172.21.15.63:0/3584283482 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29f806e040 0x7f29f806e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:37.873 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.859+0000 7f29f67fc640 10 monclient: get_auth_request con 0x7f29f806e040 auth_method 0 2024-04-26T04:24:37.873 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.859+0000 7f29f67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:37.873 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.859+0000 7f29f67fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:37.873 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.859+0000 7f29f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:37.873 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.859+0000 7f29f67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:37.874 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.859+0000 7f29f67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:37.874 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.859+0000 7f29f67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:24:37.874 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.863+0000 7f29f67fc640 10 monclient(hunting): handle_auth_done global_id 5328 payload 824 2024-04-26T04:24:37.874 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.863+0000 7f29f67fc640 1 -- 172.21.15.63:0/3584283482 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f29f8062e10 con 0x7f29f806e040 2024-04-26T04:24:37.878 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.863+0000 7f29f67fc640 1 -- 172.21.15.63:0/3584283482 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29f806e040 msgr2=0x7f29f806e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:24:37.878 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.863+0000 7f29f67fc640 1 --2- 172.21.15.63:0/3584283482 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29f806e040 0x7f29f806e450 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f29e8019000 tx=0x7f29e80047e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:37.878 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.863+0000 7f29ef7fe640 1 -- 172.21.15.63:0/3584283482 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29e8017070 con 0x7f29f806e040 2024-04-26T04:24:37.878 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.863+0000 7f29ef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:37.879 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.863+0000 7f29ef7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:24:37.879 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.863+0000 7f29ef7fe640 10 monclient: dump: 2024-04-26T04:24:37.879 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:37.879 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:37.879 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:37.879 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:37.879 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:37.879 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:37.879 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:37.879 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:37.879 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:37.879 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:37.879 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.863+0000 7f29ef7fe640 1 -- 172.21.15.63:0/3584283482 <== mon.0 v2:172.21.15.63:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4723 (secure 0 0 0) 0x7f29e8006600 con 0x7f29f806e040 2024-04-26T04:24:37.879 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.863+0000 7f29ef7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T04:24:37.880 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.863+0000 7f29ef7fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:24:37.880 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.863+0000 7f29ef7fe640 1 -- 172.21.15.63:0/3584283482 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29f806e040 msgr2=0x7f29f806e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:37.880 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.863+0000 7f29ef7fe640 1 --2- 172.21.15.63:0/3584283482 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29f806e040 0x7f29f806e450 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f29e8019000 tx=0x7f29e80047e0 comp rx=0 tx=0).stop 2024-04-26T04:24:37.880 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.863+0000 7f29ef7fe640 1 -- 172.21.15.63:0/3584283482 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29f806e040 msgr2=0x7f29f806e450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:24:37.880 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.863+0000 7f29ef7fe640 1 --2- 172.21.15.63:0/3584283482 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29f806e040 0x7f29f806e450 secure :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f29e8019000 tx=0x7f29e80047e0 comp rx=0 tx=0).stop 2024-04-26T04:24:37.880 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-26T04:24:37.880 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a", 2024-04-26T04:24:37.880 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 0, 2024-04-26T04:24:37.880 INFO:teuthology.orchestra.run.smithi063.stdout: "state": "leader", 2024-04-26T04:24:37.880 INFO:teuthology.orchestra.run.smithi063.stdout: "election_epoch": 26, 2024-04-26T04:24:37.881 INFO:teuthology.orchestra.run.smithi063.stdout: "uptime": 369016, 2024-04-26T04:24:37.881 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum": [ 2024-04-26T04:24:37.881 INFO:teuthology.orchestra.run.smithi063.stdout: 0, 2024-04-26T04:24:37.881 INFO:teuthology.orchestra.run.smithi063.stdout: 1, 2024-04-26T04:24:37.881 INFO:teuthology.orchestra.run.smithi063.stdout: 2 2024-04-26T04:24:37.881 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:24:37.881 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_age": 1, 2024-04-26T04:24:37.881 INFO:teuthology.orchestra.run.smithi063.stdout: "features": { 2024-04-26T04:24:37.881 INFO:teuthology.orchestra.run.smithi063.stdout: "required_con": "2450521708007866372", 2024-04-26T04:24:37.881 INFO:teuthology.orchestra.run.smithi063.stdout: "required_mon": [ 2024-04-26T04:24:37.881 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:24:37.881 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:24:37.881 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:24:37.882 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:24:37.882 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:24:37.882 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:24:37.882 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:24:37.882 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:24:37.882 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:24:37.882 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:24:37.882 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:24:37.882 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:24:37.882 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_con": "4540701547738038271", 2024-04-26T04:24:37.882 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_mon": [ 2024-04-26T04:24:37.882 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:24:37.882 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:24:37.882 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:24:37.882 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:24:37.883 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:24:37.883 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:24:37.883 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:24:37.883 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:24:37.883 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:24:37.883 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:24:37.883 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:24:37.883 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:24:37.883 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:37.883 INFO:teuthology.orchestra.run.smithi063.stdout: "outside_quorum": [], 2024-04-26T04:24:37.883 INFO:teuthology.orchestra.run.smithi063.stdout: "extra_probe_peers": [], 2024-04-26T04:24:37.883 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_provider": [], 2024-04-26T04:24:37.883 INFO:teuthology.orchestra.run.smithi063.stdout: "monmap": { 2024-04-26T04:24:37.883 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 1, 2024-04-26T04:24:37.883 INFO:teuthology.orchestra.run.smithi063.stdout: "fsid": "cd1c6a0c-4e4b-4014-89c0-2e535479ef39", 2024-04-26T04:24:37.884 INFO:teuthology.orchestra.run.smithi063.stdout: "modified": "2024-04-26T04:18:03.702339Z", 2024-04-26T04:24:37.884 INFO:teuthology.orchestra.run.smithi063.stdout: "created": "2024-04-26T04:18:03.702339Z", 2024-04-26T04:24:37.884 INFO:teuthology.orchestra.run.smithi063.stdout: "min_mon_release": 19, 2024-04-26T04:24:37.884 INFO:teuthology.orchestra.run.smithi063.stdout: "min_mon_release_name": "squid", 2024-04-26T04:24:37.884 INFO:teuthology.orchestra.run.smithi063.stdout: "election_strategy": 1, 2024-04-26T04:24:37.884 INFO:teuthology.orchestra.run.smithi063.stdout: "disallowed_leaders: ": "", 2024-04-26T04:24:37.884 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": false, 2024-04-26T04:24:37.884 INFO:teuthology.orchestra.run.smithi063.stdout: "tiebreaker_mon": "", 2024-04-26T04:24:37.884 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_ranks: ": "", 2024-04-26T04:24:37.884 INFO:teuthology.orchestra.run.smithi063.stdout: "features": { 2024-04-26T04:24:37.884 INFO:teuthology.orchestra.run.smithi063.stdout: "persistent": [ 2024-04-26T04:24:37.884 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:24:37.884 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:24:37.884 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:24:37.884 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:24:37.885 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:24:37.885 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:24:37.885 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:24:37.885 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:24:37.885 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:24:37.885 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:24:37.885 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:24:37.885 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:24:37.885 INFO:teuthology.orchestra.run.smithi063.stdout: "optional": [] 2024-04-26T04:24:37.885 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:37.885 INFO:teuthology.orchestra.run.smithi063.stdout: "mons": [ 2024-04-26T04:24:37.885 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:37.885 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 0, 2024-04-26T04:24:37.885 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a", 2024-04-26T04:24:37.885 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:24:37.886 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:24:37.886 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:37.886 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:24:37.886 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:3300", 2024-04-26T04:24:37.886 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:24:37.886 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:37.886 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:37.886 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:24:37.886 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6789", 2024-04-26T04:24:37.886 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:24:37.886 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:24:37.886 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:24:37.886 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:37.886 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6789/0", 2024-04-26T04:24:37.887 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6789/0", 2024-04-26T04:24:37.887 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:24:37.887 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:24:37.887 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:24:37.887 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:37.887 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:37.887 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 1, 2024-04-26T04:24:37.887 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b", 2024-04-26T04:24:37.887 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:24:37.887 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:24:37.887 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:37.887 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:24:37.887 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:3300", 2024-04-26T04:24:37.887 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:24:37.887 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:37.888 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:37.888 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:24:37.888 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:6789", 2024-04-26T04:24:37.888 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:24:37.888 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:24:37.888 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:24:37.888 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:37.888 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:6789/0", 2024-04-26T04:24:37.888 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.186:6789/0", 2024-04-26T04:24:37.888 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:24:37.888 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:24:37.888 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:24:37.888 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:37.888 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:37.888 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 2, 2024-04-26T04:24:37.889 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "c", 2024-04-26T04:24:37.889 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:24:37.889 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:24:37.889 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:37.889 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:24:37.889 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:3301", 2024-04-26T04:24:37.889 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:24:37.889 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:37.889 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:37.889 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:24:37.889 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6790", 2024-04-26T04:24:37.889 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:24:37.889 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:24:37.889 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:24:37.890 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:37.890 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6790/0", 2024-04-26T04:24:37.890 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6790/0", 2024-04-26T04:24:37.890 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:24:37.890 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:24:37.890 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:24:37.890 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:24:37.890 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:24:37.890 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:37.890 INFO:teuthology.orchestra.run.smithi063.stdout: "feature_map": { 2024-04-26T04:24:37.891 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": [ 2024-04-26T04:24:37.891 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:37.891 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:24:37.891 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:24:37.891 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 1 2024-04-26T04:24:37.891 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:24:37.891 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:24:37.891 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": [ 2024-04-26T04:24:37.891 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:37.891 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:24:37.891 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:24:37.892 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 3 2024-04-26T04:24:37.892 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:24:37.892 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:24:37.892 INFO:teuthology.orchestra.run.smithi063.stdout: "client": [ 2024-04-26T04:24:37.892 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:37.892 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:24:37.892 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:24:37.892 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 3 2024-04-26T04:24:37.892 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:24:37.892 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:24:37.892 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:37.892 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": false 2024-04-26T04:24:37.892 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-26T04:24:37.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.863+0000 7f29fd3a4640 1 -- 172.21.15.63:0/3584283482 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f29d8035f10 msgr2=0x7f29d80383d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T04:24:37.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.863+0000 7f29fd3a4640 1 --2- 172.21.15.63:0/3584283482 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f29d8035f10 0x7f29d80383d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:37.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.863+0000 7f29fd3a4640 10 monclient: shutdown 2024-04-26T04:24:37.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.863+0000 7f29fd3a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:24:37.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.863+0000 7f29fd3a4640 1 -- 172.21.15.63:0/3584283482 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f29f8098090 msgr2=0x7f29f813d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:37.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.863+0000 7f29fd3a4640 1 --2- 172.21.15.63:0/3584283482 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f29f8098090 0x7f29f813d150 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f29e00ae040 tx=0x7f29e001ab10 comp rx=0 tx=0).stop 2024-04-26T04:24:37.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.863+0000 7f29f77fe640 1 -- 172.21.15.63:0/3584283482 reap_dead start 2024-04-26T04:24:37.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.863+0000 7f29f77fe640 1 -- 172.21.15.63:0/3584283482 reap_dead start 2024-04-26T04:24:37.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.863+0000 7f29fd3a4640 1 -- 172.21.15.63:0/3584283482 shutdown_connections 2024-04-26T04:24:37.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.863+0000 7f29fd3a4640 1 -- 172.21.15.63:0/3584283482 >> 172.21.15.63:0/3584283482 conn(0x7f29f80a1e70 msgr2=0x7f29f80a2fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:24:37.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.863+0000 7f29fd3a4640 1 -- 172.21.15.63:0/3584283482 shutdown_connections 2024-04-26T04:24:37.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.863+0000 7f29fd3a4640 1 -- 172.21.15.63:0/3584283482 wait complete. 2024-04-26T04:24:37.895 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T04:24:37.895 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-26T04:24:37.895 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T04:24:38.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.999+0000 7f318e5ae640 10 monclient: get_monmap_and_config 2024-04-26T04:24:38.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.999+0000 7f318e5ae640 10 monclient: build_initial_monmap 2024-04-26T04:24:38.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.999+0000 7f318e5ae640 10 monclient: monmap: 2024-04-26T04:24:38.004 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:24:38.005 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:38.005 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:24:38.004609+0000 2024-04-26T04:24:38.005 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:24:38.004609+0000 2024-04-26T04:24:38.005 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:24:38.005 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:38.005 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:24:38.005 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:24:38.005 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:24:38.005 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:24:38.005 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:24:38.005 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:38.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.999+0000 7f318e5ae640 1 Processor -- start 2024-04-26T04:24:38.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.999+0000 7f318e5ae640 1 -- start start 2024-04-26T04:24:38.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:37.999+0000 7f318e5ae640 10 monclient: init 2024-04-26T04:24:38.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.003+0000 7f318e5ae640 10 monclient: _reopen_session rank -1 2024-04-26T04:24:38.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.003+0000 7f318e5ae640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-04-26T04:24:38.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.003+0000 7f318e5ae640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f31880a0bb0 addr v1:172.21.15.63:6789/0 2024-04-26T04:24:38.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.003+0000 7f318e5ae640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3188136310 addr v1:172.21.15.186:6789/0 2024-04-26T04:24:38.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.003+0000 7f318e5ae640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f3188139e20 0x7f318813c210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:38.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.003+0000 7f318e5ae640 10 monclient(hunting): picked mon.noname-a con 0x7f3188139e20 addr v2:172.21.15.63:3300/0 2024-04-26T04:24:38.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.003+0000 7f318e5ae640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f318809f270 con 0x7f31880a0bb0 2024-04-26T04:24:38.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.003+0000 7f318e5ae640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31880a0f90 con 0x7f3188136310 2024-04-26T04:24:38.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.003+0000 7f318e5ae640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:38.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.003+0000 7f318e5ae640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7f3188099240 con 0x7f3188139e20 2024-04-26T04:24:38.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.003+0000 7f318e5ae640 10 monclient(hunting): _renew_subs 2024-04-26T04:24:38.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.003+0000 7f318e5ae640 10 monclient(hunting): authenticate will time out at 1309.712180s 2024-04-26T04:24:38.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.003+0000 7f318cb4b640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f3188139e20 0x7f318813c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:38.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.003+0000 7f3187fff640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f31880a0bb0 0x7f318809eb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:55410/0 (socket says 172.21.15.63:55410) 2024-04-26T04:24:38.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.003+0000 7f3187fff640 1 -- 172.21.15.63:0/1414952317 learned_addr learned my addr 172.21.15.63:0/1414952317 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-04-26T04:24:38.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.003+0000 7f318cb4b640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f3188139e20 0x7f318813c210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:33260/0 (socket says 172.21.15.63:33260) 2024-04-26T04:24:38.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.003+0000 7f318cb4b640 10 monclient(hunting): get_auth_request con 0x7f3188139e20 auth_method 0 2024-04-26T04:24:38.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.003+0000 7f318cb4b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:38.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.003+0000 7f318cb4b640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:38.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.003+0000 7f318cb4b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:38.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.003+0000 7f318cb4b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:38.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.003+0000 7f318cb4b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:38.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.003+0000 7f318cb4b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:24:38.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.003+0000 7f318cb4b640 10 monclient(hunting): handle_auth_done global_id 5343 payload 274 2024-04-26T04:24:38.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.003+0000 7f318cb4b640 1 -- 172.21.15.63:0/1414952317 >> v1:172.21.15.186:6789/0 conn(0x7f3188136310 legacy=0x7f3188138700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:24:38.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.003+0000 7f318cb4b640 1 -- 172.21.15.63:0/1414952317 >> v1:172.21.15.63:6789/0 conn(0x7f31880a0bb0 legacy=0x7f318809eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:24:38.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.003+0000 7f318cb4b640 10 monclient: _finish_hunting 0 2024-04-26T04:24:38.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.003+0000 7f318cb4b640 1 monclient: found mon.noname-a 2024-04-26T04:24:38.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.003+0000 7f318cb4b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:24:38.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.003+0000 7f318cb4b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:24:38.010 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.003+0000 7f318cb4b640 1 -- 172.21.15.63:0/1414952317 --> v2:172.21.15.63:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f318809b820 con 0x7f3188139e20 2024-04-26T04:24:38.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f318cb4b640 1 -- 172.21.15.63:0/1414952317 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3188139e20 msgr2=0x7f318813c210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:24:38.012 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f318cb4b640 1 --2- 172.21.15.63:0/1414952317 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3188139e20 0x7f318813c210 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f31780097f0 tx=0x7f317801c0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8afedc7c98b92fb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:38.012 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f3186ffd640 1 -- 172.21.15.63:0/1414952317 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31780b0070 con 0x7f3188139e20 2024-04-26T04:24:38.012 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f3186ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:38.012 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f3186ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:24:38.012 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f3186ffd640 10 monclient: dump: 2024-04-26T04:24:38.012 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:38.012 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:38.012 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:38.012 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:38.012 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:38.012 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:38.012 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:38.012 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:38.012 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:38.013 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:38.013 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f3186ffd640 10 monclient: _finish_auth 0 2024-04-26T04:24:38.013 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f3186ffd640 10 monclient: _check_auth_tickets 2024-04-26T04:24:38.013 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f3186ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:24:38.013 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f3186ffd640 1 -- 172.21.15.63:0/1414952317 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31780ab600 con 0x7f3188139e20 2024-04-26T04:24:38.013 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f3186ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T04:24:38.013 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f3186ffd640 1 -- 172.21.15.63:0/1414952317 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31780ab9d0 con 0x7f3188139e20 2024-04-26T04:24:38.013 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f318e5ae640 5 monclient: authenticate success, global_id 5343 2024-04-26T04:24:38.013 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f318e5ae640 10 monclient: get_monmap_and_config success 2024-04-26T04:24:38.013 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f318e5ae640 4 set_mon_vals no callback set 2024-04-26T04:24:38.013 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f3186ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:38.013 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f3186ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:24:38.013 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f3186ffd640 10 monclient: dump: 2024-04-26T04:24:38.013 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:38.013 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:38.014 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:38.014 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:38.014 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:38.014 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:38.014 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:38.014 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:38.014 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:38.014 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:38.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f318e5ae640 10 monclient: shutdown 2024-04-26T04:24:38.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f318e5ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:24:38.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f318e5ae640 1 -- 172.21.15.63:0/1414952317 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3188139e20 msgr2=0x7f318813c210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:38.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f318e5ae640 1 --2- 172.21.15.63:0/1414952317 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3188139e20 0x7f318813c210 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f31780097f0 tx=0x7f317801c0d0 comp rx=0 tx=0).stop 2024-04-26T04:24:38.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f318e5ae640 1 -- 172.21.15.63:0/1414952317 shutdown_connections 2024-04-26T04:24:38.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f318e5ae640 1 --2- 172.21.15.63:0/1414952317 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3188139e20 0x7f318813c210 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:38.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f318e5ae640 1 -- 172.21.15.63:0/1414952317 >> 172.21.15.63:0/1414952317 conn(0x7f31880a1e70 msgr2=0x7f31880a2280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:24:38.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f318e5ae640 1 -- 172.21.15.63:0/1414952317 shutdown_connections 2024-04-26T04:24:38.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f318e5ae640 1 -- 172.21.15.63:0/1414952317 wait complete. 2024-04-26T04:24:38.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f318e5ae640 10 monclient: build_initial_monmap 2024-04-26T04:24:38.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f318e5ae640 10 monclient: monmap: 2024-04-26T04:24:38.015 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:24:38.015 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:24:38.015 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:24:38.015 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:24:38.015 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:24:38.015 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:38.015 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:24:38.015 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:24:38.015 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:24:38.015 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:38.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f318e5ae640 1 Processor -- start 2024-04-26T04:24:38.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f318e5ae640 1 -- start start 2024-04-26T04:24:38.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f318e5ae640 10 monclient: init 2024-04-26T04:24:38.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f318e5ae640 10 monclient: _reopen_session rank -1 2024-04-26T04:24:38.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f318e5ae640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T04:24:38.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f318e5ae640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f31880a0bb0 0x7f318812aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:38.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f318e5ae640 10 monclient(hunting): picked mon.noname-c con 0x7f31880a0bb0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:24:38.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f318e5ae640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3188136310 0x7f3188135480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:38.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f318e5ae640 10 monclient(hunting): picked mon.noname-b con 0x7f3188136310 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:24:38.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f318e5ae640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3188139e20 0x7f31881359c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:38.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f318e5ae640 10 monclient(hunting): picked mon.noname-a con 0x7f3188139e20 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:24:38.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f318e5ae640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:38.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f318e5ae640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f318813f450 con 0x7f3188139e20 2024-04-26T04:24:38.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f318e5ae640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:38.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f318e5ae640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f318813f2d0 con 0x7f3188136310 2024-04-26T04:24:38.022 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f318e5ae640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:38.022 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f318e5ae640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f318813f5d0 con 0x7f31880a0bb0 2024-04-26T04:24:38.022 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f318e5ae640 10 monclient(hunting): _renew_subs 2024-04-26T04:24:38.022 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f318e5ae640 10 monclient(hunting): authenticate will time out at 1309.718889s 2024-04-26T04:24:38.022 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f3187fff640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f31880a0bb0 0x7f318812aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:38.022 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f318cb4b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3188139e20 0x7f31881359c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:38.022 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f318cb4b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3188139e20 0x7f31881359c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:33268/0 (socket says 172.21.15.63:33268) 2024-04-26T04:24:38.022 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f318cb4b640 1 -- 172.21.15.63:0/1763815446 learned_addr learned my addr 172.21.15.63:0/1763815446 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:24:38.022 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f318cb4b640 10 monclient(hunting): get_auth_request con 0x7f3188139e20 auth_method 0 2024-04-26T04:24:38.022 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f318cb4b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:38.022 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f318cb4b640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:38.022 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f318cb4b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:38.022 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f3187fff640 10 monclient(hunting): get_auth_request con 0x7f31880a0bb0 auth_method 0 2024-04-26T04:24:38.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f3187fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:38.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f3187fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:38.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f3187fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:38.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f318cb4b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:38.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f318cb4b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:38.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f318cb4b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:24:38.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f3187fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:38.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f3187fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:38.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f3187fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:24:38.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f31877fe640 1 --2- 172.21.15.63:0/1763815446 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3188136310 0x7f3188135480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:38.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f318cb4b640 10 monclient(hunting): handle_auth_done global_id 5346 payload 824 2024-04-26T04:24:38.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f318cb4b640 1 -- 172.21.15.63:0/1763815446 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f31880a0bb0 msgr2=0x7f318812aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:38.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f318cb4b640 1 --2- 172.21.15.63:0/1763815446 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f31880a0bb0 0x7f318812aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:38.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f318cb4b640 1 -- 172.21.15.63:0/1763815446 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3188136310 msgr2=0x7f3188135480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:38.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f318cb4b640 1 --2- 172.21.15.63:0/1763815446 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3188136310 0x7f3188135480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:38.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f318cb4b640 10 monclient: _finish_hunting 0 2024-04-26T04:24:38.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f318cb4b640 1 monclient: found mon.noname-a 2024-04-26T04:24:38.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f318cb4b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:24:38.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f318cb4b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:24:38.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f318cb4b640 1 -- 172.21.15.63:0/1763815446 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f318812cee0 con 0x7f3188139e20 2024-04-26T04:24:38.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f31877fe640 10 monclient: get_auth_request con 0x7f3188136310 auth_method 0 2024-04-26T04:24:38.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f3187fff640 1 --2- 172.21.15.63:0/1763815446 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f31880a0bb0 0x7f318812aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T04:24:38.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.007+0000 7f31877fe640 1 --2- 172.21.15.63:0/1763815446 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3188136310 0x7f3188135480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T04:24:38.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.011+0000 7f318cb4b640 1 -- 172.21.15.63:0/1763815446 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3188139e20 msgr2=0x7f31881359c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:24:38.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.011+0000 7f318cb4b640 1 --2- 172.21.15.63:0/1763815446 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3188139e20 0x7f31881359c0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f3178018ce0 tx=0x7f3178018e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:38.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.011+0000 7f3184ff9640 1 -- 172.21.15.63:0/1763815446 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31780b9070 con 0x7f3188139e20 2024-04-26T04:24:38.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.011+0000 7f3184ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:38.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.011+0000 7f3184ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:24:38.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.011+0000 7f3184ff9640 10 monclient: dump: 2024-04-26T04:24:38.025 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:38.025 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:38.025 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:38.025 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:38.025 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:38.025 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:38.025 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:38.025 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:38.025 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:38.025 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:38.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.011+0000 7f3184ff9640 10 monclient: _finish_auth 0 2024-04-26T04:24:38.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.011+0000 7f3184ff9640 10 monclient: _check_auth_tickets 2024-04-26T04:24:38.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.011+0000 7f3184ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:24:38.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.011+0000 7f3184ff9640 1 -- 172.21.15.63:0/1763815446 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31780b5690 con 0x7f3188139e20 2024-04-26T04:24:38.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.011+0000 7f318e5ae640 5 monclient: authenticate success, global_id 5346 2024-04-26T04:24:38.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.011+0000 7f318e5ae640 10 monclient: _renew_subs 2024-04-26T04:24:38.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.011+0000 7f318e5ae640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:24:38.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.011+0000 7f318e5ae640 1 -- 172.21.15.63:0/1763815446 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3188071f00 con 0x7f3188139e20 2024-04-26T04:24:38.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.011+0000 7f3184ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T04:24:38.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.011+0000 7f3184ff9640 1 -- 172.21.15.63:0/1763815446 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31780b0040 con 0x7f3188139e20 2024-04-26T04:24:38.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.011+0000 7f3184ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:38.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.011+0000 7f3184ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:24:38.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.011+0000 7f3184ff9640 10 monclient: dump: 2024-04-26T04:24:38.026 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:38.026 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:38.026 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:38.026 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:38.027 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:38.027 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:38.027 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:38.027 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:38.027 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:38.027 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:38.027 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.011+0000 7f318e5ae640 10 monclient: _renew_subs 2024-04-26T04:24:38.027 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.011+0000 7f318e5ae640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:24:38.027 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.011+0000 7f318e5ae640 1 -- 172.21.15.63:0/1763815446 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f318812af80 con 0x7f3188139e20 2024-04-26T04:24:38.027 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.011+0000 7f31867fc640 4 set_mon_vals no callback set 2024-04-26T04:24:38.027 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.011+0000 7f318e5ae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:24:38.027 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.011+0000 7f318e5ae640 1 --2- 172.21.15.63:0/1763815446 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f31881cc8d0 0x7f31881cccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:38.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.011+0000 7f318e5ae640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:38.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.011+0000 7f318e5ae640 1 -- 172.21.15.63:0/1763815446 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f318812cd60 con 0x7f31881cc8d0 2024-04-26T04:24:38.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.011+0000 7f3187fff640 1 --2- 172.21.15.63:0/1763815446 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f31881cc8d0 0x7f31881cccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:38.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.011+0000 7f3187fff640 10 monclient: get_auth_request con 0x7f31881cc8d0 auth_method 0 2024-04-26T04:24:38.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.011+0000 7f3187fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:38.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.011+0000 7f3187fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:38.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.011+0000 7f3187fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:38.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.011+0000 7f3187fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:38.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.011+0000 7f3187fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:38.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.011+0000 7f3187fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:24:38.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.011+0000 7f3187fff640 10 monclient(hunting): handle_auth_done global_id 24242 payload 824 2024-04-26T04:24:38.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.011+0000 7f3187fff640 1 -- 172.21.15.63:0/1763815446 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3188062e10 con 0x7f31881cc8d0 2024-04-26T04:24:38.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.011+0000 7f3184ff9640 1 -- 172.21.15.63:0/1763815446 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f31780bc070 con 0x7f3188139e20 2024-04-26T04:24:38.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.011+0000 7f3184ff9640 1 --2- 172.21.15.63:0/1763815446 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f3174035ef0 0x7f31740383b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:38.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.011+0000 7f3184ff9640 1 -- 172.21.15.63:0/1763815446 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(220..220 src has 1..220) ==== 4346+0+0 (secure 0 0 0) 0x7f31780e2ef0 con 0x7f3188139e20 2024-04-26T04:24:38.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.011+0000 7f31877fe640 1 --2- 172.21.15.63:0/1763815446 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f3174035ef0 0x7f31740383b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:38.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.011+0000 7f31877fe640 10 monclient: get_auth_request con 0x7f3174035ef0 auth_method 0 2024-04-26T04:24:38.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.015+0000 7f3187fff640 1 -- 172.21.15.63:0/1763815446 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f31881cc8d0 msgr2=0x7f31881cccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:24:38.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.015+0000 7f3187fff640 1 --2- 172.21.15.63:0/1763815446 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f31881cc8d0 0x7f31881cccb0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f318000d3b0 tx=0x7f318000d870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:38.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.015+0000 7f3184ff9640 1 -- 172.21.15.63:0/1763815446 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3180019070 con 0x7f31881cc8d0 2024-04-26T04:24:38.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.015+0000 7f3184ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:38.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.015+0000 7f3184ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:24:38.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.015+0000 7f3184ff9640 10 monclient: dump: 2024-04-26T04:24:38.029 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:38.029 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:38.029 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:38.029 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:38.029 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:38.029 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:38.030 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:38.030 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:38.030 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:38.030 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:38.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.015+0000 7f3184ff9640 1 -- 172.21.15.63:0/1763815446 <== mon.2 v2:172.21.15.63:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3180007530 con 0x7f31881cc8d0 2024-04-26T04:24:38.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.015+0000 7f3184ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:24:38.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.015+0000 7f3184ff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:24:38.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.015+0000 7f3184ff9640 1 -- 172.21.15.63:0/1763815446 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f31881cc8d0 msgr2=0x7f31881cccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:38.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.015+0000 7f3184ff9640 1 --2- 172.21.15.63:0/1763815446 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f31881cc8d0 0x7f31881cccb0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f318000d3b0 tx=0x7f318000d870 comp rx=0 tx=0).stop 2024-04-26T04:24:38.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.015+0000 7f3184ff9640 1 -- 172.21.15.63:0/1763815446 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f31881cc8d0 msgr2=0x7f31881cccb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:24:38.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.015+0000 7f3184ff9640 1 --2- 172.21.15.63:0/1763815446 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f31881cc8d0 0x7f31881cccb0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:38.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.015+0000 7f31877fe640 1 --2- 172.21.15.63:0/1763815446 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f3174035ef0 0x7f31740383b0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f318812a7d0 tx=0x7f317c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:38.032 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.019+0000 7f318e5ae640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T04:24:38.032 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.019+0000 7f318e5ae640 1 --2- 172.21.15.63:0/1763815446 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f31881cd1f0 0x7f318806da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:38.032 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.019+0000 7f318e5ae640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:38.032 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.019+0000 7f318e5ae640 1 -- 172.21.15.63:0/1763815446 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f31880728c0 con 0x7f31881cd1f0 2024-04-26T04:24:38.032 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.019+0000 7f3187fff640 1 --2- 172.21.15.63:0/1763815446 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f31881cd1f0 0x7f318806da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:38.032 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.019+0000 7f3187fff640 10 monclient: get_auth_request con 0x7f31881cd1f0 auth_method 0 2024-04-26T04:24:38.032 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.019+0000 7f3187fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:38.032 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.019+0000 7f3187fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:38.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.019+0000 7f3187fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:38.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.019+0000 7f3187fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:38.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.019+0000 7f3187fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:38.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.019+0000 7f3187fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:24:38.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.019+0000 7f3187fff640 10 monclient(hunting): handle_auth_done global_id 24245 payload 824 2024-04-26T04:24:38.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.019+0000 7f3187fff640 1 -- 172.21.15.63:0/1763815446 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3188068430 con 0x7f31881cd1f0 2024-04-26T04:24:38.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.023+0000 7f3187fff640 1 -- 172.21.15.63:0/1763815446 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f31881cd1f0 msgr2=0x7f318806da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:24:38.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.023+0000 7f3187fff640 1 --2- 172.21.15.63:0/1763815446 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f31881cd1f0 0x7f318806da20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3180016000 tx=0x7f3180004cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:38.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.023+0000 7f3184ff9640 1 -- 172.21.15.63:0/1763815446 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3180019070 con 0x7f31881cd1f0 2024-04-26T04:24:38.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.023+0000 7f3184ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:38.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.023+0000 7f3184ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:24:38.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.023+0000 7f3184ff9640 10 monclient: dump: 2024-04-26T04:24:38.033 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:38.033 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:38.034 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:38.034 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:38.034 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:38.034 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:38.034 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:38.034 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:38.034 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:38.034 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:38.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.023+0000 7f3184ff9640 1 -- 172.21.15.63:0/1763815446 <== mon.2 v2:172.21.15.63:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4721 (secure 0 0 0) 0x7f318001b620 con 0x7f31881cd1f0 2024-04-26T04:24:38.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.023+0000 7f3184ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T04:24:38.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.023+0000 7f3184ff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:24:38.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.023+0000 7f3184ff9640 1 -- 172.21.15.63:0/1763815446 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f31881cd1f0 msgr2=0x7f318806da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:38.034 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.023+0000 7f3184ff9640 1 --2- 172.21.15.63:0/1763815446 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f31881cd1f0 0x7f318806da20 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3180016000 tx=0x7f3180004cd0 comp rx=0 tx=0).stop 2024-04-26T04:24:38.035 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.023+0000 7f3184ff9640 1 -- 172.21.15.63:0/1763815446 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f31881cd1f0 msgr2=0x7f318806da20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:24:38.035 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.023+0000 7f3184ff9640 1 --2- 172.21.15.63:0/1763815446 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f31881cd1f0 0x7f318806da20 secure :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3180016000 tx=0x7f3180004cd0 comp rx=0 tx=0).stop 2024-04-26T04:24:38.035 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-26T04:24:38.035 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "c", 2024-04-26T04:24:38.035 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 2, 2024-04-26T04:24:38.035 INFO:teuthology.orchestra.run.smithi063.stdout: "state": "peon", 2024-04-26T04:24:38.035 INFO:teuthology.orchestra.run.smithi063.stdout: "election_epoch": 26, 2024-04-26T04:24:38.035 INFO:teuthology.orchestra.run.smithi063.stdout: "uptime": 125622, 2024-04-26T04:24:38.035 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum": [ 2024-04-26T04:24:38.035 INFO:teuthology.orchestra.run.smithi063.stdout: 0, 2024-04-26T04:24:38.036 INFO:teuthology.orchestra.run.smithi063.stdout: 1, 2024-04-26T04:24:38.036 INFO:teuthology.orchestra.run.smithi063.stdout: 2 2024-04-26T04:24:38.036 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:24:38.036 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_age": 1, 2024-04-26T04:24:38.036 INFO:teuthology.orchestra.run.smithi063.stdout: "features": { 2024-04-26T04:24:38.036 INFO:teuthology.orchestra.run.smithi063.stdout: "required_con": "2450521708007866372", 2024-04-26T04:24:38.036 INFO:teuthology.orchestra.run.smithi063.stdout: "required_mon": [ 2024-04-26T04:24:38.036 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:24:38.036 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:24:38.036 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:24:38.036 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:24:38.036 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:24:38.036 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:24:38.037 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:24:38.037 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:24:38.037 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:24:38.037 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:24:38.037 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:24:38.037 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:24:38.037 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_con": "4540701547738038271", 2024-04-26T04:24:38.037 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_mon": [ 2024-04-26T04:24:38.037 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:24:38.037 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:24:38.037 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:24:38.037 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:24:38.037 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:24:38.037 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:24:38.038 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:24:38.038 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:24:38.038 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:24:38.038 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:24:38.038 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:24:38.038 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:24:38.038 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:38.038 INFO:teuthology.orchestra.run.smithi063.stdout: "outside_quorum": [], 2024-04-26T04:24:38.038 INFO:teuthology.orchestra.run.smithi063.stdout: "extra_probe_peers": [], 2024-04-26T04:24:38.039 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_provider": [], 2024-04-26T04:24:38.039 INFO:teuthology.orchestra.run.smithi063.stdout: "monmap": { 2024-04-26T04:24:38.039 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 1, 2024-04-26T04:24:38.039 INFO:teuthology.orchestra.run.smithi063.stdout: "fsid": "cd1c6a0c-4e4b-4014-89c0-2e535479ef39", 2024-04-26T04:24:38.039 INFO:teuthology.orchestra.run.smithi063.stdout: "modified": "2024-04-26T04:18:03.702339Z", 2024-04-26T04:24:38.039 INFO:teuthology.orchestra.run.smithi063.stdout: "created": "2024-04-26T04:18:03.702339Z", 2024-04-26T04:24:38.039 INFO:teuthology.orchestra.run.smithi063.stdout: "min_mon_release": 19, 2024-04-26T04:24:38.039 INFO:teuthology.orchestra.run.smithi063.stdout: "min_mon_release_name": "squid", 2024-04-26T04:24:38.039 INFO:teuthology.orchestra.run.smithi063.stdout: "election_strategy": 1, 2024-04-26T04:24:38.039 INFO:teuthology.orchestra.run.smithi063.stdout: "disallowed_leaders: ": "", 2024-04-26T04:24:38.039 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": false, 2024-04-26T04:24:38.039 INFO:teuthology.orchestra.run.smithi063.stdout: "tiebreaker_mon": "", 2024-04-26T04:24:38.039 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_ranks: ": "", 2024-04-26T04:24:38.039 INFO:teuthology.orchestra.run.smithi063.stdout: "features": { 2024-04-26T04:24:38.040 INFO:teuthology.orchestra.run.smithi063.stdout: "persistent": [ 2024-04-26T04:24:38.040 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:24:38.040 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:24:38.040 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:24:38.040 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:24:38.040 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:24:38.040 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:24:38.040 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:24:38.040 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:24:38.040 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:24:38.040 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:24:38.040 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:24:38.040 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:24:38.040 INFO:teuthology.orchestra.run.smithi063.stdout: "optional": [] 2024-04-26T04:24:38.041 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:38.041 INFO:teuthology.orchestra.run.smithi063.stdout: "mons": [ 2024-04-26T04:24:38.041 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:38.041 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 0, 2024-04-26T04:24:38.041 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a", 2024-04-26T04:24:38.041 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:24:38.041 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:24:38.041 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:38.041 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:24:38.041 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:3300", 2024-04-26T04:24:38.041 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:24:38.041 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:38.041 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:38.041 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:24:38.042 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6789", 2024-04-26T04:24:38.042 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:24:38.042 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:24:38.042 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:24:38.042 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:38.042 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6789/0", 2024-04-26T04:24:38.042 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6789/0", 2024-04-26T04:24:38.042 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:24:38.042 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:24:38.042 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:24:38.042 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:38.042 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:38.042 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 1, 2024-04-26T04:24:38.043 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b", 2024-04-26T04:24:38.043 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:24:38.043 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:24:38.043 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:38.043 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:24:38.043 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:3300", 2024-04-26T04:24:38.043 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:24:38.043 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:38.043 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:38.043 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:24:38.043 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:6789", 2024-04-26T04:24:38.043 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:24:38.043 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:24:38.044 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:24:38.044 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:38.044 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:6789/0", 2024-04-26T04:24:38.044 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.186:6789/0", 2024-04-26T04:24:38.044 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:24:38.044 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:24:38.044 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:24:38.044 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:38.044 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:38.044 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 2, 2024-04-26T04:24:38.044 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "c", 2024-04-26T04:24:38.044 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:24:38.044 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:24:38.044 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:38.044 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:24:38.045 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:3301", 2024-04-26T04:24:38.045 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:24:38.045 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:38.045 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:38.045 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:24:38.045 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6790", 2024-04-26T04:24:38.045 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:24:38.045 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:24:38.045 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:24:38.045 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:38.045 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6790/0", 2024-04-26T04:24:38.045 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6790/0", 2024-04-26T04:24:38.046 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:24:38.046 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:24:38.046 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:24:38.046 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:24:38.046 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:24:38.046 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:38.046 INFO:teuthology.orchestra.run.smithi063.stdout: "feature_map": { 2024-04-26T04:24:38.046 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": [ 2024-04-26T04:24:38.046 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:38.046 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:24:38.046 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:24:38.046 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 1 2024-04-26T04:24:38.046 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:24:38.046 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:24:38.047 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": [ 2024-04-26T04:24:38.048 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:38.048 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:24:38.048 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:24:38.048 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 3 2024-04-26T04:24:38.048 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:24:38.048 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:24:38.048 INFO:teuthology.orchestra.run.smithi063.stdout: "client": [ 2024-04-26T04:24:38.048 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:38.048 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:24:38.048 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:24:38.048 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 3 2024-04-26T04:24:38.048 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:24:38.048 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:24:38.049 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:38.049 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": false 2024-04-26T04:24:38.049 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-26T04:24:38.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.023+0000 7f318e5ae640 1 -- 172.21.15.63:0/1763815446 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f3174035ef0 msgr2=0x7f31740383b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:38.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.023+0000 7f318e5ae640 1 --2- 172.21.15.63:0/1763815446 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f3174035ef0 0x7f31740383b0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f318812a7d0 tx=0x7f317c009040 comp rx=0 tx=0).stop 2024-04-26T04:24:38.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.023+0000 7f318e5ae640 10 monclient: shutdown 2024-04-26T04:24:38.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.023+0000 7f318e5ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:24:38.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.023+0000 7f318e5ae640 1 -- 172.21.15.63:0/1763815446 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3188139e20 msgr2=0x7f31881359c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:38.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.023+0000 7f318e5ae640 1 --2- 172.21.15.63:0/1763815446 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3188139e20 0x7f31881359c0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f3178018ce0 tx=0x7f3178018e70 comp rx=0 tx=0).stop 2024-04-26T04:24:38.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.023+0000 7f318cb4b640 1 -- 172.21.15.63:0/1763815446 reap_dead start 2024-04-26T04:24:38.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.023+0000 7f318cb4b640 1 -- 172.21.15.63:0/1763815446 reap_dead start 2024-04-26T04:24:38.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.023+0000 7f318e5ae640 1 -- 172.21.15.63:0/1763815446 shutdown_connections 2024-04-26T04:24:38.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.023+0000 7f318e5ae640 1 -- 172.21.15.63:0/1763815446 >> 172.21.15.63:0/1763815446 conn(0x7f31880a1e70 msgr2=0x7f3188099fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:24:38.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.023+0000 7f318e5ae640 1 -- 172.21.15.63:0/1763815446 shutdown_connections 2024-04-26T04:24:38.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.023+0000 7f318e5ae640 1 -- 172.21.15.63:0/1763815446 wait complete. 2024-04-26T04:24:38.055 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T04:24:38.055 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-26T04:24:38.055 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T04:24:38.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.167+0000 7fa52d3bd640 10 monclient: get_monmap_and_config 2024-04-26T04:24:38.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.167+0000 7fa52d3bd640 10 monclient: build_initial_monmap 2024-04-26T04:24:38.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.167+0000 7fa52d3bd640 10 monclient: monmap: 2024-04-26T04:24:38.170 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:24:38.170 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:38.170 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:24:38.170316+0000 2024-04-26T04:24:38.170 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:24:38.170316+0000 2024-04-26T04:24:38.170 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:24:38.171 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:38.171 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:24:38.171 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:24:38.171 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:24:38.171 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:24:38.171 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:24:38.171 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:38.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.167+0000 7fa52d3bd640 1 Processor -- start 2024-04-26T04:24:38.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.167+0000 7fa52d3bd640 1 -- start start 2024-04-26T04:24:38.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.167+0000 7fa52d3bd640 10 monclient: init 2024-04-26T04:24:38.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.167+0000 7fa52d3bd640 10 monclient: _reopen_session rank -1 2024-04-26T04:24:38.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.167+0000 7fa52d3bd640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-04-26T04:24:38.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.167+0000 7fa52d3bd640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa5280a2c60 0x7fa5280a3040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:38.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.167+0000 7fa52d3bd640 10 monclient(hunting): picked mon.noname-c con 0x7fa5280a2c60 addr v2:172.21.15.186:3300/0 2024-04-26T04:24:38.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.167+0000 7fa52d3bd640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7fa5280a20c0 0x7fa5280a3db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:38.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.167+0000 7fa52d3bd640 10 monclient(hunting): picked mon.noname-a con 0x7fa5280a20c0 addr v2:172.21.15.63:3300/0 2024-04-26T04:24:38.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.167+0000 7fa52d3bd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa5280a24a0 addr v1:172.21.15.186:6789/0 2024-04-26T04:24:38.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.167+0000 7fa52d3bd640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5280a5010 con 0x7fa5280a24a0 2024-04-26T04:24:38.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.167+0000 7fa52d3bd640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:38.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.167+0000 7fa52d3bd640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7fa5280a0df0 con 0x7fa5280a20c0 2024-04-26T04:24:38.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.167+0000 7fa52d3bd640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:38.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.167+0000 7fa52d3bd640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fa5280a13b0 con 0x7fa5280a2c60 2024-04-26T04:24:38.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.167+0000 7fa52d3bd640 10 monclient(hunting): _renew_subs 2024-04-26T04:24:38.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.167+0000 7fa52d3bd640 10 monclient(hunting): authenticate will time out at 1309.878794s 2024-04-26T04:24:38.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.167+0000 7fa51e7fc640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7fa5280a20c0 0x7fa5280a3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:38.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.167+0000 7fa51e7fc640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7fa5280a20c0 0x7fa5280a3db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:33270/0 (socket says 172.21.15.63:33270) 2024-04-26T04:24:38.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.167+0000 7fa51e7fc640 1 -- 172.21.15.63:0/2571780561 learned_addr learned my addr 172.21.15.63:0/2571780561 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:24:38.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.167+0000 7fa51e7fc640 10 monclient(hunting): get_auth_request con 0x7fa5280a20c0 auth_method 0 2024-04-26T04:24:38.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.167+0000 7fa51e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:38.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.167+0000 7fa51e7fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:38.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.167+0000 7fa51e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:38.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.167+0000 7fa51e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:38.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.167+0000 7fa51e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:38.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.167+0000 7fa51e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:24:38.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.167+0000 7fa51e7fc640 10 monclient(hunting): handle_auth_done global_id 5349 payload 274 2024-04-26T04:24:38.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.167+0000 7fa51e7fc640 1 -- 172.21.15.63:0/2571780561 >> v2:172.21.15.186:3300/0 conn(0x7fa5280a2c60 msgr2=0x7fa5280a3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:24:38.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.167+0000 7fa51e7fc640 1 --2- 172.21.15.63:0/2571780561 >> v2:172.21.15.186:3300/0 conn(0x7fa5280a2c60 0x7fa5280a3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:38.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.167+0000 7fa51e7fc640 1 -- 172.21.15.63:0/2571780561 >> v1:172.21.15.186:6789/0 conn(0x7fa5280a24a0 legacy=0x7fa5280a42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:24:38.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.167+0000 7fa51e7fc640 10 monclient: _finish_hunting 0 2024-04-26T04:24:38.176 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.167+0000 7fa51e7fc640 1 monclient: found mon.noname-a 2024-04-26T04:24:38.176 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.167+0000 7fa51e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:24:38.176 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.167+0000 7fa51e7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:24:38.176 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.167+0000 7fa51e7fc640 1 -- 172.21.15.63:0/2571780561 --> v2:172.21.15.63:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5280a10d0 con 0x7fa5280a20c0 2024-04-26T04:24:38.178 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.171+0000 7fa51e7fc640 1 -- 172.21.15.63:0/2571780561 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa5280a20c0 msgr2=0x7fa5280a3db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:24:38.178 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.171+0000 7fa51e7fc640 1 --2- 172.21.15.63:0/2571780561 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa5280a20c0 0x7fa5280a3db0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fa5280a54a0 tx=0x7fa51001c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9ddccc199e7f675 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:38.178 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.171+0000 7fa5267fc640 1 -- 172.21.15.63:0/2571780561 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5100b1070 con 0x7fa5280a20c0 2024-04-26T04:24:38.178 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.171+0000 7fa5267fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:38.178 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.171+0000 7fa5267fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:24:38.178 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.171+0000 7fa5267fc640 10 monclient: dump: 2024-04-26T04:24:38.178 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:38.178 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:38.178 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:38.178 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:38.179 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:38.179 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:38.179 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:38.179 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:38.179 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:38.179 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:38.179 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.171+0000 7fa5267fc640 10 monclient: _finish_auth 0 2024-04-26T04:24:38.179 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.171+0000 7fa5267fc640 10 monclient: _check_auth_tickets 2024-04-26T04:24:38.179 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.171+0000 7fa5267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:24:38.179 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.171+0000 7fa5267fc640 1 -- 172.21.15.63:0/2571780561 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5100ac620 con 0x7fa5280a20c0 2024-04-26T04:24:38.179 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.171+0000 7fa5267fc640 10 monclient: handle_config config(0 keys) 2024-04-26T04:24:38.179 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.171+0000 7fa5267fc640 1 -- 172.21.15.63:0/2571780561 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5100ac9f0 con 0x7fa5280a20c0 2024-04-26T04:24:38.179 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.171+0000 7fa52d3bd640 5 monclient: authenticate success, global_id 5349 2024-04-26T04:24:38.179 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.171+0000 7fa52d3bd640 10 monclient: get_monmap_and_config success 2024-04-26T04:24:38.180 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.171+0000 7fa52d3bd640 4 set_mon_vals no callback set 2024-04-26T04:24:38.180 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.171+0000 7fa5267fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:38.180 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.171+0000 7fa5267fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:24:38.180 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.171+0000 7fa5267fc640 10 monclient: dump: 2024-04-26T04:24:38.180 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:38.180 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:38.180 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:38.180 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:38.180 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:38.180 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:38.180 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:38.180 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:38.180 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:38.180 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:38.181 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.175+0000 7fa52d3bd640 10 monclient: shutdown 2024-04-26T04:24:38.181 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.175+0000 7fa52d3bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:24:38.181 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.175+0000 7fa52d3bd640 1 -- 172.21.15.63:0/2571780561 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa5280a20c0 msgr2=0x7fa5280a3db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:38.181 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.175+0000 7fa52d3bd640 1 --2- 172.21.15.63:0/2571780561 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa5280a20c0 0x7fa5280a3db0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fa5280a54a0 tx=0x7fa51001c3a0 comp rx=0 tx=0).stop 2024-04-26T04:24:38.181 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.175+0000 7fa52d3bd640 1 -- 172.21.15.63:0/2571780561 shutdown_connections 2024-04-26T04:24:38.181 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.175+0000 7fa52d3bd640 1 --2- 172.21.15.63:0/2571780561 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa5280a20c0 0x7fa5280a3db0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:38.181 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.175+0000 7fa52d3bd640 1 --2- 172.21.15.63:0/2571780561 >> v2:172.21.15.186:3300/0 conn(0x7fa5280a2c60 0x7fa5280a3040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:38.181 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.175+0000 7fa52d3bd640 1 -- 172.21.15.63:0/2571780561 >> 172.21.15.63:0/2571780561 conn(0x7fa5280a5fb0 msgr2=0x7fa5280a63c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:24:38.181 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.175+0000 7fa52d3bd640 1 -- 172.21.15.63:0/2571780561 shutdown_connections 2024-04-26T04:24:38.181 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.175+0000 7fa52d3bd640 1 -- 172.21.15.63:0/2571780561 wait complete. 2024-04-26T04:24:38.188 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.175+0000 7fa52d3bd640 10 monclient: build_initial_monmap 2024-04-26T04:24:38.188 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.175+0000 7fa52d3bd640 10 monclient: monmap: 2024-04-26T04:24:38.188 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:24:38.188 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:24:38.188 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:24:38.188 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:24:38.188 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:24:38.188 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:38.188 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:24:38.188 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:24:38.189 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:24:38.189 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:38.189 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.175+0000 7fa52d3bd640 1 Processor -- start 2024-04-26T04:24:38.189 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.175+0000 7fa52d3bd640 1 -- start start 2024-04-26T04:24:38.189 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.175+0000 7fa52d3bd640 10 monclient: init 2024-04-26T04:24:38.189 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.175+0000 7fa52d3bd640 10 monclient: _reopen_session rank -1 2024-04-26T04:24:38.189 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.175+0000 7fa52d3bd640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T04:24:38.189 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.175+0000 7fa52d3bd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5280a20c0 0x7fa528127780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:38.189 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.175+0000 7fa52d3bd640 10 monclient(hunting): picked mon.noname-b con 0x7fa5280a20c0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:24:38.189 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.175+0000 7fa52d3bd640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fa5280a24a0 0x7fa528125240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:38.189 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.175+0000 7fa52d3bd640 10 monclient(hunting): picked mon.noname-c con 0x7fa5280a24a0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:24:38.189 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.175+0000 7fa52d3bd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa5280a2c60 0x7fa52813e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:38.189 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.175+0000 7fa52d3bd640 10 monclient(hunting): picked mon.noname-a con 0x7fa5280a2c60 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:24:38.189 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.175+0000 7fa52d3bd640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:38.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.175+0000 7fa52d3bd640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fa528141b60 con 0x7fa5280a2c60 2024-04-26T04:24:38.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.175+0000 7fa52d3bd640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:38.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.175+0000 7fa52d3bd640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa5281419e0 con 0x7fa5280a20c0 2024-04-26T04:24:38.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.175+0000 7fa52d3bd640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:38.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.175+0000 7fa52d3bd640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7fa528141ce0 con 0x7fa5280a24a0 2024-04-26T04:24:38.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.175+0000 7fa52d3bd640 10 monclient(hunting): _renew_subs 2024-04-26T04:24:38.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.175+0000 7fa52d3bd640 10 monclient(hunting): authenticate will time out at 1309.886364s 2024-04-26T04:24:38.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.175+0000 7fa51e7fc640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fa5280a24a0 0x7fa528125240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:38.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.175+0000 7fa51e7fc640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fa5280a24a0 0x7fa528125240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:40142/0 (socket says 172.21.15.63:40142) 2024-04-26T04:24:38.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.175+0000 7fa51e7fc640 1 -- 172.21.15.63:0/306042928 learned_addr learned my addr 172.21.15.63:0/306042928 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:24:38.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.175+0000 7fa51e7fc640 10 monclient(hunting): get_auth_request con 0x7fa5280a24a0 auth_method 0 2024-04-26T04:24:38.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.175+0000 7fa51e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:38.190 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.175+0000 7fa51e7fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:38.191 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.175+0000 7fa51e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:38.191 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.175+0000 7fa5277fe640 1 --2- 172.21.15.63:0/306042928 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa5280a2c60 0x7fa52813e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:38.191 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.175+0000 7fa5277fe640 10 monclient(hunting): get_auth_request con 0x7fa5280a2c60 auth_method 0 2024-04-26T04:24:38.191 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.175+0000 7fa5277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:38.191 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.175+0000 7fa5277fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:38.191 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.175+0000 7fa5277fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:38.191 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.175+0000 7fa51e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:38.191 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.175+0000 7fa51e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:38.191 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.175+0000 7fa51e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:24:38.191 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.175+0000 7fa5277fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:38.191 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.175+0000 7fa5277fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:38.191 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.175+0000 7fa5277fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:24:38.191 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.175+0000 7fa526ffd640 1 --2- 172.21.15.63:0/306042928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5280a20c0 0x7fa528127780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:38.191 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.175+0000 7fa526ffd640 10 monclient(hunting): get_auth_request con 0x7fa5280a20c0 auth_method 0 2024-04-26T04:24:38.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.175+0000 7fa526ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:38.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.175+0000 7fa526ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:38.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.175+0000 7fa526ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:38.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.175+0000 7fa51e7fc640 10 monclient(hunting): handle_auth_done global_id 24248 payload 824 2024-04-26T04:24:38.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.175+0000 7fa51e7fc640 1 -- 172.21.15.63:0/306042928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5280a20c0 msgr2=0x7fa528127780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:38.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.175+0000 7fa51e7fc640 1 --2- 172.21.15.63:0/306042928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5280a20c0 0x7fa528127780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:38.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.175+0000 7fa51e7fc640 1 -- 172.21.15.63:0/306042928 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa5280a2c60 msgr2=0x7fa52813e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:38.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.175+0000 7fa51e7fc640 1 --2- 172.21.15.63:0/306042928 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa5280a2c60 0x7fa52813e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:38.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.175+0000 7fa51e7fc640 10 monclient: _finish_hunting 0 2024-04-26T04:24:38.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.175+0000 7fa51e7fc640 1 monclient: found mon.noname-c 2024-04-26T04:24:38.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.175+0000 7fa51e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:24:38.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.175+0000 7fa51e7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.63:3301/0 2024-04-26T04:24:38.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.175+0000 7fa51e7fc640 1 -- 172.21.15.63:0/306042928 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa528128cf0 con 0x7fa5280a24a0 2024-04-26T04:24:38.193 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.179+0000 7fa51e7fc640 1 -- 172.21.15.63:0/306042928 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fa5280a24a0 msgr2=0x7fa528125240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:24:38.193 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.179+0000 7fa51e7fc640 1 --2- 172.21.15.63:0/306042928 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fa5280a24a0 0x7fa528125240 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa510019fd0 tx=0x7fa51001abe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:38.193 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.179+0000 7fa51ffff640 1 -- 172.21.15.63:0/306042928 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5100bb020 con 0x7fa5280a24a0 2024-04-26T04:24:38.193 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.179+0000 7fa51ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:38.193 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.179+0000 7fa51ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:24:38.193 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.179+0000 7fa51ffff640 10 monclient: dump: 2024-04-26T04:24:38.193 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:38.193 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:38.193 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:38.193 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:38.193 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:38.193 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:38.193 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:38.193 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:38.194 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:38.194 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:38.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.179+0000 7fa51ffff640 10 monclient: _finish_auth 0 2024-04-26T04:24:38.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.179+0000 7fa51ffff640 10 monclient: _check_auth_tickets 2024-04-26T04:24:38.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.179+0000 7fa51ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:24:38.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.179+0000 7fa51ffff640 1 -- 172.21.15.63:0/306042928 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5100b7420 con 0x7fa5280a24a0 2024-04-26T04:24:38.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.179+0000 7fa51ffff640 10 monclient: handle_config config(0 keys) 2024-04-26T04:24:38.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.179+0000 7fa51ffff640 1 -- 172.21.15.63:0/306042928 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5100b1040 con 0x7fa5280a24a0 2024-04-26T04:24:38.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.179+0000 7fa51ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:38.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.179+0000 7fa51ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:24:38.194 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.179+0000 7fa51ffff640 10 monclient: dump: 2024-04-26T04:24:38.194 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:38.195 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:38.195 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:38.195 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:38.195 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:38.195 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:38.195 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:38.195 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:38.195 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:38.195 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:38.195 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.179+0000 7fa52d3bd640 5 monclient: authenticate success, global_id 24248 2024-04-26T04:24:38.195 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.179+0000 7fa525ffb640 4 set_mon_vals no callback set 2024-04-26T04:24:38.195 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.179+0000 7fa52d3bd640 10 monclient: _renew_subs 2024-04-26T04:24:38.195 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.179+0000 7fa52d3bd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:24:38.196 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.179+0000 7fa52d3bd640 1 -- 172.21.15.63:0/306042928 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5280728b0 con 0x7fa5280a24a0 2024-04-26T04:24:38.196 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.179+0000 7fa52d3bd640 10 monclient: _renew_subs 2024-04-26T04:24:38.196 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.179+0000 7fa52d3bd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:24:38.196 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.179+0000 7fa52d3bd640 1 -- 172.21.15.63:0/306042928 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa528128450 con 0x7fa5280a24a0 2024-04-26T04:24:38.196 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.179+0000 7fa52d3bd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:24:38.196 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.179+0000 7fa52d3bd640 1 --2- 172.21.15.63:0/306042928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa52813e8a0 0x7fa5281cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:38.196 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.179+0000 7fa52d3bd640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:38.196 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.179+0000 7fa52d3bd640 1 -- 172.21.15.63:0/306042928 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa5281288e0 con 0x7fa52813e8a0 2024-04-26T04:24:38.196 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.179+0000 7fa51ffff640 1 -- 172.21.15.63:0/306042928 <== mon.2 v2:172.21.15.63:3301/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fa5100be020 con 0x7fa5280a24a0 2024-04-26T04:24:38.196 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.179+0000 7fa526ffd640 1 --2- 172.21.15.63:0/306042928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa52813e8a0 0x7fa5281cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:38.196 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.179+0000 7fa526ffd640 10 monclient: get_auth_request con 0x7fa52813e8a0 auth_method 0 2024-04-26T04:24:38.196 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.179+0000 7fa526ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:38.196 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.179+0000 7fa526ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:38.196 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.179+0000 7fa526ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:38.197 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.179+0000 7fa51ffff640 1 --2- 172.21.15.63:0/306042928 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fa508035ec0 0x7fa508038380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:38.197 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.179+0000 7fa51ffff640 1 -- 172.21.15.63:0/306042928 <== mon.2 v2:172.21.15.63:3301/0 5 ==== osd_map(220..220 src has 1..220) ==== 4346+0+0 (secure 0 0 0) 0x7fa5100e2b40 con 0x7fa5280a24a0 2024-04-26T04:24:38.197 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.179+0000 7fa526ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:38.197 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.179+0000 7fa526ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:38.197 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.179+0000 7fa526ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:24:38.197 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.179+0000 7fa5277fe640 1 --2- 172.21.15.63:0/306042928 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fa508035ec0 0x7fa508038380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:38.197 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.179+0000 7fa5277fe640 10 monclient: get_auth_request con 0x7fa508035ec0 auth_method 0 2024-04-26T04:24:38.197 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.179+0000 7fa526ffd640 10 monclient(hunting): handle_auth_done global_id 34114 payload 824 2024-04-26T04:24:38.197 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.179+0000 7fa526ffd640 1 -- 172.21.15.63:0/306042928 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa528062e10 con 0x7fa52813e8a0 2024-04-26T04:24:38.197 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.183+0000 7fa526ffd640 1 -- 172.21.15.63:0/306042928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa52813e8a0 msgr2=0x7fa5281cf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:24:38.197 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.183+0000 7fa526ffd640 1 --2- 172.21.15.63:0/306042928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa52813e8a0 0x7fa5281cf1b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa528126600 tx=0x7fa51800db90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:38.197 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.183+0000 7fa5277fe640 1 --2- 172.21.15.63:0/306042928 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fa508035ec0 0x7fa508038380 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fa520001ea0 tx=0x7fa520009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:38.197 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.183+0000 7fa51ffff640 1 -- 172.21.15.63:0/306042928 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa518018030 con 0x7fa52813e8a0 2024-04-26T04:24:38.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.183+0000 7fa51ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:38.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.183+0000 7fa51ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T04:24:38.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.183+0000 7fa51ffff640 10 monclient: dump: 2024-04-26T04:24:38.198 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:38.198 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:38.198 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:38.198 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:38.198 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:38.198 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:38.198 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:38.198 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:38.198 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:38.198 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:38.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.183+0000 7fa51ffff640 1 -- 172.21.15.63:0/306042928 <== mon.1 v2:172.21.15.186:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa518009920 con 0x7fa52813e8a0 2024-04-26T04:24:38.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.183+0000 7fa51ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:24:38.199 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.183+0000 7fa51ffff640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:24:38.199 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.183+0000 7fa51ffff640 1 -- 172.21.15.63:0/306042928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa52813e8a0 msgr2=0x7fa5281cf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:38.199 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.183+0000 7fa51ffff640 1 --2- 172.21.15.63:0/306042928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa52813e8a0 0x7fa5281cf1b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa528126600 tx=0x7fa51800db90 comp rx=0 tx=0).stop 2024-04-26T04:24:38.199 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.183+0000 7fa51ffff640 1 -- 172.21.15.63:0/306042928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa52813e8a0 msgr2=0x7fa5281cf1b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:24:38.199 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.183+0000 7fa51ffff640 1 --2- 172.21.15.63:0/306042928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa52813e8a0 0x7fa5281cf1b0 secure :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa528126600 tx=0x7fa51800db90 comp rx=0 tx=0).stop 2024-04-26T04:24:38.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.191+0000 7fa52d3bd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T04:24:38.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.191+0000 7fa52d3bd640 1 --2- 172.21.15.63:0/306042928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5281cf6f0 0x7fa5281cfb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:38.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.191+0000 7fa52d3bd640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:38.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.191+0000 7fa52d3bd640 1 -- 172.21.15.63:0/306042928 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa5281d0040 con 0x7fa5281cf6f0 2024-04-26T04:24:38.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.191+0000 7fa526ffd640 1 --2- 172.21.15.63:0/306042928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5281cf6f0 0x7fa5281cfb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:38.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.191+0000 7fa526ffd640 10 monclient: get_auth_request con 0x7fa5281cf6f0 auth_method 0 2024-04-26T04:24:38.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.191+0000 7fa526ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:38.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.191+0000 7fa526ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:38.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.191+0000 7fa526ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:38.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.191+0000 7fa526ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:38.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.191+0000 7fa526ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:38.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.191+0000 7fa526ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:24:38.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.191+0000 7fa526ffd640 10 monclient(hunting): handle_auth_done global_id 34117 payload 824 2024-04-26T04:24:38.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.191+0000 7fa526ffd640 1 -- 172.21.15.63:0/306042928 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa5280a8280 con 0x7fa5281cf6f0 2024-04-26T04:24:38.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.191+0000 7fa526ffd640 1 -- 172.21.15.63:0/306042928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5281cf6f0 msgr2=0x7fa5281cfb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:24:38.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.191+0000 7fa526ffd640 1 --2- 172.21.15.63:0/306042928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5281cf6f0 0x7fa5281cfb00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa528126600 tx=0x7fa5180047e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:38.202 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.195+0000 7fa51ffff640 1 -- 172.21.15.63:0/306042928 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa518028420 con 0x7fa5281cf6f0 2024-04-26T04:24:38.202 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.195+0000 7fa51ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:38.202 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.195+0000 7fa51ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T04:24:38.202 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.195+0000 7fa51ffff640 10 monclient: dump: 2024-04-26T04:24:38.202 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:38.202 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:38.202 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:38.202 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:38.202 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:38.202 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:38.202 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:38.202 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:38.202 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:38.202 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:38.203 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.195+0000 7fa51ffff640 1 -- 172.21.15.63:0/306042928 <== mon.1 v2:172.21.15.186:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4720 (secure 0 0 0) 0x7fa518028420 con 0x7fa5281cf6f0 2024-04-26T04:24:38.203 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.195+0000 7fa51ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T04:24:38.203 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.195+0000 7fa51ffff640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:24:38.203 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.195+0000 7fa51ffff640 1 -- 172.21.15.63:0/306042928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5281cf6f0 msgr2=0x7fa5281cfb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:38.203 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.195+0000 7fa51ffff640 1 --2- 172.21.15.63:0/306042928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5281cf6f0 0x7fa5281cfb00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa528126600 tx=0x7fa5180047e0 comp rx=0 tx=0).stop 2024-04-26T04:24:38.203 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.195+0000 7fa51ffff640 1 -- 172.21.15.63:0/306042928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5281cf6f0 msgr2=0x7fa5281cfb00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:24:38.203 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.195+0000 7fa51ffff640 1 --2- 172.21.15.63:0/306042928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5281cf6f0 0x7fa5281cfb00 secure :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa528126600 tx=0x7fa5180047e0 comp rx=0 tx=0).stop 2024-04-26T04:24:38.203 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-26T04:24:38.203 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b", 2024-04-26T04:24:38.203 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 1, 2024-04-26T04:24:38.203 INFO:teuthology.orchestra.run.smithi063.stdout: "state": "peon", 2024-04-26T04:24:38.204 INFO:teuthology.orchestra.run.smithi063.stdout: "election_epoch": 26, 2024-04-26T04:24:38.204 INFO:teuthology.orchestra.run.smithi063.stdout: "uptime": 10975, 2024-04-26T04:24:38.204 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum": [ 2024-04-26T04:24:38.204 INFO:teuthology.orchestra.run.smithi063.stdout: 0, 2024-04-26T04:24:38.204 INFO:teuthology.orchestra.run.smithi063.stdout: 1, 2024-04-26T04:24:38.204 INFO:teuthology.orchestra.run.smithi063.stdout: 2 2024-04-26T04:24:38.204 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:24:38.204 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_age": 1, 2024-04-26T04:24:38.204 INFO:teuthology.orchestra.run.smithi063.stdout: "features": { 2024-04-26T04:24:38.204 INFO:teuthology.orchestra.run.smithi063.stdout: "required_con": "2450521708007866372", 2024-04-26T04:24:38.204 INFO:teuthology.orchestra.run.smithi063.stdout: "required_mon": [ 2024-04-26T04:24:38.204 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:24:38.204 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:24:38.204 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:24:38.205 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:24:38.205 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:24:38.205 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:24:38.205 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:24:38.205 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:24:38.205 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:24:38.205 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:24:38.205 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:24:38.205 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:24:38.205 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_con": "4540701547738038271", 2024-04-26T04:24:38.205 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_mon": [ 2024-04-26T04:24:38.205 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:24:38.205 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:24:38.205 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:24:38.206 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:24:38.206 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:24:38.206 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:24:38.206 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:24:38.206 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:24:38.206 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:24:38.206 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:24:38.206 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:24:38.206 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:24:38.206 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:38.206 INFO:teuthology.orchestra.run.smithi063.stdout: "outside_quorum": [], 2024-04-26T04:24:38.206 INFO:teuthology.orchestra.run.smithi063.stdout: "extra_probe_peers": [], 2024-04-26T04:24:38.206 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_provider": [], 2024-04-26T04:24:38.207 INFO:teuthology.orchestra.run.smithi063.stdout: "monmap": { 2024-04-26T04:24:38.207 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 1, 2024-04-26T04:24:38.207 INFO:teuthology.orchestra.run.smithi063.stdout: "fsid": "cd1c6a0c-4e4b-4014-89c0-2e535479ef39", 2024-04-26T04:24:38.207 INFO:teuthology.orchestra.run.smithi063.stdout: "modified": "2024-04-26T04:18:03.702339Z", 2024-04-26T04:24:38.207 INFO:teuthology.orchestra.run.smithi063.stdout: "created": "2024-04-26T04:18:03.702339Z", 2024-04-26T04:24:38.207 INFO:teuthology.orchestra.run.smithi063.stdout: "min_mon_release": 19, 2024-04-26T04:24:38.207 INFO:teuthology.orchestra.run.smithi063.stdout: "min_mon_release_name": "squid", 2024-04-26T04:24:38.207 INFO:teuthology.orchestra.run.smithi063.stdout: "election_strategy": 1, 2024-04-26T04:24:38.207 INFO:teuthology.orchestra.run.smithi063.stdout: "disallowed_leaders: ": "", 2024-04-26T04:24:38.207 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": false, 2024-04-26T04:24:38.207 INFO:teuthology.orchestra.run.smithi063.stdout: "tiebreaker_mon": "", 2024-04-26T04:24:38.207 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_ranks: ": "", 2024-04-26T04:24:38.207 INFO:teuthology.orchestra.run.smithi063.stdout: "features": { 2024-04-26T04:24:38.208 INFO:teuthology.orchestra.run.smithi063.stdout: "persistent": [ 2024-04-26T04:24:38.208 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:24:38.208 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:24:38.208 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:24:38.208 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:24:38.208 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:24:38.208 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:24:38.208 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:24:38.208 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:24:38.208 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:24:38.208 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:24:38.208 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:24:38.208 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:24:38.208 INFO:teuthology.orchestra.run.smithi063.stdout: "optional": [] 2024-04-26T04:24:38.208 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:38.209 INFO:teuthology.orchestra.run.smithi063.stdout: "mons": [ 2024-04-26T04:24:38.209 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:38.209 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 0, 2024-04-26T04:24:38.209 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a", 2024-04-26T04:24:38.209 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:24:38.209 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:24:38.209 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:38.209 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:24:38.209 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:3300", 2024-04-26T04:24:38.209 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:24:38.209 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:38.209 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:38.209 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:24:38.209 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6789", 2024-04-26T04:24:38.209 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:24:38.210 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:24:38.210 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:24:38.210 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:38.210 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6789/0", 2024-04-26T04:24:38.210 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6789/0", 2024-04-26T04:24:38.210 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:24:38.210 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:24:38.210 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:24:38.210 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:38.210 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:38.210 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 1, 2024-04-26T04:24:38.210 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b", 2024-04-26T04:24:38.210 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:24:38.210 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:24:38.211 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:38.211 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:24:38.211 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:3300", 2024-04-26T04:24:38.211 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:24:38.211 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:38.211 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:38.211 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:24:38.211 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:6789", 2024-04-26T04:24:38.211 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:24:38.211 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:24:38.211 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:24:38.211 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:38.211 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:6789/0", 2024-04-26T04:24:38.212 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.186:6789/0", 2024-04-26T04:24:38.212 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:24:38.212 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:24:38.212 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:24:38.212 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:38.212 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:38.213 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 2, 2024-04-26T04:24:38.213 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "c", 2024-04-26T04:24:38.213 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:24:38.213 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:24:38.213 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:38.213 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:24:38.213 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:3301", 2024-04-26T04:24:38.213 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:24:38.213 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:38.213 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:38.213 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:24:38.213 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6790", 2024-04-26T04:24:38.214 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:24:38.214 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:24:38.214 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:24:38.214 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:38.214 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6790/0", 2024-04-26T04:24:38.214 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6790/0", 2024-04-26T04:24:38.214 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:24:38.214 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:24:38.214 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:24:38.214 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:24:38.214 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:24:38.215 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:38.215 INFO:teuthology.orchestra.run.smithi063.stdout: "feature_map": { 2024-04-26T04:24:38.215 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": [ 2024-04-26T04:24:38.215 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:38.215 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:24:38.215 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:24:38.215 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 1 2024-04-26T04:24:38.215 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:24:38.215 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:24:38.215 INFO:teuthology.orchestra.run.smithi063.stdout: "client": [ 2024-04-26T04:24:38.215 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:38.215 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:24:38.215 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:24:38.216 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 2 2024-04-26T04:24:38.216 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:24:38.216 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:24:38.216 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": [ 2024-04-26T04:24:38.216 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:38.216 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:24:38.216 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:24:38.217 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 1 2024-04-26T04:24:38.217 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:24:38.217 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:24:38.217 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:38.217 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": false 2024-04-26T04:24:38.217 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-26T04:24:38.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.195+0000 7fa52d3bd640 1 -- 172.21.15.63:0/306042928 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fa508035ec0 msgr2=0x7fa508038380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:38.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.195+0000 7fa52d3bd640 1 --2- 172.21.15.63:0/306042928 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fa508035ec0 0x7fa508038380 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fa520001ea0 tx=0x7fa520009040 comp rx=0 tx=0).stop 2024-04-26T04:24:38.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.195+0000 7fa52d3bd640 10 monclient: shutdown 2024-04-26T04:24:38.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.195+0000 7fa52d3bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:24:38.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.195+0000 7fa52d3bd640 1 -- 172.21.15.63:0/306042928 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fa5280a24a0 msgr2=0x7fa528125240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:38.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.195+0000 7fa52d3bd640 1 --2- 172.21.15.63:0/306042928 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fa5280a24a0 0x7fa528125240 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa510019fd0 tx=0x7fa51001abe0 comp rx=0 tx=0).stop 2024-04-26T04:24:38.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.195+0000 7fa5277fe640 1 -- 172.21.15.63:0/306042928 reap_dead start 2024-04-26T04:24:38.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.195+0000 7fa5277fe640 1 -- 172.21.15.63:0/306042928 reap_dead start 2024-04-26T04:24:38.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.195+0000 7fa52d3bd640 1 -- 172.21.15.63:0/306042928 shutdown_connections 2024-04-26T04:24:38.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.195+0000 7fa52d3bd640 1 -- 172.21.15.63:0/306042928 >> 172.21.15.63:0/306042928 conn(0x7fa5280a5fb0 msgr2=0x7fa5280a73b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:24:38.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.195+0000 7fa52d3bd640 1 -- 172.21.15.63:0/306042928 shutdown_connections 2024-04-26T04:24:38.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.195+0000 7fa52d3bd640 1 -- 172.21.15.63:0/306042928 wait complete. 2024-04-26T04:24:38.221 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T04:24:38.221 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-04-26T04:24:38.221 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-04-26T04:24:38.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.327+0000 7fa47a914640 10 monclient: get_monmap_and_config 2024-04-26T04:24:38.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.331+0000 7fa47a914640 10 monclient: build_initial_monmap 2024-04-26T04:24:38.334 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.331+0000 7fa47a914640 10 monclient: monmap: 2024-04-26T04:24:38.334 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:24:38.334 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:38.334 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:24:38.333919+0000 2024-04-26T04:24:38.334 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:24:38.333919+0000 2024-04-26T04:24:38.334 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:24:38.334 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:38.334 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:24:38.334 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:24:38.334 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:24:38.334 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:24:38.334 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:24:38.334 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:38.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.331+0000 7fa47a914640 1 Processor -- start 2024-04-26T04:24:38.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.331+0000 7fa47a914640 1 -- start start 2024-04-26T04:24:38.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.331+0000 7fa47a914640 10 monclient: init 2024-04-26T04:24:38.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.331+0000 7fa47a914640 10 monclient: _reopen_session rank -1 2024-04-26T04:24:38.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.331+0000 7fa47a914640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-04-26T04:24:38.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.331+0000 7fa47a914640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa47409fe60 0x7fa4741372f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:38.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.331+0000 7fa47a914640 10 monclient(hunting): picked mon.noname-c con 0x7fa47409fe60 addr v2:172.21.15.186:3300/0 2024-04-26T04:24:38.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.331+0000 7fa47a914640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa4740a0240 addr v1:172.21.15.63:6789/0 2024-04-26T04:24:38.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.331+0000 7fa47a914640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa47413c100 addr v1:172.21.15.186:6789/0 2024-04-26T04:24:38.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.331+0000 7fa47a914640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa474137830 con 0x7fa4740a0240 2024-04-26T04:24:38.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.331+0000 7fa47a914640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4740a0c60 con 0x7fa47413c100 2024-04-26T04:24:38.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.331+0000 7fa47a914640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:38.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.331+0000 7fa47a914640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fa4740a0f40 con 0x7fa47409fe60 2024-04-26T04:24:38.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.331+0000 7fa47a914640 10 monclient(hunting): _renew_subs 2024-04-26T04:24:38.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.331+0000 7fa47a914640 10 monclient(hunting): authenticate will time out at 1310.041774s 2024-04-26T04:24:38.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.331+0000 7fa4737fe640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7fa4740a0240 0x7fa4740a2a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:55420/0 (socket says 172.21.15.63:55420) 2024-04-26T04:24:38.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.331+0000 7fa4737fe640 1 -- 172.21.15.63:0/4133477924 learned_addr learned my addr 172.21.15.63:0/4133477924 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-04-26T04:24:38.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.331+0000 7fa473fff640 1 --2- 172.21.15.63:0/4133477924 >> v2:172.21.15.186:3300/0 conn(0x7fa47409fe60 0x7fa4741372f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:38.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.331+0000 7fa473fff640 10 monclient(hunting): get_auth_request con 0x7fa47409fe60 auth_method 0 2024-04-26T04:24:38.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.331+0000 7fa473fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:38.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.331+0000 7fa473fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:38.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.331+0000 7fa473fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:38.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.331+0000 7fa473fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:38.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.331+0000 7fa473fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:38.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.331+0000 7fa473fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:24:38.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.331+0000 7fa473fff640 10 monclient(hunting): handle_auth_done global_id 34120 payload 274 2024-04-26T04:24:38.339 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.331+0000 7fa473fff640 1 -- 172.21.15.63:0/4133477924 >> v1:172.21.15.186:6789/0 conn(0x7fa47413c100 legacy=0x7fa47413e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:24:38.339 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.331+0000 7fa473fff640 1 -- 172.21.15.63:0/4133477924 >> v1:172.21.15.63:6789/0 conn(0x7fa4740a0240 legacy=0x7fa4740a2a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:24:38.339 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.331+0000 7fa473fff640 10 monclient: _finish_hunting 0 2024-04-26T04:24:38.339 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.331+0000 7fa473fff640 1 monclient: found mon.noname-c 2024-04-26T04:24:38.339 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.331+0000 7fa473fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:24:38.339 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.331+0000 7fa473fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.186:3300/0 2024-04-26T04:24:38.339 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.331+0000 7fa473fff640 1 -- 172.21.15.63:0/4133477924 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4740a1220 con 0x7fa47409fe60 2024-04-26T04:24:38.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.335+0000 7fa473fff640 1 -- 172.21.15.63:0/4133477924 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa47409fe60 msgr2=0x7fa4741372f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:24:38.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.335+0000 7fa473fff640 1 --2- 172.21.15.63:0/4133477924 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa47409fe60 0x7fa4741372f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa45c0097c0 tx=0x7fa45c01c2e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d326887bc123c1a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:38.342 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.335+0000 7fa472ffd640 1 -- 172.21.15.63:0/4133477924 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa45c0b0070 con 0x7fa47409fe60 2024-04-26T04:24:38.342 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.335+0000 7fa472ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:38.342 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.335+0000 7fa472ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:24:38.342 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.335+0000 7fa472ffd640 10 monclient: dump: 2024-04-26T04:24:38.342 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:38.342 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:38.342 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:38.342 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:38.342 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:38.343 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:38.343 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:38.343 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:38.343 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:38.343 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:38.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.335+0000 7fa472ffd640 10 monclient: _finish_auth 0 2024-04-26T04:24:38.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.335+0000 7fa472ffd640 10 monclient: _check_auth_tickets 2024-04-26T04:24:38.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.335+0000 7fa472ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:24:38.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.335+0000 7fa472ffd640 1 -- 172.21.15.63:0/4133477924 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa45c0ab600 con 0x7fa47409fe60 2024-04-26T04:24:38.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.335+0000 7fa47a914640 5 monclient: authenticate success, global_id 34120 2024-04-26T04:24:38.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.335+0000 7fa47a914640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T04:24:38.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.335+0000 7fa472ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T04:24:38.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.335+0000 7fa47a914640 10 monclient: get_monmap_and_config success 2024-04-26T04:24:38.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.335+0000 7fa47a914640 4 set_mon_vals no callback set 2024-04-26T04:24:38.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.335+0000 7fa472ffd640 1 -- 172.21.15.63:0/4133477924 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa45c0ab9d0 con 0x7fa47409fe60 2024-04-26T04:24:38.344 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.335+0000 7fa472ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:38.344 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.335+0000 7fa472ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T04:24:38.344 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.335+0000 7fa472ffd640 10 monclient: dump: 2024-04-26T04:24:38.344 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:38.344 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:38.344 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:38.344 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:38.344 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:38.344 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:38.344 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:38.344 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:38.344 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:38.344 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:38.344 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.339+0000 7fa47a914640 10 monclient: shutdown 2024-04-26T04:24:38.344 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.339+0000 7fa47a914640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:24:38.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.339+0000 7fa47a914640 1 -- 172.21.15.63:0/4133477924 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa47409fe60 msgr2=0x7fa4741372f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:38.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.339+0000 7fa47a914640 1 --2- 172.21.15.63:0/4133477924 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa47409fe60 0x7fa4741372f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fa45c0097c0 tx=0x7fa45c01c2e0 comp rx=0 tx=0).stop 2024-04-26T04:24:38.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.339+0000 7fa47a914640 1 -- 172.21.15.63:0/4133477924 shutdown_connections 2024-04-26T04:24:38.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.339+0000 7fa47a914640 1 --2- 172.21.15.63:0/4133477924 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa47409fe60 0x7fa4741372f0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:38.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.339+0000 7fa47a914640 1 -- 172.21.15.63:0/4133477924 >> 172.21.15.63:0/4133477924 conn(0x7fa4740a5fb0 msgr2=0x7fa4740a63c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:24:38.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.339+0000 7fa47a914640 1 -- 172.21.15.63:0/4133477924 shutdown_connections 2024-04-26T04:24:38.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.339+0000 7fa47a914640 1 -- 172.21.15.63:0/4133477924 wait complete. 2024-04-26T04:24:38.349 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.339+0000 7fa47a914640 10 monclient: build_initial_monmap 2024-04-26T04:24:38.349 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.339+0000 7fa47a914640 10 monclient: monmap: 2024-04-26T04:24:38.349 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:24:38.349 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:24:38.349 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:24:38.349 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:24:38.349 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:24:38.349 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:38.350 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:24:38.350 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:24:38.350 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:24:38.350 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:38.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.339+0000 7fa47a914640 1 Processor -- start 2024-04-26T04:24:38.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.339+0000 7fa47a914640 1 -- start start 2024-04-26T04:24:38.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.339+0000 7fa47a914640 10 monclient: init 2024-04-26T04:24:38.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.339+0000 7fa47a914640 10 monclient: _reopen_session rank -1 2024-04-26T04:24:38.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.339+0000 7fa47a914640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T04:24:38.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.339+0000 7fa47a914640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa47409fe60 0x7fa474132240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:38.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.339+0000 7fa47a914640 10 monclient(hunting): picked mon.noname-b con 0x7fa47409fe60 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:24:38.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.339+0000 7fa47a914640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa4740a0240 0x7fa474133c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:38.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.339+0000 7fa47a914640 10 monclient(hunting): picked mon.noname-a con 0x7fa4740a0240 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:24:38.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.339+0000 7fa47a914640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fa47413c100 0x7fa474134140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:38.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.339+0000 7fa47a914640 10 monclient(hunting): picked mon.noname-c con 0x7fa47413c100 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:24:38.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.339+0000 7fa47a914640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:38.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.339+0000 7fa47a914640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fa474142e50 con 0x7fa4740a0240 2024-04-26T04:24:38.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.339+0000 7fa47a914640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:38.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.339+0000 7fa47a914640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa474142cd0 con 0x7fa47409fe60 2024-04-26T04:24:38.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.339+0000 7fa47a914640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:38.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.339+0000 7fa47a914640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7fa474142fd0 con 0x7fa47413c100 2024-04-26T04:24:38.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.339+0000 7fa47a914640 10 monclient(hunting): _renew_subs 2024-04-26T04:24:38.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.339+0000 7fa47a914640 10 monclient(hunting): authenticate will time out at 1310.050175s 2024-04-26T04:24:38.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.339+0000 7fa4737fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa4740a0240 0x7fa474133c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:38.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.339+0000 7fa478eb1640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fa47413c100 0x7fa474134140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:38.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.339+0000 7fa4737fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa4740a0240 0x7fa474133c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:33290/0 (socket says 172.21.15.63:33290) 2024-04-26T04:24:38.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.339+0000 7fa4737fe640 1 -- 172.21.15.63:0/454258217 learned_addr learned my addr 172.21.15.63:0/454258217 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:24:38.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.339+0000 7fa4737fe640 10 monclient(hunting): get_auth_request con 0x7fa4740a0240 auth_method 0 2024-04-26T04:24:38.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.339+0000 7fa4737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:38.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.339+0000 7fa4737fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:38.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.339+0000 7fa4737fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:38.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.339+0000 7fa478eb1640 10 monclient(hunting): get_auth_request con 0x7fa47413c100 auth_method 0 2024-04-26T04:24:38.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.339+0000 7fa478eb1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:38.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.339+0000 7fa478eb1640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:38.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.339+0000 7fa478eb1640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:38.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.339+0000 7fa4737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:38.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.339+0000 7fa4737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:38.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.339+0000 7fa4737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:24:38.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.339+0000 7fa478eb1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:38.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.339+0000 7fa478eb1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:38.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.339+0000 7fa478eb1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:24:38.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.339+0000 7fa473fff640 1 --2- 172.21.15.63:0/454258217 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa47409fe60 0x7fa474132240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:38.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.339+0000 7fa473fff640 10 monclient(hunting): get_auth_request con 0x7fa47409fe60 auth_method 0 2024-04-26T04:24:38.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.339+0000 7fa473fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:38.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.339+0000 7fa473fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:38.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.339+0000 7fa473fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:38.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.339+0000 7fa4737fe640 10 monclient(hunting): handle_auth_done global_id 5355 payload 824 2024-04-26T04:24:38.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.339+0000 7fa4737fe640 1 -- 172.21.15.63:0/454258217 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fa47413c100 msgr2=0x7fa474134140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:38.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.339+0000 7fa4737fe640 1 --2- 172.21.15.63:0/454258217 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fa47413c100 0x7fa474134140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:38.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.339+0000 7fa4737fe640 1 -- 172.21.15.63:0/454258217 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa47409fe60 msgr2=0x7fa474132240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:38.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.339+0000 7fa4737fe640 1 --2- 172.21.15.63:0/454258217 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa47409fe60 0x7fa474132240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:38.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.339+0000 7fa4737fe640 10 monclient: _finish_hunting 0 2024-04-26T04:24:38.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.339+0000 7fa4737fe640 1 monclient: found mon.noname-a 2024-04-26T04:24:38.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.339+0000 7fa4737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:24:38.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.339+0000 7fa4737fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:24:38.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.339+0000 7fa4737fe640 1 -- 172.21.15.63:0/454258217 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa47406ea50 con 0x7fa4740a0240 2024-04-26T04:24:38.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.343+0000 7fa4737fe640 1 -- 172.21.15.63:0/454258217 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa4740a0240 msgr2=0x7fa474133c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:24:38.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.343+0000 7fa4737fe640 1 --2- 172.21.15.63:0/454258217 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa4740a0240 0x7fa474133c00 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fa46400c480 tx=0x7fa46400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:38.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.343+0000 7fa470ff9640 1 -- 172.21.15.63:0/454258217 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa464007070 con 0x7fa4740a0240 2024-04-26T04:24:38.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.343+0000 7fa470ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:38.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.343+0000 7fa470ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:24:38.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.343+0000 7fa470ff9640 10 monclient: dump: 2024-04-26T04:24:38.354 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:38.354 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:38.354 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:38.354 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:38.354 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:38.354 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:38.354 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:38.354 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:38.354 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:38.354 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:38.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.343+0000 7fa470ff9640 10 monclient: _finish_auth 0 2024-04-26T04:24:38.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.343+0000 7fa470ff9640 10 monclient: _check_auth_tickets 2024-04-26T04:24:38.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.343+0000 7fa470ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:24:38.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.343+0000 7fa47a914640 5 monclient: authenticate success, global_id 5355 2024-04-26T04:24:38.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.343+0000 7fa470ff9640 1 -- 172.21.15.63:0/454258217 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa464007210 con 0x7fa4740a0240 2024-04-26T04:24:38.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.343+0000 7fa47a914640 10 monclient: _renew_subs 2024-04-26T04:24:38.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.343+0000 7fa47a914640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:24:38.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.343+0000 7fa47a914640 1 -- 172.21.15.63:0/454258217 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa47406ed30 con 0x7fa4740a0240 2024-04-26T04:24:38.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.343+0000 7fa470ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T04:24:38.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.343+0000 7fa470ff9640 1 -- 172.21.15.63:0/454258217 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa464007600 con 0x7fa4740a0240 2024-04-26T04:24:38.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.343+0000 7fa470ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:38.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.343+0000 7fa470ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:24:38.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.343+0000 7fa470ff9640 10 monclient: dump: 2024-04-26T04:24:38.355 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:38.355 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:38.355 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:38.355 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:38.356 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:38.356 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:38.356 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:38.356 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:38.356 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:38.356 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:38.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.343+0000 7fa47a914640 10 monclient: _renew_subs 2024-04-26T04:24:38.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.343+0000 7fa4727fc640 4 set_mon_vals no callback set 2024-04-26T04:24:38.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.343+0000 7fa47a914640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:24:38.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.343+0000 7fa47a914640 1 -- 172.21.15.63:0/454258217 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4740a7640 con 0x7fa4740a0240 2024-04-26T04:24:38.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.343+0000 7fa47a914640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:24:38.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.343+0000 7fa47a914640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:24:38.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.343+0000 7fa47a914640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:24:38.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.343+0000 7fa470ff9640 1 -- 172.21.15.63:0/454258217 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fa46401b400 con 0x7fa4740a0240 2024-04-26T04:24:38.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.343+0000 7fa47a914640 1 -- 172.21.15.63:0/454258217 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa47406e600 con 0x7fa4740a0240 2024-04-26T04:24:38.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.343+0000 7fa470ff9640 1 --2- 172.21.15.63:0/454258217 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fa454035f10 0x7fa4540383d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:38.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.343+0000 7fa470ff9640 1 -- 172.21.15.63:0/454258217 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(220..220 src has 1..220) ==== 4346+0+0 (secure 0 0 0) 0x7fa464048a10 con 0x7fa4740a0240 2024-04-26T04:24:38.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.343+0000 7fa473fff640 1 --2- 172.21.15.63:0/454258217 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fa454035f10 0x7fa4540383d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:38.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.343+0000 7fa473fff640 10 monclient: get_auth_request con 0x7fa454035f10 auth_method 0 2024-04-26T04:24:38.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.347+0000 7fa473fff640 1 --2- 172.21.15.63:0/454258217 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fa454035f10 0x7fa4540383d0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa47406dab0 tx=0x7fa45c0a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:38.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.347+0000 7fa470ff9640 1 -- 172.21.15.63:0/454258217 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fa47406e600 con 0x7fa4740a0240 2024-04-26T04:24:38.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.347+0000 7fa470ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:24:38.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.347+0000 7fa470ff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:24:38.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.575+0000 7fa47a914640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-04-26T04:24:38.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.575+0000 7fa47a914640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-04-26T04:24:38.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.575+0000 7fa47a914640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:24:38.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.575+0000 7fa47a914640 1 -- 172.21.15.63:0/454258217 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fa474062e10 con 0x7fa4740a0240 2024-04-26T04:24:38.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.575+0000 7fa470ff9640 1 -- 172.21.15.63:0/454258217 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fa4640475c0 con 0x7fa4740a0240 2024-04-26T04:24:38.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.575+0000 7fa470ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-04-26T04:24:38.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.575+0000 7fa470ff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:24:38.581 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.575+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:24:38.582 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.575+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=10,config=1,health=11,kv=12,logm=66} crc {auth=3349286680,config=1228700967,health=3427408276,kv=3642618011,logm=1580325458}) 2024-04-26T04:24:38.582 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.575+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=9,config=2,health=12,kv=12,logm=65} crc {auth=2909308837,config=2586702516,health=793372127,kv=3642618011,logm=2527504039}) 2024-04-26T04:24:38.582 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.575+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:24:38.582 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.575+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=10,config=1,health=11,kv=12,logm=66} crc {auth=3349286680,config=1228700967,health=3427408276,kv=3642618011,logm=1580325458}) 2024-04-26T04:24:38.582 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.575+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=10,config=2,health=12,kv=12,logm=64} crc {auth=3349286680,config=2586702516,health=3873195142,kv=3642618011,logm=1155343595}) 2024-04-26T04:24:38.584 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.579+0000 7fa47a914640 1 -- 172.21.15.63:0/454258217 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fa454035f10 msgr2=0x7fa4540383d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:38.584 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.579+0000 7fa47a914640 1 --2- 172.21.15.63:0/454258217 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fa454035f10 0x7fa4540383d0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa47406dab0 tx=0x7fa45c0a6000 comp rx=0 tx=0).stop 2024-04-26T04:24:38.584 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.579+0000 7fa47a914640 10 monclient: shutdown 2024-04-26T04:24:38.584 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.579+0000 7fa47a914640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:24:38.584 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.579+0000 7fa47a914640 1 -- 172.21.15.63:0/454258217 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa4740a0240 msgr2=0x7fa474133c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:38.584 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.579+0000 7fa47a914640 1 --2- 172.21.15.63:0/454258217 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa4740a0240 0x7fa474133c00 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fa46400c480 tx=0x7fa46400c940 comp rx=0 tx=0).stop 2024-04-26T04:24:38.584 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.579+0000 7fa47a914640 1 -- 172.21.15.63:0/454258217 shutdown_connections 2024-04-26T04:24:38.584 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.579+0000 7fa47a914640 1 --2- 172.21.15.63:0/454258217 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fa454035f10 0x7fa4540383d0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:38.584 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.579+0000 7fa47a914640 1 --2- 172.21.15.63:0/454258217 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fa47413c100 0x7fa474134140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:38.584 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.579+0000 7fa47a914640 1 --2- 172.21.15.63:0/454258217 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa4740a0240 0x7fa474133c00 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:38.585 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.579+0000 7fa47a914640 1 --2- 172.21.15.63:0/454258217 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa47409fe60 0x7fa474132240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:38.585 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.579+0000 7fa47a914640 1 -- 172.21.15.63:0/454258217 >> 172.21.15.63:0/454258217 conn(0x7fa4740a5fb0 msgr2=0x7fa4740a19f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:24:38.585 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.579+0000 7fa47a914640 1 -- 172.21.15.63:0/454258217 shutdown_connections 2024-04-26T04:24:38.585 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:38.579+0000 7fa47a914640 1 -- 172.21.15.63:0/454258217 wait complete. 2024-04-26T04:24:38.587 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.579+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:24:38.587 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.579+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2095014528}) 2024-04-26T04:24:38.587 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.579+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=971379381}) 2024-04-26T04:24:38.588 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.579+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:24:38.588 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.579+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2095014528}) 2024-04-26T04:24:38.588 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.579+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=681460402}) 2024-04-26T04:24:38.588 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.579+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:24:38.588 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.579+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2753414745}) 2024-04-26T04:24:38.588 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.579+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=718707645}) 2024-04-26T04:24:38.588 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.579+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:24:38.588 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.579+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2753414745}) 2024-04-26T04:24:38.588 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.579+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=152499338}) 2024-04-26T04:24:38.588 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.583+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:24:38.588 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.583+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=936377663}) 2024-04-26T04:24:38.588 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.583+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=262512979}) 2024-04-26T04:24:38.589 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.583+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:24:38.589 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.583+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=936377663}) 2024-04-26T04:24:38.589 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.583+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3526596219}) 2024-04-26T04:24:38.590 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.583+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:24:38.590 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.583+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1341444725}) 2024-04-26T04:24:38.590 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.583+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1638710423}) 2024-04-26T04:24:38.591 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.583+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:24:38.591 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.583+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1341444725}) 2024-04-26T04:24:38.591 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.583+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=843182653}) 2024-04-26T04:24:38.591 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.587+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:24:38.591 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.587+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=738833268}) 2024-04-26T04:24:38.591 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.587+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3382668156}) 2024-04-26T04:24:38.591 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.587+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:24:38.591 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.587+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=738833268}) 2024-04-26T04:24:38.591 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.587+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1682314858}) 2024-04-26T04:24:38.592 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.587+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:24:38.592 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.587+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=896312462}) 2024-04-26T04:24:38.593 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.587+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2224757576}) 2024-04-26T04:24:38.593 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.587+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:24:38.593 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.587+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=896312462}) 2024-04-26T04:24:38.593 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.587+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1102003875}) 2024-04-26T04:24:38.594 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.591+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:24:38.594 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.591+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=662994365}) 2024-04-26T04:24:38.594 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.591+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2476581299}) 2024-04-26T04:24:38.594 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.591+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:24:38.594 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.591+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=662994365}) 2024-04-26T04:24:38.594 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.591+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=644724481}) 2024-04-26T04:24:38.595 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.591+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:24:38.596 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.591+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2149373157}) 2024-04-26T04:24:38.596 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.591+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2943814649}) 2024-04-26T04:24:38.596 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.591+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:24:38.596 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.591+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2149373157}) 2024-04-26T04:24:38.596 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.591+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3331295732}) 2024-04-26T04:24:38.597 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.591+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:24:38.597 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.591+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3138263644}) 2024-04-26T04:24:38.597 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.591+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2829714084}) 2024-04-26T04:24:38.598 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.591+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:24:38.598 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.591+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3138263644}) 2024-04-26T04:24:38.598 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.591+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2275204852}) 2024-04-26T04:24:38.599 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.595+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:24:38.600 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.595+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1688242662}) 2024-04-26T04:24:38.600 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.595+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=452270435}) 2024-04-26T04:24:38.600 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.595+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:24:38.600 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.595+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1688242662}) 2024-04-26T04:24:38.600 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.595+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4266840698}) 2024-04-26T04:24:38.601 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.595+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:24:38.601 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.595+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1538722886}) 2024-04-26T04:24:38.603 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-04-26T04:24:38.605 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.595+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1648604260}) 2024-04-26T04:24:38.606 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.595+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:24:38.606 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.595+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1538722886}) 2024-04-26T04:24:38.606 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.595+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2977305580}) 2024-04-26T04:24:38.606 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.599+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:24:38.606 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.599+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=26,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=16,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=23} crc {logm=1692932203,mdsmap=2840952586,mgr=4110609431,mgr_command_descs=3057339438,mgr_metadata=3709105003,mgrstat=1273314723,mon_config_key=2976374045,monmap=1192322635,osd_metadata=4080886778,osd_pg_creating=1348551291,osdmap=3000752984}) 2024-04-26T04:24:38.606 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.599+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=27,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=15,monmap=3,osd_metadata=5,osdmap=26} crc {logm=4120529568,mdsmap=2840952586,mgr=4110609431,mgr_command_descs=3057339438,mgr_metadata=3709105003,mgrstat=3975887365,mon_config_key=1236337183,monmap=1192322635,osd_metadata=1089039992,osdmap=2707311207}) 2024-04-26T04:24:38.607 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.599+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:24:38.607 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.599+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=26,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=16,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=23} crc {logm=1692932203,mdsmap=2840952586,mgr=4110609431,mgr_command_descs=3057339438,mgr_metadata=3709105003,mgrstat=1273314723,mon_config_key=2976374045,monmap=1192322635,osd_metadata=4080886778,osd_pg_creating=1348551291,osdmap=3000752984}) 2024-04-26T04:24:38.607 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.599+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=26,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=15,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=27} crc {logm=1710483204,mdsmap=3764978221,mgr=3783745836,mgr_command_descs=3057339438,mgr_metadata=3709105003,mgrstat=3145120965,mon_config_key=3260272251,monmap=1192322635,osd_metadata=1397308133,osd_pg_creating=1348551291,osdmap=641493424}) 2024-04-26T04:24:38.607 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.599+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:24:38.607 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.599+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2985142651}) 2024-04-26T04:24:38.607 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.599+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1094552287}) 2024-04-26T04:24:38.608 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.599+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:24:38.608 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.599+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2985142651}) 2024-04-26T04:24:38.608 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.599+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=825387165}) 2024-04-26T04:24:38.612 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.603+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:24:38.612 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.603+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3941463778}) 2024-04-26T04:24:38.612 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.603+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3912489066}) 2024-04-26T04:24:38.612 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.603+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:24:38.612 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.603+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3941463778}) 2024-04-26T04:24:38.612 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.603+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=736085786}) 2024-04-26T04:24:38.613 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.607+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:24:38.613 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.607+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=164613111}) 2024-04-26T04:24:38.613 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.607+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=999058287}) 2024-04-26T04:24:38.613 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.607+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:24:38.614 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.607+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=164613111}) 2024-04-26T04:24:38.614 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.607+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1127657485}) 2024-04-26T04:24:38.614 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.607+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:24:38.614 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.607+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=8} crc {osdmap=3851548802}) 2024-04-26T04:24:38.614 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.607+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=9} crc {osdmap=3628588002}) 2024-04-26T04:24:38.614 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.607+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:24:38.615 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.607+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=8} crc {osdmap=3851548802}) 2024-04-26T04:24:38.615 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:24:38.607+0000 7f2c5657c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=9} crc {osdmap=3628588002}) 2024-04-26T04:24:39.604 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-26T04:24:39.604 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T04:24:39.738 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.731+0000 7f67208e8640 10 monclient: get_monmap_and_config 2024-04-26T04:24:39.738 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.731+0000 7f67208e8640 10 monclient: build_initial_monmap 2024-04-26T04:24:39.738 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.731+0000 7f67208e8640 10 monclient: monmap: 2024-04-26T04:24:39.739 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:24:39.739 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:39.739 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:24:39.735258+0000 2024-04-26T04:24:39.739 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:24:39.735258+0000 2024-04-26T04:24:39.739 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:24:39.739 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:39.739 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:24:39.739 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:24:39.739 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:24:39.739 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:24:39.739 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:24:39.739 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:39.739 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.731+0000 7f67208e8640 1 Processor -- start 2024-04-26T04:24:39.740 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.731+0000 7f67208e8640 1 -- start start 2024-04-26T04:24:39.740 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.731+0000 7f67208e8640 10 monclient: init 2024-04-26T04:24:39.740 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.731+0000 7f67208e8640 10 monclient: _reopen_session rank -1 2024-04-26T04:24:39.740 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.731+0000 7f67208e8640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-04-26T04:24:39.740 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.731+0000 7f67208e8640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f671c09c6a0 0x7f671c0996b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:39.740 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.731+0000 7f67208e8640 10 monclient(hunting): picked mon.noname-b con 0x7f671c09c6a0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:24:39.740 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.731+0000 7f67208e8640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f671c09ca80 0x7f671c09ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:39.740 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.731+0000 7f67208e8640 10 monclient(hunting): picked mon.noname-a con 0x7f671c09ca80 addr v2:172.21.15.63:3300/0 2024-04-26T04:24:39.740 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.731+0000 7f67208e8640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f671c098090 0x7f671c0984a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:39.740 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.731+0000 7f67208e8640 10 monclient(hunting): picked mon.noname-c con 0x7f671c098090 addr v2:172.21.15.186:3300/0 2024-04-26T04:24:39.740 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.731+0000 7f67208e8640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:39.740 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.731+0000 7f67208e8640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7f671c09c1d0 con 0x7f671c09ca80 2024-04-26T04:24:39.740 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.731+0000 7f67208e8640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:39.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.731+0000 7f67208e8640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f671c09d9f0 con 0x7f671c098090 2024-04-26T04:24:39.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.731+0000 7f67208e8640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:39.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.731+0000 7f67208e8640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f671c0992d0 con 0x7f671c09c6a0 2024-04-26T04:24:39.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.731+0000 7f67208e8640 10 monclient(hunting): _renew_subs 2024-04-26T04:24:39.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.731+0000 7f67208e8640 10 monclient(hunting): authenticate will time out at 1311.443009s 2024-04-26T04:24:39.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.731+0000 7f671a59c640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f671c09c6a0 0x7f671c0996b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:39.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.731+0000 7f6719d9b640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f671c09ca80 0x7f671c09ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:39.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.731+0000 7f671a59c640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f671c09c6a0 0x7f671c0996b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:59234/0 (socket says 172.21.15.63:59234) 2024-04-26T04:24:39.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.731+0000 7f671a59c640 1 -- 172.21.15.63:0/256960098 learned_addr learned my addr 172.21.15.63:0/256960098 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:24:39.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.731+0000 7f671a59c640 10 monclient(hunting): get_auth_request con 0x7f671c09c6a0 auth_method 0 2024-04-26T04:24:39.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.731+0000 7f671a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:39.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.731+0000 7f671a59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:39.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.731+0000 7f671a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:39.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.731+0000 7f6719d9b640 10 monclient(hunting): get_auth_request con 0x7f671c09ca80 auth_method 0 2024-04-26T04:24:39.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.731+0000 7f6719d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:39.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.731+0000 7f6719d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:39.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.731+0000 7f6719d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:39.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.731+0000 7f6719d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:39.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.731+0000 7f6719d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:39.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.731+0000 7f6719d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:24:39.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.731+0000 7f671a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:39.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.731+0000 7f671a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:39.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.731+0000 7f671a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:24:39.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.731+0000 7f671ad9d640 1 --2- 172.21.15.63:0/256960098 >> v2:172.21.15.186:3300/0 conn(0x7f671c098090 0x7f671c0984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:39.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.731+0000 7f671ad9d640 10 monclient(hunting): get_auth_request con 0x7f671c098090 auth_method 0 2024-04-26T04:24:39.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.731+0000 7f671ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:39.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.731+0000 7f671ad9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:39.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.731+0000 7f671ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:39.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.731+0000 7f6719d9b640 10 monclient(hunting): handle_auth_done global_id 5358 payload 274 2024-04-26T04:24:39.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.731+0000 7f6719d9b640 1 -- 172.21.15.63:0/256960098 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f671c09c6a0 msgr2=0x7f671c0996b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:24:39.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.731+0000 7f6719d9b640 1 --2- 172.21.15.63:0/256960098 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f671c09c6a0 0x7f671c0996b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:39.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.731+0000 7f6719d9b640 1 -- 172.21.15.63:0/256960098 >> v2:172.21.15.186:3300/0 conn(0x7f671c098090 msgr2=0x7f671c0984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:24:39.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.731+0000 7f6719d9b640 1 --2- 172.21.15.63:0/256960098 >> v2:172.21.15.186:3300/0 conn(0x7f671c098090 0x7f671c0984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:39.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.731+0000 7f6719d9b640 10 monclient: _finish_hunting 0 2024-04-26T04:24:39.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.731+0000 7f6719d9b640 1 monclient: found mon.noname-a 2024-04-26T04:24:39.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.731+0000 7f6719d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:24:39.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.731+0000 7f6719d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:24:39.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.731+0000 7f6719d9b640 1 -- 172.21.15.63:0/256960098 --> v2:172.21.15.63:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f671c09dc30 con 0x7f671c09ca80 2024-04-26T04:24:39.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.735+0000 7f6719d9b640 1 -- 172.21.15.63:0/256960098 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f671c09ca80 msgr2=0x7f671c09ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:24:39.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.735+0000 7f6719d9b640 1 --2- 172.21.15.63:0/256960098 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f671c09ca80 0x7f671c09ce60 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f671c09d7d0 tx=0x7f671001c310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9ee66ed3d9bac3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:39.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f671959a640 1 -- 172.21.15.63:0/256960098 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67100b0070 con 0x7f671c09ca80 2024-04-26T04:24:39.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f671959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:39.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f671959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:24:39.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f671959a640 10 monclient: dump: 2024-04-26T04:24:39.750 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:39.750 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:39.750 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:39.750 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:39.750 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:39.751 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:39.751 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:39.751 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:39.751 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:39.751 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:39.751 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f671959a640 10 monclient: _finish_auth 0 2024-04-26T04:24:39.751 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f671959a640 10 monclient: _check_auth_tickets 2024-04-26T04:24:39.751 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f671959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:24:39.751 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f67208e8640 5 monclient: authenticate success, global_id 5358 2024-04-26T04:24:39.751 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f67208e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T04:24:39.751 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f671959a640 1 -- 172.21.15.63:0/256960098 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67100ab600 con 0x7f671c09ca80 2024-04-26T04:24:39.751 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f671959a640 10 monclient: handle_config config(0 keys) 2024-04-26T04:24:39.751 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f67208e8640 10 monclient: get_monmap_and_config success 2024-04-26T04:24:39.751 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f671959a640 1 -- 172.21.15.63:0/256960098 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67100ab9d0 con 0x7f671c09ca80 2024-04-26T04:24:39.751 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f67208e8640 4 set_mon_vals no callback set 2024-04-26T04:24:39.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f671959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:39.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f671959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:24:39.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f671959a640 10 monclient: dump: 2024-04-26T04:24:39.752 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:39.752 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:39.752 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:39.752 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:39.752 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:39.752 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:39.752 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:39.752 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:39.752 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:39.752 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:39.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f67208e8640 10 monclient: shutdown 2024-04-26T04:24:39.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f67208e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:24:39.753 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f67208e8640 1 -- 172.21.15.63:0/256960098 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f671c09ca80 msgr2=0x7f671c09ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:39.753 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f67208e8640 1 --2- 172.21.15.63:0/256960098 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f671c09ca80 0x7f671c09ce60 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f671c09d7d0 tx=0x7f671001c310 comp rx=0 tx=0).stop 2024-04-26T04:24:39.753 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f67208e8640 1 -- 172.21.15.63:0/256960098 shutdown_connections 2024-04-26T04:24:39.753 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f67208e8640 1 --2- 172.21.15.63:0/256960098 >> v2:172.21.15.186:3300/0 conn(0x7f671c098090 0x7f671c0984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:39.753 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f67208e8640 1 --2- 172.21.15.63:0/256960098 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f671c09ca80 0x7f671c09ce60 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:39.753 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f67208e8640 1 --2- 172.21.15.63:0/256960098 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f671c09c6a0 0x7f671c0996b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:39.753 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f67208e8640 1 -- 172.21.15.63:0/256960098 >> 172.21.15.63:0/256960098 conn(0x7f671c0a1e70 msgr2=0x7f671c0a2280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:24:39.753 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f67208e8640 1 -- 172.21.15.63:0/256960098 shutdown_connections 2024-04-26T04:24:39.753 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f67208e8640 1 -- 172.21.15.63:0/256960098 wait complete. 2024-04-26T04:24:39.753 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f67208e8640 10 monclient: build_initial_monmap 2024-04-26T04:24:39.753 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f67208e8640 10 monclient: monmap: 2024-04-26T04:24:39.753 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:24:39.753 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:24:39.753 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:24:39.753 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:24:39.753 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:24:39.754 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:39.754 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:24:39.754 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:24:39.754 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:24:39.754 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:39.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f67208e8640 1 Processor -- start 2024-04-26T04:24:39.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f67208e8640 1 -- start start 2024-04-26T04:24:39.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f67208e8640 10 monclient: init 2024-04-26T04:24:39.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f67208e8640 10 monclient: _reopen_session rank -1 2024-04-26T04:24:39.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f67208e8640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T04:24:39.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f67208e8640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f671c098090 0x7f671c1cace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:39.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f67208e8640 10 monclient(hunting): picked mon.noname-c con 0x7f671c098090 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:24:39.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f67208e8640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f671c09c6a0 0x7f671c097590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:39.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f67208e8640 10 monclient(hunting): picked mon.noname-b con 0x7f671c09c6a0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:24:39.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f67208e8640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f671c09ca80 0x7f671c097ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:39.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f67208e8640 10 monclient(hunting): picked mon.noname-a con 0x7f671c09ca80 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:24:39.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f67208e8640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:39.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f67208e8640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f671c0a16a0 con 0x7f671c09ca80 2024-04-26T04:24:39.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f67208e8640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:39.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f67208e8640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f671c09d4f0 con 0x7f671c09c6a0 2024-04-26T04:24:39.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f67208e8640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:39.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f67208e8640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f671c0a1820 con 0x7f671c098090 2024-04-26T04:24:39.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f67208e8640 10 monclient(hunting): _renew_subs 2024-04-26T04:24:39.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f67208e8640 10 monclient(hunting): authenticate will time out at 1311.450316s 2024-04-26T04:24:39.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f671a59c640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f671c098090 0x7f671c1cace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:39.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f671a59c640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f671c098090 0x7f671c1cace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:59236/0 (socket says 172.21.15.63:59236) 2024-04-26T04:24:39.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f671a59c640 1 -- 172.21.15.63:0/1852270588 learned_addr learned my addr 172.21.15.63:0/1852270588 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:24:39.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f671a59c640 10 monclient(hunting): get_auth_request con 0x7f671c098090 auth_method 0 2024-04-26T04:24:39.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f671a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:39.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f671a59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:39.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f671a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:39.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f671ad9d640 1 --2- 172.21.15.63:0/1852270588 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f671c09ca80 0x7f671c097ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:39.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f671ad9d640 10 monclient(hunting): get_auth_request con 0x7f671c09ca80 auth_method 0 2024-04-26T04:24:39.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f671ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:39.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f671ad9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:39.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f671ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:39.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f671a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:39.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f671a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:39.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f671a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:24:39.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f6719d9b640 1 --2- 172.21.15.63:0/1852270588 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f671c09c6a0 0x7f671c097590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:39.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f6719d9b640 10 monclient(hunting): get_auth_request con 0x7f671c09c6a0 auth_method 0 2024-04-26T04:24:39.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f6719d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:39.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f6719d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:39.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f6719d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:39.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f671ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:39.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f671ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:39.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f671ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:24:39.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f671a59c640 10 monclient(hunting): handle_auth_done global_id 24257 payload 824 2024-04-26T04:24:39.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f671a59c640 1 -- 172.21.15.63:0/1852270588 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f671c09c6a0 msgr2=0x7f671c097590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:39.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f671a59c640 1 --2- 172.21.15.63:0/1852270588 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f671c09c6a0 0x7f671c097590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:39.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f671a59c640 1 -- 172.21.15.63:0/1852270588 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f671c09ca80 msgr2=0x7f671c097ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:39.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f671a59c640 1 --2- 172.21.15.63:0/1852270588 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f671c09ca80 0x7f671c097ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:39.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f671a59c640 10 monclient: _finish_hunting 0 2024-04-26T04:24:39.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f671a59c640 1 monclient: found mon.noname-c 2024-04-26T04:24:39.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f671a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:24:39.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f671a59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.63:3301/0 2024-04-26T04:24:39.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f671a59c640 1 -- 172.21.15.63:0/1852270588 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f671c090de0 con 0x7f671c098090 2024-04-26T04:24:39.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.739+0000 7f6719d9b640 1 --2- 172.21.15.63:0/1852270588 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f671c09c6a0 0x7f671c097590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T04:24:39.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.743+0000 7f671a59c640 1 -- 172.21.15.63:0/1852270588 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f671c098090 msgr2=0x7f671c1cace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:24:39.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.743+0000 7f671a59c640 1 --2- 172.21.15.63:0/1852270588 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f671c098090 0x7f671c1cace0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f670c00f930 tx=0x7f670c00fdf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:39.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.743+0000 7f670affd640 1 -- 172.21.15.63:0/1852270588 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f670c00b040 con 0x7f671c098090 2024-04-26T04:24:39.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.743+0000 7f670affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:39.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.743+0000 7f670affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:24:39.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.743+0000 7f670affd640 10 monclient: dump: 2024-04-26T04:24:39.758 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:39.758 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:39.758 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:39.758 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:39.758 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:39.758 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:39.758 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:39.758 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:39.758 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:39.758 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:39.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.743+0000 7f670affd640 10 monclient: _finish_auth 0 2024-04-26T04:24:39.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.743+0000 7f670affd640 10 monclient: _check_auth_tickets 2024-04-26T04:24:39.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.743+0000 7f670affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:24:39.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.743+0000 7f670affd640 1 -- 172.21.15.63:0/1852270588 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f670c012460 con 0x7f671c098090 2024-04-26T04:24:39.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.743+0000 7f670affd640 10 monclient: handle_config config(0 keys) 2024-04-26T04:24:39.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.743+0000 7f670affd640 1 -- 172.21.15.63:0/1852270588 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f670c012830 con 0x7f671c098090 2024-04-26T04:24:39.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.743+0000 7f670affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:39.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.743+0000 7f670affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:24:39.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.743+0000 7f670affd640 10 monclient: dump: 2024-04-26T04:24:39.759 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:39.759 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:39.759 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:39.759 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:39.759 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:39.759 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:39.759 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:39.760 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:39.760 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:39.760 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:39.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.743+0000 7f67208e8640 5 monclient: authenticate success, global_id 24257 2024-04-26T04:24:39.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.743+0000 7f67208e8640 10 monclient: _renew_subs 2024-04-26T04:24:39.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.743+0000 7f670bfff640 4 set_mon_vals no callback set 2024-04-26T04:24:39.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.743+0000 7f67208e8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:24:39.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.743+0000 7f67208e8640 1 -- 172.21.15.63:0/1852270588 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f671c08eb10 con 0x7f671c098090 2024-04-26T04:24:39.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.743+0000 7f67208e8640 10 monclient: _renew_subs 2024-04-26T04:24:39.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.743+0000 7f67208e8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:24:39.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.743+0000 7f67208e8640 1 -- 172.21.15.63:0/1852270588 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f671c09eb90 con 0x7f671c098090 2024-04-26T04:24:39.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.743+0000 7f67208e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:24:39.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.743+0000 7f67208e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:24:39.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.743+0000 7f67208e8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:24:39.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.743+0000 7f670affd640 1 -- 172.21.15.63:0/1852270588 <== mon.2 v2:172.21.15.63:3301/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f670c012a10 con 0x7f671c098090 2024-04-26T04:24:39.761 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.743+0000 7f67208e8640 1 -- 172.21.15.63:0/1852270588 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f671c062e10 con 0x7f671c098090 2024-04-26T04:24:39.761 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.743+0000 7f670affd640 1 --2- 172.21.15.63:0/1852270588 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f66fc035f40 0x7f66fc038400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:39.761 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.743+0000 7f670affd640 1 -- 172.21.15.63:0/1852270588 <== mon.2 v2:172.21.15.63:3301/0 5 ==== osd_map(221..221 src has 1..221) ==== 4738+0+0 (secure 0 0 0) 0x7f670c04a640 con 0x7f671c098090 2024-04-26T04:24:39.761 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.743+0000 7f6719d9b640 1 --2- 172.21.15.63:0/1852270588 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f66fc035f40 0x7f66fc038400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:39.761 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.743+0000 7f6719d9b640 10 monclient: get_auth_request con 0x7f66fc035f40 auth_method 0 2024-04-26T04:24:39.761 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.747+0000 7f6719d9b640 1 --2- 172.21.15.63:0/1852270588 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f66fc035f40 0x7f66fc038400 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f671c06bde0 tx=0x7f6710008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:39.761 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.747+0000 7f670affd640 1 -- 172.21.15.63:0/1852270588 <== mon.2 v2:172.21.15.63:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f670c017030 con 0x7f671c098090 2024-04-26T04:24:39.762 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.747+0000 7f670affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:24:39.762 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.747+0000 7f670affd640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:24:40.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.995+0000 7f67208e8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T04:24:40.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.995+0000 7f67208e8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T04:24:40.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.995+0000 7f67208e8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:24:40.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.995+0000 7f67208e8640 1 -- 172.21.15.63:0/1852270588 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f671c08ef50 con 0x7f671c098090 2024-04-26T04:24:40.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.995+0000 7f670affd640 1 -- 172.21.15.63:0/1852270588 <== mon.2 v2:172.21.15.63:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f670c049060 con 0x7f671c098090 2024-04-26T04:24:40.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.995+0000 7f670affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T04:24:40.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.995+0000 7f670affd640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:24:40.001 INFO:teuthology.orchestra.run.smithi063.stdout:{"election_epoch":26,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cd1c6a0c-4e4b-4014-89c0-2e535479ef39","modified":"2024-04-26T04:18:03.702339Z","created":"2024-04-26T04:18:03.702339Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3300","nonce":0},{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:3300","nonce":0},{"type":"v1","addr":"172.21.15.186:6789","nonce":0}]},"addr":"172.21.15.186:6789/0","public_addr":"172.21.15.186:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3301","nonce":0},{"type":"v1","addr":"172.21.15.63:6790","nonce":0}]},"addr":"172.21.15.63:6790/0","public_addr":"172.21.15.63:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T04:24:40.003 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.999+0000 7f67208e8640 1 -- 172.21.15.63:0/1852270588 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f66fc035f40 msgr2=0x7f66fc038400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:40.003 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.999+0000 7f67208e8640 1 --2- 172.21.15.63:0/1852270588 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f66fc035f40 0x7f66fc038400 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f671c06bde0 tx=0x7f6710008040 comp rx=0 tx=0).stop 2024-04-26T04:24:40.003 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.999+0000 7f67208e8640 10 monclient: shutdown 2024-04-26T04:24:40.003 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.999+0000 7f67208e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:24:40.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.999+0000 7f67208e8640 1 -- 172.21.15.63:0/1852270588 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f671c098090 msgr2=0x7f671c1cace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:40.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.999+0000 7f67208e8640 1 --2- 172.21.15.63:0/1852270588 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f671c098090 0x7f671c1cace0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f670c00f930 tx=0x7f670c00fdf0 comp rx=0 tx=0).stop 2024-04-26T04:24:40.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.999+0000 7f67208e8640 1 -- 172.21.15.63:0/1852270588 shutdown_connections 2024-04-26T04:24:40.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.999+0000 7f67208e8640 1 --2- 172.21.15.63:0/1852270588 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f66fc035f40 0x7f66fc038400 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:40.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.999+0000 7f67208e8640 1 --2- 172.21.15.63:0/1852270588 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f671c09ca80 0x7f671c097ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:40.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.999+0000 7f67208e8640 1 --2- 172.21.15.63:0/1852270588 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f671c09c6a0 0x7f671c097590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:40.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.999+0000 7f67208e8640 1 --2- 172.21.15.63:0/1852270588 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f671c098090 0x7f671c1cace0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:40.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.999+0000 7f67208e8640 1 -- 172.21.15.63:0/1852270588 >> 172.21.15.63:0/1852270588 conn(0x7f671c0a1e70 msgr2=0x7f671c0a3e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:24:40.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.999+0000 7f67208e8640 1 -- 172.21.15.63:0/1852270588 shutdown_connections 2024-04-26T04:24:40.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:39.999+0000 7f67208e8640 1 -- 172.21.15.63:0/1852270588 wait complete. 2024-04-26T04:24:40.020 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-26T04:24:40.021 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-04-26T04:24:40.021 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-26T04:24:40.021 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T04:24:40.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.143+0000 7f9196558640 10 monclient: get_monmap_and_config 2024-04-26T04:24:40.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.143+0000 7f9196558640 10 monclient: build_initial_monmap 2024-04-26T04:24:40.147 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.143+0000 7f9196558640 10 monclient: monmap: 2024-04-26T04:24:40.147 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:24:40.147 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:40.147 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:24:40.146055+0000 2024-04-26T04:24:40.147 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:24:40.146055+0000 2024-04-26T04:24:40.147 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:24:40.147 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:40.147 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:24:40.147 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:24:40.147 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:24:40.147 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:24:40.147 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:24:40.148 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:40.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.143+0000 7f9196558640 1 Processor -- start 2024-04-26T04:24:40.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.143+0000 7f9196558640 1 -- start start 2024-04-26T04:24:40.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.143+0000 7f9196558640 10 monclient: init 2024-04-26T04:24:40.150 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.143+0000 7f9196558640 10 monclient: _reopen_session rank -1 2024-04-26T04:24:40.150 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.143+0000 7f9196558640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-04-26T04:24:40.150 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.143+0000 7f9196558640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f91900a2cd0 addr v1:172.21.15.63:6789/0 2024-04-26T04:24:40.150 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.143+0000 7f9196558640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f91900a30b0 0x7f919013c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:40.150 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.143+0000 7f9196558640 10 monclient(hunting): picked mon.noname-a con 0x7f91900a30b0 addr v2:172.21.15.63:3300/0 2024-04-26T04:24:40.150 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.143+0000 7f9196558640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f919013c540 addr v1:172.21.15.186:6789/0 2024-04-26T04:24:40.150 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.143+0000 7f9196558640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91900a4ee0 con 0x7f91900a2cd0 2024-04-26T04:24:40.150 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.143+0000 7f9196558640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91900a0fb0 con 0x7f919013c540 2024-04-26T04:24:40.151 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.143+0000 7f9196558640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:40.151 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.143+0000 7f9196558640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7f91900a5380 con 0x7f91900a30b0 2024-04-26T04:24:40.151 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.143+0000 7f9196558640 10 monclient(hunting): _renew_subs 2024-04-26T04:24:40.151 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.143+0000 7f9196558640 10 monclient(hunting): authenticate will time out at 1311.853774s 2024-04-26T04:24:40.151 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.143+0000 7f918ffff640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f91900a2cd0 0x7f91900a47d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:42408/0 (socket says 172.21.15.63:42408) 2024-04-26T04:24:40.151 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.143+0000 7f918ffff640 1 -- 172.21.15.63:0/2295455409 learned_addr learned my addr 172.21.15.63:0/2295455409 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-04-26T04:24:40.151 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.143+0000 7f918f7fe640 1 --2- 172.21.15.63:0/2295455409 >> v2:172.21.15.63:3300/0 conn(0x7f91900a30b0 0x7f919013c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:40.151 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.143+0000 7f918f7fe640 10 monclient(hunting): get_auth_request con 0x7f91900a30b0 auth_method 0 2024-04-26T04:24:40.151 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.143+0000 7f918f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:40.151 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.143+0000 7f918f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:40.151 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.143+0000 7f918f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:40.151 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.143+0000 7f918f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:40.151 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.143+0000 7f918f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:40.151 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.143+0000 7f918f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:24:40.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.143+0000 7f918f7fe640 10 monclient(hunting): handle_auth_done global_id 5364 payload 274 2024-04-26T04:24:40.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.143+0000 7f918f7fe640 1 -- 172.21.15.63:0/2295455409 >> v1:172.21.15.186:6789/0 conn(0x7f919013c540 legacy=0x7f919013e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:24:40.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.143+0000 7f918f7fe640 1 -- 172.21.15.63:0/2295455409 >> v1:172.21.15.63:6789/0 conn(0x7f91900a2cd0 legacy=0x7f91900a47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:24:40.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.143+0000 7f918f7fe640 10 monclient: _finish_hunting 0 2024-04-26T04:24:40.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.143+0000 7f918f7fe640 1 monclient: found mon.noname-a 2024-04-26T04:24:40.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.143+0000 7f918f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:24:40.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.143+0000 7f918f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:24:40.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.143+0000 7f918f7fe640 1 -- 172.21.15.63:0/2295455409 --> v2:172.21.15.63:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91900a1290 con 0x7f91900a30b0 2024-04-26T04:24:40.157 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.147+0000 7f918f7fe640 1 -- 172.21.15.63:0/2295455409 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f91900a30b0 msgr2=0x7f919013c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:24:40.157 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.147+0000 7f918f7fe640 1 --2- 172.21.15.63:0/2295455409 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f91900a30b0 0x7f919013c000 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f91780097c0 tx=0x7f917801c370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3f49114eaf318f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:40.157 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.147+0000 7f918effd640 1 -- 172.21.15.63:0/2295455409 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91780b0070 con 0x7f91900a30b0 2024-04-26T04:24:40.157 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.147+0000 7f918effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:40.157 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.147+0000 7f918effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:24:40.157 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.147+0000 7f918effd640 10 monclient: dump: 2024-04-26T04:24:40.157 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:40.157 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:40.157 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:40.157 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:40.157 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:40.157 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:40.158 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:40.158 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:40.158 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:40.158 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:40.158 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.147+0000 7f918effd640 10 monclient: _finish_auth 0 2024-04-26T04:24:40.158 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.147+0000 7f918effd640 10 monclient: _check_auth_tickets 2024-04-26T04:24:40.158 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.147+0000 7f918effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:24:40.158 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.147+0000 7f918effd640 1 -- 172.21.15.63:0/2295455409 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91780ab600 con 0x7f91900a30b0 2024-04-26T04:24:40.158 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.147+0000 7f918effd640 10 monclient: handle_config config(0 keys) 2024-04-26T04:24:40.158 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.147+0000 7f918effd640 1 -- 172.21.15.63:0/2295455409 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91780ab9d0 con 0x7f91900a30b0 2024-04-26T04:24:40.158 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.147+0000 7f9196558640 5 monclient: authenticate success, global_id 5364 2024-04-26T04:24:40.158 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.147+0000 7f9196558640 10 monclient: get_monmap_and_config success 2024-04-26T04:24:40.158 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.147+0000 7f9196558640 4 set_mon_vals no callback set 2024-04-26T04:24:40.159 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.147+0000 7f918effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:40.159 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.147+0000 7f918effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:24:40.159 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.147+0000 7f918effd640 10 monclient: dump: 2024-04-26T04:24:40.159 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:40.159 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:40.159 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:40.159 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:40.159 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:40.159 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:40.159 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:40.159 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:40.159 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:40.159 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:40.159 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.147+0000 7f9196558640 10 monclient: shutdown 2024-04-26T04:24:40.160 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.147+0000 7f9196558640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:24:40.160 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.147+0000 7f9196558640 1 -- 172.21.15.63:0/2295455409 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f91900a30b0 msgr2=0x7f919013c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:40.160 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.147+0000 7f9196558640 1 --2- 172.21.15.63:0/2295455409 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f91900a30b0 0x7f919013c000 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f91780097c0 tx=0x7f917801c370 comp rx=0 tx=0).stop 2024-04-26T04:24:40.160 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.147+0000 7f9196558640 1 -- 172.21.15.63:0/2295455409 shutdown_connections 2024-04-26T04:24:40.160 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.147+0000 7f9196558640 1 --2- 172.21.15.63:0/2295455409 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f91900a30b0 0x7f919013c000 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:40.160 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.147+0000 7f9196558640 1 -- 172.21.15.63:0/2295455409 >> 172.21.15.63:0/2295455409 conn(0x7f91900a5f40 msgr2=0x7f91900a6320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:24:40.160 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.147+0000 7f9196558640 1 -- 172.21.15.63:0/2295455409 shutdown_connections 2024-04-26T04:24:40.160 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.147+0000 7f9196558640 1 -- 172.21.15.63:0/2295455409 wait complete. 2024-04-26T04:24:40.160 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.151+0000 7f9196558640 10 monclient: build_initial_monmap 2024-04-26T04:24:40.160 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.151+0000 7f9196558640 10 monclient: monmap: 2024-04-26T04:24:40.160 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:24:40.160 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:24:40.160 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:24:40.160 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:24:40.160 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:24:40.161 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:40.161 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:24:40.161 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:24:40.161 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:24:40.161 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:40.161 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.151+0000 7f9196558640 1 Processor -- start 2024-04-26T04:24:40.161 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.151+0000 7f9196558640 1 -- start start 2024-04-26T04:24:40.161 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.151+0000 7f9196558640 10 monclient: init 2024-04-26T04:24:40.161 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.151+0000 7f9196558640 10 monclient: _reopen_session rank -1 2024-04-26T04:24:40.161 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.151+0000 7f9196558640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T04:24:40.161 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.151+0000 7f9196558640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f91900a2cd0 0x7f91901cef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:40.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.151+0000 7f9196558640 10 monclient(hunting): picked mon.noname-a con 0x7f91900a2cd0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:24:40.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.151+0000 7f9196558640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f91900a30b0 0x7f91901cf740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:40.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.151+0000 7f9196558640 10 monclient(hunting): picked mon.noname-c con 0x7f91900a30b0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:24:40.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.151+0000 7f9196558640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f919013c540 0x7f919006f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:40.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.151+0000 7f9196558640 10 monclient(hunting): picked mon.noname-b con 0x7f919013c540 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:24:40.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.151+0000 7f9196558640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:40.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.151+0000 7f9196558640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f9190142c00 con 0x7f91900a2cd0 2024-04-26T04:24:40.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.151+0000 7f9196558640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:40.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.151+0000 7f9196558640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9190142a80 con 0x7f919013c540 2024-04-26T04:24:40.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.151+0000 7f9196558640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:40.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.151+0000 7f9196558640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f9190142d80 con 0x7f91900a30b0 2024-04-26T04:24:40.162 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.151+0000 7f9196558640 10 monclient(hunting): _renew_subs 2024-04-26T04:24:40.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.151+0000 7f9196558640 10 monclient(hunting): authenticate will time out at 1311.861150s 2024-04-26T04:24:40.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.151+0000 7f918ffff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f91900a2cd0 0x7f91901cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:40.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.151+0000 7f918f7fe640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f91900a30b0 0x7f91901cf740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:40.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.151+0000 7f918ffff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f91900a2cd0 0x7f91901cef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:52956/0 (socket says 172.21.15.63:52956) 2024-04-26T04:24:40.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.151+0000 7f918ffff640 1 -- 172.21.15.63:0/1363567965 learned_addr learned my addr 172.21.15.63:0/1363567965 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:24:40.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.151+0000 7f918ffff640 10 monclient(hunting): get_auth_request con 0x7f91900a2cd0 auth_method 0 2024-04-26T04:24:40.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.151+0000 7f918ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:40.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.151+0000 7f918ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:40.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.151+0000 7f918ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:40.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.151+0000 7f918f7fe640 10 monclient(hunting): get_auth_request con 0x7f91900a30b0 auth_method 0 2024-04-26T04:24:40.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.151+0000 7f918f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:40.164 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.151+0000 7f918f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:40.164 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.151+0000 7f918f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:40.164 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.151+0000 7f918ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:40.164 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.151+0000 7f918ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:40.164 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.151+0000 7f918ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:24:40.164 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.151+0000 7f918f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:40.164 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.151+0000 7f918f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:40.164 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.151+0000 7f918f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:24:40.164 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.151+0000 7f9194af5640 1 --2- 172.21.15.63:0/1363567965 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f919013c540 0x7f919006f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:40.164 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.151+0000 7f9194af5640 10 monclient(hunting): get_auth_request con 0x7f919013c540 auth_method 0 2024-04-26T04:24:40.164 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.151+0000 7f9194af5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:40.164 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.151+0000 7f9194af5640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:40.164 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.151+0000 7f9194af5640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:40.164 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.151+0000 7f918f7fe640 10 monclient(hunting): handle_auth_done global_id 24260 payload 824 2024-04-26T04:24:40.165 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.151+0000 7f918f7fe640 1 -- 172.21.15.63:0/1363567965 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f919013c540 msgr2=0x7f919006f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:40.165 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.151+0000 7f918f7fe640 1 --2- 172.21.15.63:0/1363567965 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f919013c540 0x7f919006f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:40.165 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.151+0000 7f918f7fe640 1 -- 172.21.15.63:0/1363567965 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f91900a2cd0 msgr2=0x7f91901cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:40.165 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.151+0000 7f918f7fe640 1 --2- 172.21.15.63:0/1363567965 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f91900a2cd0 0x7f91901cef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:40.165 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.151+0000 7f918f7fe640 10 monclient: _finish_hunting 0 2024-04-26T04:24:40.165 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.151+0000 7f918f7fe640 1 monclient: found mon.noname-c 2024-04-26T04:24:40.165 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.151+0000 7f918f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:24:40.165 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.151+0000 7f918f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.63:3301/0 2024-04-26T04:24:40.165 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.151+0000 7f918f7fe640 1 -- 172.21.15.63:0/1363567965 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9190070c60 con 0x7f91900a30b0 2024-04-26T04:24:40.165 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.151+0000 7f918ffff640 1 --2- 172.21.15.63:0/1363567965 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f91900a2cd0 0x7f91901cef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T04:24:40.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.155+0000 7f918f7fe640 1 -- 172.21.15.63:0/1363567965 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f91900a30b0 msgr2=0x7f91901cf740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:24:40.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.155+0000 7f918f7fe640 1 --2- 172.21.15.63:0/1363567965 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f91900a30b0 0x7f91901cf740 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f919006c790 tx=0x7f917801abb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:40.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.155+0000 7f918cff9640 1 -- 172.21.15.63:0/1363567965 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91780b0050 con 0x7f91900a30b0 2024-04-26T04:24:40.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.155+0000 7f918cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:40.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.155+0000 7f918cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:24:40.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.155+0000 7f918cff9640 10 monclient: dump: 2024-04-26T04:24:40.169 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:40.169 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:40.169 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:40.169 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:40.169 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:40.169 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:40.170 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:40.170 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:40.170 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:40.170 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:40.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.155+0000 7f918cff9640 10 monclient: _finish_auth 0 2024-04-26T04:24:40.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.155+0000 7f918cff9640 10 monclient: _check_auth_tickets 2024-04-26T04:24:40.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.155+0000 7f918cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:24:40.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.155+0000 7f918cff9640 1 -- 172.21.15.63:0/1363567965 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91780bc070 con 0x7f91900a30b0 2024-04-26T04:24:40.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.155+0000 7f918cff9640 10 monclient: handle_config config(0 keys) 2024-04-26T04:24:40.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.155+0000 7f918cff9640 1 -- 172.21.15.63:0/1363567965 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91780b5ca0 con 0x7f91900a30b0 2024-04-26T04:24:40.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.155+0000 7f918cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:40.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.155+0000 7f918cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:24:40.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.155+0000 7f918cff9640 10 monclient: dump: 2024-04-26T04:24:40.170 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:40.171 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:40.171 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:40.171 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:40.171 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:40.171 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:40.171 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:40.171 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:40.171 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:40.171 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:40.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.155+0000 7f9196558640 5 monclient: authenticate success, global_id 24260 2024-04-26T04:24:40.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.155+0000 7f918e7fc640 4 set_mon_vals no callback set 2024-04-26T04:24:40.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.155+0000 7f9196558640 10 monclient: _renew_subs 2024-04-26T04:24:40.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.155+0000 7f9196558640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:24:40.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.155+0000 7f9196558640 1 -- 172.21.15.63:0/1363567965 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91900726b0 con 0x7f91900a30b0 2024-04-26T04:24:40.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.155+0000 7f9196558640 10 monclient: _renew_subs 2024-04-26T04:24:40.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.155+0000 7f9196558640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:24:40.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.155+0000 7f9196558640 1 -- 172.21.15.63:0/1363567965 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f91901cfc80 con 0x7f91900a30b0 2024-04-26T04:24:40.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.155+0000 7f9196558640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:24:40.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.155+0000 7f9196558640 1 --2- 172.21.15.63:0/1363567965 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f919006f540 0x7f919006f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:40.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.155+0000 7f9196558640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:40.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.155+0000 7f9196558640 1 -- 172.21.15.63:0/1363567965 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f919013c110 con 0x7f919006f540 2024-04-26T04:24:40.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.155+0000 7f918ffff640 1 --2- 172.21.15.63:0/1363567965 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f919006f540 0x7f919006f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:40.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.155+0000 7f918ffff640 10 monclient: get_auth_request con 0x7f919006f540 auth_method 0 2024-04-26T04:24:40.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.155+0000 7f918ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:40.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.155+0000 7f918ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:40.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.155+0000 7f918ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:40.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.155+0000 7f918cff9640 1 -- 172.21.15.63:0/1363567965 <== mon.2 v2:172.21.15.63:3301/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f91780ab4c0 con 0x7f91900a30b0 2024-04-26T04:24:40.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.155+0000 7f918ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:40.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.155+0000 7f918ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:40.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.155+0000 7f918ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:24:40.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.155+0000 7f918cff9640 1 --2- 172.21.15.63:0/1363567965 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f9170035ec0 0x7f9170038380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:40.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.155+0000 7f918cff9640 1 -- 172.21.15.63:0/1363567965 <== mon.2 v2:172.21.15.63:3301/0 5 ==== osd_map(222..222 src has 1..222) ==== 4738+0+0 (secure 0 0 0) 0x7f91780e3690 con 0x7f91900a30b0 2024-04-26T04:24:40.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.155+0000 7f918ffff640 10 monclient(hunting): handle_auth_done global_id 5370 payload 824 2024-04-26T04:24:40.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.155+0000 7f918ffff640 1 -- 172.21.15.63:0/1363567965 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f91900a37d0 con 0x7f919006f540 2024-04-26T04:24:40.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.155+0000 7f9194af5640 1 --2- 172.21.15.63:0/1363567965 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f9170035ec0 0x7f9170038380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:40.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.155+0000 7f9194af5640 10 monclient: get_auth_request con 0x7f9170035ec0 auth_method 0 2024-04-26T04:24:40.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.155+0000 7f918ffff640 1 -- 172.21.15.63:0/1363567965 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f919006f540 msgr2=0x7f919006f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:24:40.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.155+0000 7f918ffff640 1 --2- 172.21.15.63:0/1363567965 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f919006f540 0x7f919006f920 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f918000d3d0 tx=0x7f918000d890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:40.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.155+0000 7f918cff9640 1 -- 172.21.15.63:0/1363567965 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91800075b0 con 0x7f919006f540 2024-04-26T04:24:40.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.155+0000 7f918cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:40.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.155+0000 7f918cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:24:40.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.155+0000 7f918cff9640 10 monclient: dump: 2024-04-26T04:24:40.173 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:40.173 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:40.173 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:40.174 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:40.174 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:40.174 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:40.174 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:40.174 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:40.174 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:40.174 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:40.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.159+0000 7f918cff9640 1 -- 172.21.15.63:0/1363567965 <== mon.0 v2:172.21.15.63:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9180018070 con 0x7f919006f540 2024-04-26T04:24:40.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.159+0000 7f918cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:24:40.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.159+0000 7f918cff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:24:40.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.159+0000 7f918cff9640 1 -- 172.21.15.63:0/1363567965 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f919006f540 msgr2=0x7f919006f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:40.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.159+0000 7f918cff9640 1 --2- 172.21.15.63:0/1363567965 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f919006f540 0x7f919006f920 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f918000d3d0 tx=0x7f918000d890 comp rx=0 tx=0).stop 2024-04-26T04:24:40.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.159+0000 7f918cff9640 1 -- 172.21.15.63:0/1363567965 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f919006f540 msgr2=0x7f919006f920 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:24:40.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.159+0000 7f918cff9640 1 --2- 172.21.15.63:0/1363567965 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f919006f540 0x7f919006f920 secure :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f918000d3d0 tx=0x7f918000d890 comp rx=0 tx=0).stop 2024-04-26T04:24:40.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.159+0000 7f9194af5640 1 --2- 172.21.15.63:0/1363567965 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f9170035ec0 0x7f9170038380 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f919009f890 tx=0x7f9188009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:40.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.163+0000 7f9196558640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T04:24:40.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.163+0000 7f9196558640 1 --2- 172.21.15.63:0/1363567965 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f919006fe60 0x7f9190138dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:40.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.163+0000 7f9196558640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:40.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.163+0000 7f9196558640 1 -- 172.21.15.63:0/1363567965 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f9190062e10 con 0x7f919006fe60 2024-04-26T04:24:40.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.163+0000 7f918ffff640 1 --2- 172.21.15.63:0/1363567965 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f919006fe60 0x7f9190138dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:40.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.163+0000 7f918ffff640 10 monclient: get_auth_request con 0x7f919006fe60 auth_method 0 2024-04-26T04:24:40.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.163+0000 7f918ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:40.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.163+0000 7f918ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:40.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.163+0000 7f918ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:40.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.163+0000 7f918ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:40.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.163+0000 7f918ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:40.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.163+0000 7f918ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:24:40.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.163+0000 7f918ffff640 10 monclient(hunting): handle_auth_done global_id 5373 payload 824 2024-04-26T04:24:40.176 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.163+0000 7f918ffff640 1 -- 172.21.15.63:0/1363567965 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f919013fe30 con 0x7f919006fe60 2024-04-26T04:24:40.176 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.167+0000 7f918ffff640 1 -- 172.21.15.63:0/1363567965 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f919006fe60 msgr2=0x7f9190138dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:24:40.176 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.167+0000 7f918ffff640 1 --2- 172.21.15.63:0/1363567965 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f919006fe60 0x7f9190138dc0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f918000d3d0 tx=0x7f9180004cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:40.177 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.167+0000 7f918cff9640 1 -- 172.21.15.63:0/1363567965 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f918001de20 con 0x7f919006fe60 2024-04-26T04:24:40.177 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.167+0000 7f918cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:40.177 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.167+0000 7f918cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:24:40.177 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.167+0000 7f918cff9640 10 monclient: dump: 2024-04-26T04:24:40.177 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:40.177 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:40.177 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:40.177 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:40.177 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:40.177 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:40.177 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:40.177 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:40.177 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:40.177 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:40.177 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.167+0000 7f918cff9640 1 -- 172.21.15.63:0/1363567965 <== mon.0 v2:172.21.15.63:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4723 (secure 0 0 0) 0x7f9180013600 con 0x7f919006fe60 2024-04-26T04:24:40.178 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.167+0000 7f918cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T04:24:40.178 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.167+0000 7f918cff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:24:40.178 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.167+0000 7f918cff9640 1 -- 172.21.15.63:0/1363567965 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f919006fe60 msgr2=0x7f9190138dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:40.178 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.167+0000 7f918cff9640 1 --2- 172.21.15.63:0/1363567965 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f919006fe60 0x7f9190138dc0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f918000d3d0 tx=0x7f9180004cd0 comp rx=0 tx=0).stop 2024-04-26T04:24:40.178 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.167+0000 7f918cff9640 1 -- 172.21.15.63:0/1363567965 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f919006fe60 msgr2=0x7f9190138dc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:24:40.178 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.167+0000 7f918cff9640 1 --2- 172.21.15.63:0/1363567965 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f919006fe60 0x7f9190138dc0 secure :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f918000d3d0 tx=0x7f9180004cd0 comp rx=0 tx=0).stop 2024-04-26T04:24:40.178 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-26T04:24:40.178 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a", 2024-04-26T04:24:40.178 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 0, 2024-04-26T04:24:40.178 INFO:teuthology.orchestra.run.smithi063.stdout: "state": "leader", 2024-04-26T04:24:40.179 INFO:teuthology.orchestra.run.smithi063.stdout: "election_epoch": 26, 2024-04-26T04:24:40.179 INFO:teuthology.orchestra.run.smithi063.stdout: "uptime": 371320, 2024-04-26T04:24:40.179 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum": [ 2024-04-26T04:24:40.179 INFO:teuthology.orchestra.run.smithi063.stdout: 0, 2024-04-26T04:24:40.179 INFO:teuthology.orchestra.run.smithi063.stdout: 1, 2024-04-26T04:24:40.179 INFO:teuthology.orchestra.run.smithi063.stdout: 2 2024-04-26T04:24:40.179 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:24:40.179 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_age": 3, 2024-04-26T04:24:40.179 INFO:teuthology.orchestra.run.smithi063.stdout: "features": { 2024-04-26T04:24:40.179 INFO:teuthology.orchestra.run.smithi063.stdout: "required_con": "2450521708007866372", 2024-04-26T04:24:40.179 INFO:teuthology.orchestra.run.smithi063.stdout: "required_mon": [ 2024-04-26T04:24:40.179 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:24:40.179 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:24:40.179 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:24:40.179 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:24:40.180 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:24:40.180 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:24:40.180 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:24:40.180 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:24:40.180 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:24:40.180 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:24:40.180 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:24:40.180 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:24:40.180 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_con": "4540701547738038271", 2024-04-26T04:24:40.180 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_mon": [ 2024-04-26T04:24:40.181 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:24:40.181 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:24:40.181 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:24:40.181 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:24:40.181 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:24:40.181 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:24:40.181 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:24:40.181 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:24:40.181 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:24:40.181 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:24:40.181 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:24:40.181 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:24:40.181 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:40.181 INFO:teuthology.orchestra.run.smithi063.stdout: "outside_quorum": [], 2024-04-26T04:24:40.181 INFO:teuthology.orchestra.run.smithi063.stdout: "extra_probe_peers": [], 2024-04-26T04:24:40.182 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_provider": [], 2024-04-26T04:24:40.182 INFO:teuthology.orchestra.run.smithi063.stdout: "monmap": { 2024-04-26T04:24:40.182 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 1, 2024-04-26T04:24:40.182 INFO:teuthology.orchestra.run.smithi063.stdout: "fsid": "cd1c6a0c-4e4b-4014-89c0-2e535479ef39", 2024-04-26T04:24:40.182 INFO:teuthology.orchestra.run.smithi063.stdout: "modified": "2024-04-26T04:18:03.702339Z", 2024-04-26T04:24:40.182 INFO:teuthology.orchestra.run.smithi063.stdout: "created": "2024-04-26T04:18:03.702339Z", 2024-04-26T04:24:40.182 INFO:teuthology.orchestra.run.smithi063.stdout: "min_mon_release": 19, 2024-04-26T04:24:40.182 INFO:teuthology.orchestra.run.smithi063.stdout: "min_mon_release_name": "squid", 2024-04-26T04:24:40.182 INFO:teuthology.orchestra.run.smithi063.stdout: "election_strategy": 1, 2024-04-26T04:24:40.182 INFO:teuthology.orchestra.run.smithi063.stdout: "disallowed_leaders: ": "", 2024-04-26T04:24:40.182 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": false, 2024-04-26T04:24:40.182 INFO:teuthology.orchestra.run.smithi063.stdout: "tiebreaker_mon": "", 2024-04-26T04:24:40.182 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_ranks: ": "", 2024-04-26T04:24:40.182 INFO:teuthology.orchestra.run.smithi063.stdout: "features": { 2024-04-26T04:24:40.183 INFO:teuthology.orchestra.run.smithi063.stdout: "persistent": [ 2024-04-26T04:24:40.183 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:24:40.183 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:24:40.183 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:24:40.183 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:24:40.183 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:24:40.183 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:24:40.183 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:24:40.183 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:24:40.183 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:24:40.183 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:24:40.183 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:24:40.183 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:24:40.184 INFO:teuthology.orchestra.run.smithi063.stdout: "optional": [] 2024-04-26T04:24:40.184 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:40.184 INFO:teuthology.orchestra.run.smithi063.stdout: "mons": [ 2024-04-26T04:24:40.184 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:40.184 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 0, 2024-04-26T04:24:40.184 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a", 2024-04-26T04:24:40.184 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:24:40.184 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:24:40.184 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:40.184 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:24:40.184 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:3300", 2024-04-26T04:24:40.184 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:24:40.184 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:40.184 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:40.184 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:24:40.185 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6789", 2024-04-26T04:24:40.185 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:24:40.185 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:24:40.185 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:24:40.185 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:40.185 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6789/0", 2024-04-26T04:24:40.185 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6789/0", 2024-04-26T04:24:40.185 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:24:40.185 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:24:40.185 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:24:40.185 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:40.185 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:40.185 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 1, 2024-04-26T04:24:40.185 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b", 2024-04-26T04:24:40.185 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:24:40.186 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:24:40.186 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:40.186 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:24:40.186 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:3300", 2024-04-26T04:24:40.186 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:24:40.186 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:40.186 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:40.186 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:24:40.186 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:6789", 2024-04-26T04:24:40.186 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:24:40.186 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:24:40.186 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:24:40.186 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:40.186 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:6789/0", 2024-04-26T04:24:40.187 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.186:6789/0", 2024-04-26T04:24:40.187 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:24:40.187 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:24:40.187 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:24:40.187 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:40.187 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:40.187 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 2, 2024-04-26T04:24:40.187 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "c", 2024-04-26T04:24:40.187 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:24:40.187 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:24:40.187 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:40.187 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:24:40.187 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:3301", 2024-04-26T04:24:40.187 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:24:40.187 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:40.188 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:40.188 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:24:40.188 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6790", 2024-04-26T04:24:40.188 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:24:40.188 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:24:40.188 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:24:40.188 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:40.188 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6790/0", 2024-04-26T04:24:40.188 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6790/0", 2024-04-26T04:24:40.188 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:24:40.188 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:24:40.188 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:24:40.188 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:24:40.188 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:24:40.188 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:40.189 INFO:teuthology.orchestra.run.smithi063.stdout: "feature_map": { 2024-04-26T04:24:40.189 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": [ 2024-04-26T04:24:40.189 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:40.189 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:24:40.189 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:24:40.189 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 1 2024-04-26T04:24:40.189 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:24:40.189 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:24:40.189 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": [ 2024-04-26T04:24:40.189 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:40.189 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:24:40.189 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:24:40.189 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 3 2024-04-26T04:24:40.189 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:24:40.190 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:24:40.190 INFO:teuthology.orchestra.run.smithi063.stdout: "client": [ 2024-04-26T04:24:40.190 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:40.190 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:24:40.190 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:24:40.190 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 4 2024-04-26T04:24:40.190 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:24:40.190 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:24:40.190 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:40.190 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": false 2024-04-26T04:24:40.190 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-26T04:24:40.191 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.167+0000 7f9196558640 1 -- 172.21.15.63:0/1363567965 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f9170035ec0 msgr2=0x7f9170038380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:40.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.167+0000 7f9196558640 1 --2- 172.21.15.63:0/1363567965 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f9170035ec0 0x7f9170038380 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f919009f890 tx=0x7f9188009040 comp rx=0 tx=0).stop 2024-04-26T04:24:40.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.167+0000 7f9196558640 10 monclient: shutdown 2024-04-26T04:24:40.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.167+0000 7f9196558640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:24:40.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.167+0000 7f9196558640 1 -- 172.21.15.63:0/1363567965 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f91900a30b0 msgr2=0x7f91901cf740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:40.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.167+0000 7f9196558640 1 --2- 172.21.15.63:0/1363567965 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f91900a30b0 0x7f91901cf740 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f919006c790 tx=0x7f917801abb0 comp rx=0 tx=0).stop 2024-04-26T04:24:40.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.167+0000 7f9194af5640 1 -- 172.21.15.63:0/1363567965 reap_dead start 2024-04-26T04:24:40.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.167+0000 7f9194af5640 1 -- 172.21.15.63:0/1363567965 reap_dead start 2024-04-26T04:24:40.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.167+0000 7f9196558640 1 -- 172.21.15.63:0/1363567965 shutdown_connections 2024-04-26T04:24:40.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.167+0000 7f9196558640 1 -- 172.21.15.63:0/1363567965 >> 172.21.15.63:0/1363567965 conn(0x7f91900a5f40 msgr2=0x7f91900a1a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:24:40.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.167+0000 7f9196558640 1 -- 172.21.15.63:0/1363567965 shutdown_connections 2024-04-26T04:24:40.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.167+0000 7f9196558640 1 -- 172.21.15.63:0/1363567965 wait complete. 2024-04-26T04:24:40.193 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T04:24:40.193 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-26T04:24:40.193 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T04:24:40.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.307+0000 7ff6a8808640 10 monclient: get_monmap_and_config 2024-04-26T04:24:40.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.307+0000 7ff6a8808640 10 monclient: build_initial_monmap 2024-04-26T04:24:40.310 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.307+0000 7ff6a8808640 10 monclient: monmap: 2024-04-26T04:24:40.310 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:24:40.310 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:40.310 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:24:40.310473+0000 2024-04-26T04:24:40.311 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:24:40.310473+0000 2024-04-26T04:24:40.311 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:24:40.311 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:40.311 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:24:40.311 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:24:40.311 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:24:40.311 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:24:40.311 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:24:40.311 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:40.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.307+0000 7ff6a8808640 1 Processor -- start 2024-04-26T04:24:40.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.307+0000 7ff6a8808640 1 -- start start 2024-04-26T04:24:40.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.307+0000 7ff6a8808640 10 monclient: init 2024-04-26T04:24:40.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.307+0000 7ff6a8808640 10 monclient: _reopen_session rank -1 2024-04-26T04:24:40.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.307+0000 7ff6a8808640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-04-26T04:24:40.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.307+0000 7ff6a8808640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff6a40a2c60 0x7ff6a40a3040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:40.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.307+0000 7ff6a8808640 10 monclient(hunting): picked mon.noname-b con 0x7ff6a40a2c60 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:24:40.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.307+0000 7ff6a8808640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7ff6a40a20c0 0x7ff6a40a3db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:40.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.307+0000 7ff6a8808640 10 monclient(hunting): picked mon.noname-a con 0x7ff6a40a20c0 addr v2:172.21.15.63:3300/0 2024-04-26T04:24:40.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.307+0000 7ff6a8808640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff6a40a24a0 0x7ff6a40a42f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:40.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.307+0000 7ff6a8808640 10 monclient(hunting): picked mon.noname-c con 0x7ff6a40a24a0 addr v2:172.21.15.186:3300/0 2024-04-26T04:24:40.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.307+0000 7ff6a8808640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:40.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.307+0000 7ff6a8808640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7ff6a40a50a0 con 0x7ff6a40a20c0 2024-04-26T04:24:40.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.307+0000 7ff6a8808640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:40.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.307+0000 7ff6a8808640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7ff6a40a0df0 con 0x7ff6a40a24a0 2024-04-26T04:24:40.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.307+0000 7ff6a8808640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:40.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.307+0000 7ff6a8808640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7ff6a40a13b0 con 0x7ff6a40a2c60 2024-04-26T04:24:40.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.307+0000 7ff6a8808640 10 monclient(hunting): _renew_subs 2024-04-26T04:24:40.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.307+0000 7ff6a8808640 10 monclient(hunting): authenticate will time out at 1312.018390s 2024-04-26T04:24:40.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.307+0000 7ff6a259c640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff6a40a2c60 0x7ff6a40a3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:40.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.307+0000 7ff6a1d9b640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7ff6a40a20c0 0x7ff6a40a3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:40.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.307+0000 7ff6a259c640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff6a40a2c60 0x7ff6a40a3040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:59262/0 (socket says 172.21.15.63:59262) 2024-04-26T04:24:40.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.307+0000 7ff6a259c640 1 -- 172.21.15.63:0/427933070 learned_addr learned my addr 172.21.15.63:0/427933070 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:24:40.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.307+0000 7ff6a259c640 10 monclient(hunting): get_auth_request con 0x7ff6a40a2c60 auth_method 0 2024-04-26T04:24:40.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.307+0000 7ff6a259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:40.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.307+0000 7ff6a259c640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:40.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.307+0000 7ff6a259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:40.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.307+0000 7ff6a1d9b640 10 monclient(hunting): get_auth_request con 0x7ff6a40a20c0 auth_method 0 2024-04-26T04:24:40.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.307+0000 7ff6a1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:40.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.307+0000 7ff6a1d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:40.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.307+0000 7ff6a1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:40.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.307+0000 7ff6a259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:40.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.307+0000 7ff6a259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:40.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.307+0000 7ff6a259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:24:40.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.307+0000 7ff6a1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:40.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.307+0000 7ff6a1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:40.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.307+0000 7ff6a1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:24:40.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.307+0000 7ff6a2d9d640 1 --2- 172.21.15.63:0/427933070 >> v2:172.21.15.186:3300/0 conn(0x7ff6a40a24a0 0x7ff6a40a42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:40.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.307+0000 7ff6a2d9d640 10 monclient(hunting): get_auth_request con 0x7ff6a40a24a0 auth_method 0 2024-04-26T04:24:40.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.307+0000 7ff6a2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:40.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.307+0000 7ff6a2d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:40.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.307+0000 7ff6a2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:40.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.307+0000 7ff6a1d9b640 10 monclient(hunting): handle_auth_done global_id 5376 payload 274 2024-04-26T04:24:40.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.307+0000 7ff6a1d9b640 1 -- 172.21.15.63:0/427933070 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff6a40a2c60 msgr2=0x7ff6a40a3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:24:40.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.307+0000 7ff6a1d9b640 1 --2- 172.21.15.63:0/427933070 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff6a40a2c60 0x7ff6a40a3040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:40.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.307+0000 7ff6a1d9b640 1 -- 172.21.15.63:0/427933070 >> v2:172.21.15.186:3300/0 conn(0x7ff6a40a24a0 msgr2=0x7ff6a40a42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:24:40.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.307+0000 7ff6a1d9b640 1 --2- 172.21.15.63:0/427933070 >> v2:172.21.15.186:3300/0 conn(0x7ff6a40a24a0 0x7ff6a40a42f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:40.318 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.307+0000 7ff6a1d9b640 10 monclient: _finish_hunting 0 2024-04-26T04:24:40.318 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.307+0000 7ff6a1d9b640 1 monclient: found mon.noname-a 2024-04-26T04:24:40.318 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.307+0000 7ff6a1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:24:40.318 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.307+0000 7ff6a1d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:24:40.318 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.307+0000 7ff6a1d9b640 1 -- 172.21.15.63:0/427933070 --> v2:172.21.15.63:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6a40a10d0 con 0x7ff6a40a20c0 2024-04-26T04:24:40.318 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.307+0000 7ff6a259c640 1 --2- 172.21.15.63:0/427933070 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff6a40a2c60 0x7ff6a40a3040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T04:24:40.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.311+0000 7ff6a1d9b640 1 -- 172.21.15.63:0/427933070 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff6a40a20c0 msgr2=0x7ff6a40a3db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:24:40.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.311+0000 7ff6a1d9b640 1 --2- 172.21.15.63:0/427933070 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff6a40a20c0 0x7ff6a40a3db0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7ff6a406da60 tx=0x7ff69801c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f13414db5fde734 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:40.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.311+0000 7ff6a159a640 1 -- 172.21.15.63:0/427933070 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6980b2070 con 0x7ff6a40a20c0 2024-04-26T04:24:40.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.311+0000 7ff6a159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:40.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.311+0000 7ff6a159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:24:40.325 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.311+0000 7ff6a159a640 10 monclient: dump: 2024-04-26T04:24:40.325 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:40.325 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:40.325 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:40.325 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:40.325 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:40.325 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:40.325 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:40.325 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:40.325 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:40.325 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:40.325 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.311+0000 7ff6a159a640 10 monclient: _finish_auth 0 2024-04-26T04:24:40.326 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.311+0000 7ff6a159a640 10 monclient: _check_auth_tickets 2024-04-26T04:24:40.326 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.311+0000 7ff6a159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:24:40.326 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.311+0000 7ff6a159a640 1 -- 172.21.15.63:0/427933070 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6980a5780 con 0x7ff6a40a20c0 2024-04-26T04:24:40.326 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.311+0000 7ff6a8808640 5 monclient: authenticate success, global_id 5376 2024-04-26T04:24:40.326 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.311+0000 7ff6a8808640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T04:24:40.326 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.311+0000 7ff6a159a640 10 monclient: handle_config config(0 keys) 2024-04-26T04:24:40.326 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.311+0000 7ff6a8808640 10 monclient: get_monmap_and_config success 2024-04-26T04:24:40.326 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.311+0000 7ff6a8808640 4 set_mon_vals no callback set 2024-04-26T04:24:40.326 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.311+0000 7ff6a159a640 1 -- 172.21.15.63:0/427933070 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6980a5b50 con 0x7ff6a40a20c0 2024-04-26T04:24:40.326 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.311+0000 7ff6a159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:40.326 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.311+0000 7ff6a159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:24:40.326 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.311+0000 7ff6a159a640 10 monclient: dump: 2024-04-26T04:24:40.326 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:40.326 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:40.327 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:40.327 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:40.327 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:40.327 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:40.327 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:40.327 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:40.327 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:40.327 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:40.327 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.315+0000 7ff6a8808640 10 monclient: shutdown 2024-04-26T04:24:40.327 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.315+0000 7ff6a8808640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:24:40.327 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.315+0000 7ff6a8808640 1 -- 172.21.15.63:0/427933070 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff6a40a20c0 msgr2=0x7ff6a40a3db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:40.327 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.315+0000 7ff6a8808640 1 --2- 172.21.15.63:0/427933070 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff6a40a20c0 0x7ff6a40a3db0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7ff6a406da60 tx=0x7ff69801c7f0 comp rx=0 tx=0).stop 2024-04-26T04:24:40.327 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.315+0000 7ff6a8808640 1 -- 172.21.15.63:0/427933070 shutdown_connections 2024-04-26T04:24:40.327 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.315+0000 7ff6a8808640 1 --2- 172.21.15.63:0/427933070 >> v2:172.21.15.186:3300/0 conn(0x7ff6a40a24a0 0x7ff6a40a42f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:40.328 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.315+0000 7ff6a8808640 1 --2- 172.21.15.63:0/427933070 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff6a40a20c0 0x7ff6a40a3db0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:40.328 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.315+0000 7ff6a8808640 1 --2- 172.21.15.63:0/427933070 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff6a40a2c60 0x7ff6a40a3040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:40.328 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.315+0000 7ff6a8808640 1 -- 172.21.15.63:0/427933070 >> 172.21.15.63:0/427933070 conn(0x7ff6a40a5fb0 msgr2=0x7ff6a40a63c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:24:40.328 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.315+0000 7ff6a8808640 1 -- 172.21.15.63:0/427933070 shutdown_connections 2024-04-26T04:24:40.328 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.315+0000 7ff6a8808640 1 -- 172.21.15.63:0/427933070 wait complete. 2024-04-26T04:24:40.328 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.315+0000 7ff6a8808640 10 monclient: build_initial_monmap 2024-04-26T04:24:40.328 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.315+0000 7ff6a8808640 10 monclient: monmap: 2024-04-26T04:24:40.328 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:24:40.328 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:24:40.328 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:24:40.328 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:24:40.328 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:24:40.328 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:40.328 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:24:40.329 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:24:40.329 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:24:40.329 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:40.329 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.315+0000 7ff6a8808640 1 Processor -- start 2024-04-26T04:24:40.329 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.315+0000 7ff6a8808640 1 -- start start 2024-04-26T04:24:40.329 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.315+0000 7ff6a8808640 10 monclient: init 2024-04-26T04:24:40.329 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.315+0000 7ff6a8808640 10 monclient: _reopen_session rank -1 2024-04-26T04:24:40.329 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.315+0000 7ff6a8808640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T04:24:40.329 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.315+0000 7ff6a8808640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff6a40a20c0 0x7ff6a413cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:40.329 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.315+0000 7ff6a8808640 10 monclient(hunting): picked mon.noname-a con 0x7ff6a40a20c0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:24:40.329 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.315+0000 7ff6a8808640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff6a40a24a0 0x7ff6a4124a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:40.329 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.315+0000 7ff6a8808640 10 monclient(hunting): picked mon.noname-c con 0x7ff6a40a24a0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:24:40.329 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.315+0000 7ff6a8808640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6a40a2c60 0x7ff6a4124f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:40.330 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.315+0000 7ff6a8808640 10 monclient(hunting): picked mon.noname-b con 0x7ff6a40a2c60 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:24:40.330 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.315+0000 7ff6a8808640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:40.330 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.315+0000 7ff6a8808640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff6a4140d60 con 0x7ff6a40a20c0 2024-04-26T04:24:40.330 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.315+0000 7ff6a8808640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:40.330 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.315+0000 7ff6a8808640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff6a4140be0 con 0x7ff6a40a2c60 2024-04-26T04:24:40.330 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.315+0000 7ff6a8808640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:40.330 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.315+0000 7ff6a8808640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7ff6a4140ee0 con 0x7ff6a40a24a0 2024-04-26T04:24:40.330 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.315+0000 7ff6a8808640 10 monclient(hunting): _renew_subs 2024-04-26T04:24:40.330 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.315+0000 7ff6a8808640 10 monclient(hunting): authenticate will time out at 1312.025738s 2024-04-26T04:24:40.330 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.315+0000 7ff6a259c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff6a40a20c0 0x7ff6a413cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:40.330 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.315+0000 7ff6a1d9b640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff6a40a24a0 0x7ff6a4124a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:40.330 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.315+0000 7ff6a259c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff6a40a20c0 0x7ff6a413cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:53000/0 (socket says 172.21.15.63:53000) 2024-04-26T04:24:40.330 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.315+0000 7ff6a1d9b640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff6a40a24a0 0x7ff6a4124a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:59266/0 (socket says 172.21.15.63:59266) 2024-04-26T04:24:40.331 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.315+0000 7ff6a259c640 1 -- 172.21.15.63:0/3081234897 learned_addr learned my addr 172.21.15.63:0/3081234897 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:24:40.331 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.315+0000 7ff6a259c640 10 monclient(hunting): get_auth_request con 0x7ff6a40a20c0 auth_method 0 2024-04-26T04:24:40.331 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.315+0000 7ff6a259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:40.331 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.315+0000 7ff6a259c640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:40.331 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.315+0000 7ff6a259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:40.331 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.315+0000 7ff6a1d9b640 10 monclient(hunting): get_auth_request con 0x7ff6a40a24a0 auth_method 0 2024-04-26T04:24:40.331 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.315+0000 7ff6a1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:40.331 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.315+0000 7ff6a1d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:40.331 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.315+0000 7ff6a1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:40.331 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.315+0000 7ff6a259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:40.331 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.315+0000 7ff6a259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:40.331 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.315+0000 7ff6a259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:24:40.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.315+0000 7ff6a1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:40.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.315+0000 7ff6a1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:40.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.315+0000 7ff6a1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:24:40.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.315+0000 7ff6a2d9d640 1 --2- 172.21.15.63:0/3081234897 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6a40a2c60 0x7ff6a4124f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:40.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.315+0000 7ff6a2d9d640 10 monclient(hunting): get_auth_request con 0x7ff6a40a2c60 auth_method 0 2024-04-26T04:24:40.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.315+0000 7ff6a2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:40.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.315+0000 7ff6a2d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:40.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.315+0000 7ff6a2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:40.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.315+0000 7ff6a259c640 10 monclient(hunting): handle_auth_done global_id 5379 payload 824 2024-04-26T04:24:40.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.315+0000 7ff6a259c640 1 -- 172.21.15.63:0/3081234897 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff6a40a24a0 msgr2=0x7ff6a4124a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:40.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.315+0000 7ff6a259c640 1 --2- 172.21.15.63:0/3081234897 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff6a40a24a0 0x7ff6a4124a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:40.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.315+0000 7ff6a259c640 1 -- 172.21.15.63:0/3081234897 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6a40a2c60 msgr2=0x7ff6a4124f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:40.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.315+0000 7ff6a259c640 1 --2- 172.21.15.63:0/3081234897 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6a40a2c60 0x7ff6a4124f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:40.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.315+0000 7ff6a259c640 10 monclient: _finish_hunting 0 2024-04-26T04:24:40.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.315+0000 7ff6a259c640 1 monclient: found mon.noname-a 2024-04-26T04:24:40.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.315+0000 7ff6a259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:24:40.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.315+0000 7ff6a259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:24:40.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.315+0000 7ff6a259c640 1 -- 172.21.15.63:0/3081234897 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6a41286b0 con 0x7ff6a40a20c0 2024-04-26T04:24:40.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.315+0000 7ff6a1d9b640 1 --2- 172.21.15.63:0/3081234897 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff6a40a24a0 0x7ff6a4124a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T04:24:40.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.319+0000 7ff6a259c640 1 -- 172.21.15.63:0/3081234897 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff6a40a20c0 msgr2=0x7ff6a413cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:24:40.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.319+0000 7ff6a259c640 1 --2- 172.21.15.63:0/3081234897 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff6a40a20c0 0x7ff6a413cfd0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7ff69400e9d0 tx=0x7ff69400ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:40.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.319+0000 7ff692ffd640 1 -- 172.21.15.63:0/3081234897 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff694017070 con 0x7ff6a40a20c0 2024-04-26T04:24:40.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.319+0000 7ff692ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:40.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.319+0000 7ff692ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:24:40.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.319+0000 7ff692ffd640 10 monclient: dump: 2024-04-26T04:24:40.333 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:40.334 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:40.334 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:40.334 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:40.334 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:40.334 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:40.334 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:40.334 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:40.334 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:40.334 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:40.334 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.319+0000 7ff692ffd640 10 monclient: _finish_auth 0 2024-04-26T04:24:40.334 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.319+0000 7ff692ffd640 10 monclient: _check_auth_tickets 2024-04-26T04:24:40.334 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.319+0000 7ff692ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:24:40.334 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.319+0000 7ff692ffd640 1 -- 172.21.15.63:0/3081234897 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff69400cc10 con 0x7ff6a40a20c0 2024-04-26T04:24:40.335 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.319+0000 7ff692ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T04:24:40.335 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.319+0000 7ff692ffd640 1 -- 172.21.15.63:0/3081234897 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff694013470 con 0x7ff6a40a20c0 2024-04-26T04:24:40.335 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.319+0000 7ff6a8808640 5 monclient: authenticate success, global_id 5379 2024-04-26T04:24:40.335 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.319+0000 7ff6a8808640 10 monclient: _renew_subs 2024-04-26T04:24:40.335 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.319+0000 7ff6a8808640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:24:40.335 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.319+0000 7ff6a8808640 1 -- 172.21.15.63:0/3081234897 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6a4128990 con 0x7ff6a40a20c0 2024-04-26T04:24:40.335 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.319+0000 7ff693fff640 4 set_mon_vals no callback set 2024-04-26T04:24:40.335 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.319+0000 7ff692ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:40.335 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.319+0000 7ff692ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:24:40.335 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.319+0000 7ff692ffd640 10 monclient: dump: 2024-04-26T04:24:40.335 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:40.335 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:40.335 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:40.336 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:40.336 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:40.336 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:40.336 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:40.336 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:40.336 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:40.336 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:40.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.319+0000 7ff6a8808640 10 monclient: _renew_subs 2024-04-26T04:24:40.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.319+0000 7ff6a8808640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:24:40.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.319+0000 7ff6a8808640 1 -- 172.21.15.63:0/3081234897 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6a41280f0 con 0x7ff6a40a20c0 2024-04-26T04:24:40.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.319+0000 7ff6a8808640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:24:40.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.319+0000 7ff6a8808640 1 --2- 172.21.15.63:0/3081234897 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff6a41c5e00 0x7ff6a402bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:40.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.319+0000 7ff6a8808640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:40.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.319+0000 7ff6a8808640 1 -- 172.21.15.63:0/3081234897 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7ff6a41cfd90 con 0x7ff6a41c5e00 2024-04-26T04:24:40.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.319+0000 7ff6a1d9b640 1 --2- 172.21.15.63:0/3081234897 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff6a41c5e00 0x7ff6a402bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:40.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.319+0000 7ff692ffd640 1 -- 172.21.15.63:0/3081234897 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7ff69401a070 con 0x7ff6a40a20c0 2024-04-26T04:24:40.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.319+0000 7ff6a1d9b640 10 monclient: get_auth_request con 0x7ff6a41c5e00 auth_method 0 2024-04-26T04:24:40.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.319+0000 7ff6a1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:40.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.319+0000 7ff6a1d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:40.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.319+0000 7ff6a1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:40.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.319+0000 7ff6a1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:40.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.319+0000 7ff6a1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:40.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.319+0000 7ff6a1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:24:40.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.319+0000 7ff692ffd640 1 --2- 172.21.15.63:0/3081234897 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7ff684035f10 0x7ff6840383d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:40.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.319+0000 7ff6a1d9b640 10 monclient(hunting): handle_auth_done global_id 24269 payload 824 2024-04-26T04:24:40.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.319+0000 7ff692ffd640 1 -- 172.21.15.63:0/3081234897 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(222..222 src has 1..222) ==== 4738+0+0 (secure 0 0 0) 0x7ff694048da0 con 0x7ff6a40a20c0 2024-04-26T04:24:40.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.319+0000 7ff6a1d9b640 1 -- 172.21.15.63:0/3081234897 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff6a4062e10 con 0x7ff6a41c5e00 2024-04-26T04:24:40.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.319+0000 7ff6a2d9d640 1 --2- 172.21.15.63:0/3081234897 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7ff684035f10 0x7ff6840383d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:40.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.319+0000 7ff6a2d9d640 10 monclient: get_auth_request con 0x7ff684035f10 auth_method 0 2024-04-26T04:24:40.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.323+0000 7ff6a1d9b640 1 -- 172.21.15.63:0/3081234897 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff6a41c5e00 msgr2=0x7ff6a402bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:24:40.341 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.323+0000 7ff6a1d9b640 1 --2- 172.21.15.63:0/3081234897 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff6a41c5e00 0x7ff6a402bfc0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff6980af000 tx=0x7ff698019590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:40.341 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.323+0000 7ff692ffd640 1 -- 172.21.15.63:0/3081234897 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6980b2040 con 0x7ff6a41c5e00 2024-04-26T04:24:40.341 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.323+0000 7ff692ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:40.341 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.323+0000 7ff692ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:24:40.341 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.323+0000 7ff692ffd640 10 monclient: dump: 2024-04-26T04:24:40.341 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:40.341 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:40.341 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:40.341 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:40.341 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:40.341 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:40.342 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:40.342 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:40.342 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:40.342 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:40.342 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.323+0000 7ff692ffd640 1 -- 172.21.15.63:0/3081234897 <== mon.2 v2:172.21.15.63:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff6980b9870 con 0x7ff6a41c5e00 2024-04-26T04:24:40.342 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.323+0000 7ff692ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:24:40.342 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.323+0000 7ff692ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:24:40.342 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.323+0000 7ff692ffd640 1 -- 172.21.15.63:0/3081234897 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff6a41c5e00 msgr2=0x7ff6a402bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:40.342 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.323+0000 7ff692ffd640 1 --2- 172.21.15.63:0/3081234897 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff6a41c5e00 0x7ff6a402bfc0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff6980af000 tx=0x7ff698019590 comp rx=0 tx=0).stop 2024-04-26T04:24:40.342 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.323+0000 7ff692ffd640 1 -- 172.21.15.63:0/3081234897 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff6a41c5e00 msgr2=0x7ff6a402bfc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:24:40.342 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.323+0000 7ff692ffd640 1 --2- 172.21.15.63:0/3081234897 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff6a41c5e00 0x7ff6a402bfc0 secure :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff6980af000 tx=0x7ff698019590 comp rx=0 tx=0).stop 2024-04-26T04:24:40.342 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.323+0000 7ff6a2d9d640 1 --2- 172.21.15.63:0/3081234897 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7ff684035f10 0x7ff6840383d0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ff6a4128e60 tx=0x7ff69c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:40.342 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.331+0000 7ff6a8808640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T04:24:40.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.331+0000 7ff6a8808640 1 --2- 172.21.15.63:0/3081234897 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff6a402c500 0x7ff6a402c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:40.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.331+0000 7ff6a8808640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:40.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.331+0000 7ff6a8808640 1 -- 172.21.15.63:0/3081234897 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7ff6a4121210 con 0x7ff6a402c500 2024-04-26T04:24:40.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.331+0000 7ff6a1d9b640 1 --2- 172.21.15.63:0/3081234897 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff6a402c500 0x7ff6a402c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:40.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.331+0000 7ff6a1d9b640 10 monclient: get_auth_request con 0x7ff6a402c500 auth_method 0 2024-04-26T04:24:40.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.331+0000 7ff6a1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:40.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.331+0000 7ff6a1d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:40.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.331+0000 7ff6a1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:40.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.331+0000 7ff6a1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:40.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.331+0000 7ff6a1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:40.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.331+0000 7ff6a1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:24:40.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.331+0000 7ff6a1d9b640 10 monclient(hunting): handle_auth_done global_id 24272 payload 824 2024-04-26T04:24:40.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.331+0000 7ff6a1d9b640 1 -- 172.21.15.63:0/3081234897 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff6a40683f0 con 0x7ff6a402c500 2024-04-26T04:24:40.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.335+0000 7ff6a1d9b640 1 -- 172.21.15.63:0/3081234897 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff6a402c500 msgr2=0x7ff6a402c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:24:40.346 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.335+0000 7ff6a1d9b640 1 --2- 172.21.15.63:0/3081234897 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff6a402c500 0x7ff6a402c910 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff69801aea0 tx=0x7ff6980a7000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:40.346 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.335+0000 7ff692ffd640 1 -- 172.21.15.63:0/3081234897 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6980b2070 con 0x7ff6a402c500 2024-04-26T04:24:40.346 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.335+0000 7ff692ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:40.346 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.335+0000 7ff692ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:24:40.346 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.335+0000 7ff692ffd640 10 monclient: dump: 2024-04-26T04:24:40.346 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:40.347 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:40.347 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:40.347 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:40.347 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:40.347 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:40.347 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:40.347 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:40.347 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:40.348 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:40.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.335+0000 7ff692ffd640 1 -- 172.21.15.63:0/3081234897 <== mon.2 v2:172.21.15.63:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4721 (secure 0 0 0) 0x7ff6980c9070 con 0x7ff6a402c500 2024-04-26T04:24:40.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.335+0000 7ff692ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T04:24:40.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.335+0000 7ff692ffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:24:40.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.335+0000 7ff692ffd640 1 -- 172.21.15.63:0/3081234897 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff6a402c500 msgr2=0x7ff6a402c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:40.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.335+0000 7ff692ffd640 1 --2- 172.21.15.63:0/3081234897 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff6a402c500 0x7ff6a402c910 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff69801aea0 tx=0x7ff6980a7000 comp rx=0 tx=0).stop 2024-04-26T04:24:40.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.335+0000 7ff692ffd640 1 -- 172.21.15.63:0/3081234897 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff6a402c500 msgr2=0x7ff6a402c910 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:24:40.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.335+0000 7ff692ffd640 1 --2- 172.21.15.63:0/3081234897 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff6a402c500 0x7ff6a402c910 secure :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff69801aea0 tx=0x7ff6980a7000 comp rx=0 tx=0).stop 2024-04-26T04:24:40.349 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-26T04:24:40.349 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "c", 2024-04-26T04:24:40.349 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 2, 2024-04-26T04:24:40.349 INFO:teuthology.orchestra.run.smithi063.stdout: "state": "peon", 2024-04-26T04:24:40.349 INFO:teuthology.orchestra.run.smithi063.stdout: "election_epoch": 26, 2024-04-26T04:24:40.349 INFO:teuthology.orchestra.run.smithi063.stdout: "uptime": 127934, 2024-04-26T04:24:40.350 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum": [ 2024-04-26T04:24:40.350 INFO:teuthology.orchestra.run.smithi063.stdout: 0, 2024-04-26T04:24:40.350 INFO:teuthology.orchestra.run.smithi063.stdout: 1, 2024-04-26T04:24:40.350 INFO:teuthology.orchestra.run.smithi063.stdout: 2 2024-04-26T04:24:40.350 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:24:40.350 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_age": 3, 2024-04-26T04:24:40.351 INFO:teuthology.orchestra.run.smithi063.stdout: "features": { 2024-04-26T04:24:40.351 INFO:teuthology.orchestra.run.smithi063.stdout: "required_con": "2450521708007866372", 2024-04-26T04:24:40.351 INFO:teuthology.orchestra.run.smithi063.stdout: "required_mon": [ 2024-04-26T04:24:40.351 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:24:40.351 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:24:40.351 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:24:40.351 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:24:40.351 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:24:40.351 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:24:40.351 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:24:40.351 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:24:40.351 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:24:40.352 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:24:40.352 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:24:40.352 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:24:40.353 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_con": "4540701547738038271", 2024-04-26T04:24:40.353 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_mon": [ 2024-04-26T04:24:40.353 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:24:40.353 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:24:40.353 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:24:40.353 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:24:40.353 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:24:40.353 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:24:40.353 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:24:40.353 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:24:40.353 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:24:40.353 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:24:40.353 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:24:40.353 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:24:40.353 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:40.354 INFO:teuthology.orchestra.run.smithi063.stdout: "outside_quorum": [], 2024-04-26T04:24:40.354 INFO:teuthology.orchestra.run.smithi063.stdout: "extra_probe_peers": [], 2024-04-26T04:24:40.354 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_provider": [], 2024-04-26T04:24:40.354 INFO:teuthology.orchestra.run.smithi063.stdout: "monmap": { 2024-04-26T04:24:40.354 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 1, 2024-04-26T04:24:40.354 INFO:teuthology.orchestra.run.smithi063.stdout: "fsid": "cd1c6a0c-4e4b-4014-89c0-2e535479ef39", 2024-04-26T04:24:40.354 INFO:teuthology.orchestra.run.smithi063.stdout: "modified": "2024-04-26T04:18:03.702339Z", 2024-04-26T04:24:40.354 INFO:teuthology.orchestra.run.smithi063.stdout: "created": "2024-04-26T04:18:03.702339Z", 2024-04-26T04:24:40.354 INFO:teuthology.orchestra.run.smithi063.stdout: "min_mon_release": 19, 2024-04-26T04:24:40.354 INFO:teuthology.orchestra.run.smithi063.stdout: "min_mon_release_name": "squid", 2024-04-26T04:24:40.354 INFO:teuthology.orchestra.run.smithi063.stdout: "election_strategy": 1, 2024-04-26T04:24:40.354 INFO:teuthology.orchestra.run.smithi063.stdout: "disallowed_leaders: ": "", 2024-04-26T04:24:40.354 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": false, 2024-04-26T04:24:40.354 INFO:teuthology.orchestra.run.smithi063.stdout: "tiebreaker_mon": "", 2024-04-26T04:24:40.355 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_ranks: ": "", 2024-04-26T04:24:40.355 INFO:teuthology.orchestra.run.smithi063.stdout: "features": { 2024-04-26T04:24:40.355 INFO:teuthology.orchestra.run.smithi063.stdout: "persistent": [ 2024-04-26T04:24:40.355 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:24:40.355 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:24:40.355 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:24:40.355 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:24:40.355 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:24:40.355 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:24:40.355 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:24:40.355 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:24:40.355 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:24:40.355 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:24:40.355 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:24:40.355 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:24:40.356 INFO:teuthology.orchestra.run.smithi063.stdout: "optional": [] 2024-04-26T04:24:40.356 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:40.356 INFO:teuthology.orchestra.run.smithi063.stdout: "mons": [ 2024-04-26T04:24:40.356 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:40.356 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 0, 2024-04-26T04:24:40.356 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a", 2024-04-26T04:24:40.356 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:24:40.356 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:24:40.356 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:40.356 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:24:40.356 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:3300", 2024-04-26T04:24:40.356 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:24:40.356 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:40.356 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:40.356 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:24:40.357 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6789", 2024-04-26T04:24:40.357 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:24:40.357 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:24:40.357 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:24:40.357 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:40.357 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6789/0", 2024-04-26T04:24:40.357 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6789/0", 2024-04-26T04:24:40.357 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:24:40.357 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:24:40.357 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:24:40.357 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:40.357 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:40.357 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 1, 2024-04-26T04:24:40.357 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b", 2024-04-26T04:24:40.357 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:24:40.358 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:24:40.358 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:40.358 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:24:40.358 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:3300", 2024-04-26T04:24:40.358 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:24:40.358 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:40.358 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:40.358 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:24:40.358 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:6789", 2024-04-26T04:24:40.358 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:24:40.358 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:24:40.358 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:24:40.358 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:40.358 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:6789/0", 2024-04-26T04:24:40.359 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.186:6789/0", 2024-04-26T04:24:40.359 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:24:40.359 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:24:40.359 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:24:40.359 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:40.359 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:40.359 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 2, 2024-04-26T04:24:40.359 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "c", 2024-04-26T04:24:40.359 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:24:40.359 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:24:40.359 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:40.359 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:24:40.359 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:3301", 2024-04-26T04:24:40.359 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:24:40.359 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:40.360 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:40.360 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:24:40.360 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6790", 2024-04-26T04:24:40.360 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:24:40.360 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:24:40.360 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:24:40.360 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:40.360 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6790/0", 2024-04-26T04:24:40.360 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6790/0", 2024-04-26T04:24:40.360 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:24:40.360 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:24:40.360 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:24:40.360 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:24:40.360 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:24:40.360 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:40.361 INFO:teuthology.orchestra.run.smithi063.stdout: "feature_map": { 2024-04-26T04:24:40.361 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": [ 2024-04-26T04:24:40.361 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:40.361 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:24:40.361 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:24:40.361 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 1 2024-04-26T04:24:40.361 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:24:40.361 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:24:40.361 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": [ 2024-04-26T04:24:40.361 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:40.361 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:24:40.361 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:24:40.361 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 3 2024-04-26T04:24:40.361 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:24:40.361 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:24:40.362 INFO:teuthology.orchestra.run.smithi063.stdout: "client": [ 2024-04-26T04:24:40.362 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:40.362 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:24:40.362 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:24:40.362 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 3 2024-04-26T04:24:40.362 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:24:40.362 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:24:40.362 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:40.362 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": false 2024-04-26T04:24:40.362 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-26T04:24:40.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.335+0000 7ff6a8808640 1 -- 172.21.15.63:0/3081234897 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7ff684035f10 msgr2=0x7ff6840383d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:40.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.335+0000 7ff6a8808640 1 --2- 172.21.15.63:0/3081234897 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7ff684035f10 0x7ff6840383d0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ff6a4128e60 tx=0x7ff69c008040 comp rx=0 tx=0).stop 2024-04-26T04:24:40.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.335+0000 7ff6a8808640 10 monclient: shutdown 2024-04-26T04:24:40.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.335+0000 7ff6a8808640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:24:40.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.335+0000 7ff6a8808640 1 -- 172.21.15.63:0/3081234897 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff6a40a20c0 msgr2=0x7ff6a413cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:40.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.335+0000 7ff6a8808640 1 --2- 172.21.15.63:0/3081234897 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff6a40a20c0 0x7ff6a413cfd0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7ff69400e9d0 tx=0x7ff69400ee90 comp rx=0 tx=0).stop 2024-04-26T04:24:40.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.335+0000 7ff6a2d9d640 1 -- 172.21.15.63:0/3081234897 reap_dead start 2024-04-26T04:24:40.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.335+0000 7ff6a2d9d640 1 -- 172.21.15.63:0/3081234897 reap_dead start 2024-04-26T04:24:40.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.335+0000 7ff6a8808640 1 -- 172.21.15.63:0/3081234897 shutdown_connections 2024-04-26T04:24:40.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.335+0000 7ff6a8808640 1 -- 172.21.15.63:0/3081234897 >> 172.21.15.63:0/3081234897 conn(0x7ff6a40a5fb0 msgr2=0x7ff6a40710d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:24:40.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.335+0000 7ff6a8808640 1 -- 172.21.15.63:0/3081234897 shutdown_connections 2024-04-26T04:24:40.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.335+0000 7ff6a8808640 1 -- 172.21.15.63:0/3081234897 wait complete. 2024-04-26T04:24:40.365 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T04:24:40.365 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-26T04:24:40.365 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T04:24:40.423 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[30635]: starting. 2024-04-26T04:24:40.423 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[30635]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-30634 2024-04-26T04:24:40.423 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[30635]: created object 0... 2024-04-26T04:24:40.423 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[30635]: created object 25... 2024-04-26T04:24:40.423 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[30635]: created object 49... 2024-04-26T04:24:40.423 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[30635]: finishing. 2024-04-26T04:24:40.423 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[30635]: shutting down. 2024-04-26T04:24:40.482 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.479+0000 7f09d0fcc640 10 monclient: get_monmap_and_config 2024-04-26T04:24:40.482 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.479+0000 7f09d0fcc640 10 monclient: build_initial_monmap 2024-04-26T04:24:40.482 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.479+0000 7f09d0fcc640 10 monclient: monmap: 2024-04-26T04:24:40.482 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:24:40.482 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:40.482 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:24:40.482488+0000 2024-04-26T04:24:40.483 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:24:40.482488+0000 2024-04-26T04:24:40.483 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:24:40.483 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:40.483 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:24:40.483 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:24:40.483 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:24:40.483 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:24:40.483 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:24:40.483 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:40.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.479+0000 7f09d0fcc640 1 Processor -- start 2024-04-26T04:24:40.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.479+0000 7f09d0fcc640 1 -- start start 2024-04-26T04:24:40.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.479+0000 7f09d0fcc640 10 monclient: init 2024-04-26T04:24:40.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.479+0000 7f09d0fcc640 10 monclient: _reopen_session rank -1 2024-04-26T04:24:40.485 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.479+0000 7f09d0fcc640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-04-26T04:24:40.486 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.479+0000 7f09d0fcc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f09cc0a0bb0 addr v1:172.21.15.186:6789/0 2024-04-26T04:24:40.486 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.479+0000 7f09d0fcc640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f09cc136310 0x7f09cc138700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:40.486 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.479+0000 7f09d0fcc640 10 monclient(hunting): picked mon.noname-c con 0x7f09cc136310 addr v2:172.21.15.186:3300/0 2024-04-26T04:24:40.486 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.479+0000 7f09d0fcc640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f09cc138c40 0x7f09cc13b030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:40.486 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.479+0000 7f09d0fcc640 10 monclient(hunting): picked mon.noname-a con 0x7f09cc138c40 addr v2:172.21.15.63:3300/0 2024-04-26T04:24:40.486 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.479+0000 7f09d0fcc640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09cc09f270 con 0x7f09cc0a0bb0 2024-04-26T04:24:40.486 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.479+0000 7f09d0fcc640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:40.486 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.479+0000 7f09d0fcc640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7f09cc0a0f90 con 0x7f09cc138c40 2024-04-26T04:24:40.486 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.479+0000 7f09d0fcc640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:40.486 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.479+0000 7f09d0fcc640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f09cc099240 con 0x7f09cc136310 2024-04-26T04:24:40.486 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.479+0000 7f09d0fcc640 10 monclient(hunting): _renew_subs 2024-04-26T04:24:40.486 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.479+0000 7f09d0fcc640 10 monclient(hunting): authenticate will time out at 1312.190200s 2024-04-26T04:24:40.486 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.479+0000 7f09cad9d640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f09cc138c40 0x7f09cc13b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:40.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.479+0000 7f09cad9d640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f09cc138c40 0x7f09cc13b030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:53004/0 (socket says 172.21.15.63:53004) 2024-04-26T04:24:40.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.479+0000 7f09cad9d640 1 -- 172.21.15.63:0/3801538972 learned_addr learned my addr 172.21.15.63:0/3801538972 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:24:40.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.479+0000 7f09cad9d640 10 monclient(hunting): get_auth_request con 0x7f09cc138c40 auth_method 0 2024-04-26T04:24:40.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.479+0000 7f09cad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:40.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.479+0000 7f09cad9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:40.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.479+0000 7f09cad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:40.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.479+0000 7f09cad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:40.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.479+0000 7f09cad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:40.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.479+0000 7f09cad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:24:40.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.479+0000 7f09c9d9b640 1 --2- 172.21.15.63:0/3801538972 >> v2:172.21.15.186:3300/0 conn(0x7f09cc136310 0x7f09cc138700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:40.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.479+0000 7f09cad9d640 10 monclient(hunting): handle_auth_done global_id 5382 payload 274 2024-04-26T04:24:40.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.479+0000 7f09cad9d640 1 -- 172.21.15.63:0/3801538972 >> v2:172.21.15.186:3300/0 conn(0x7f09cc136310 msgr2=0x7f09cc138700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:24:40.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.479+0000 7f09cad9d640 1 --2- 172.21.15.63:0/3801538972 >> v2:172.21.15.186:3300/0 conn(0x7f09cc136310 0x7f09cc138700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:40.487 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.479+0000 7f09cad9d640 1 -- 172.21.15.63:0/3801538972 >> v1:172.21.15.186:6789/0 conn(0x7f09cc0a0bb0 legacy=0x7f09cc09eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:24:40.488 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.479+0000 7f09cad9d640 10 monclient: _finish_hunting 0 2024-04-26T04:24:40.488 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.479+0000 7f09cad9d640 1 monclient: found mon.noname-a 2024-04-26T04:24:40.488 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.479+0000 7f09cad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:24:40.488 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.479+0000 7f09cad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:24:40.488 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.479+0000 7f09cad9d640 1 -- 172.21.15.63:0/3801538972 --> v2:172.21.15.63:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09cc09b820 con 0x7f09cc138c40 2024-04-26T04:24:40.488 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.479+0000 7f09c9d9b640 10 monclient: get_auth_request con 0x7f09cc136310 auth_method 0 2024-04-26T04:24:40.488 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.479+0000 7f09c9d9b640 1 --2- 172.21.15.63:0/3801538972 >> v2:172.21.15.186:3300/0 conn(0x7f09cc136310 0x7f09cc138700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T04:24:40.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.483+0000 7f09cad9d640 1 -- 172.21.15.63:0/3801538972 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f09cc138c40 msgr2=0x7f09cc13b030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:24:40.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.483+0000 7f09cad9d640 1 --2- 172.21.15.63:0/3801538972 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f09cc138c40 0x7f09cc13b030 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f09b4009870 tx=0x7f09b401c370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc42a8e9a7b7b201 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:40.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.483+0000 7f09c959a640 1 -- 172.21.15.63:0/3801538972 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b40b0070 con 0x7f09cc138c40 2024-04-26T04:24:40.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.483+0000 7f09c959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:40.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.483+0000 7f09c959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:24:40.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.483+0000 7f09c959a640 10 monclient: dump: 2024-04-26T04:24:40.493 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:40.493 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:40.493 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:40.493 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:40.493 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:40.493 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:40.493 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:40.493 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:40.493 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:40.493 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:40.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.483+0000 7f09c959a640 10 monclient: _finish_auth 0 2024-04-26T04:24:40.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.483+0000 7f09c959a640 10 monclient: _check_auth_tickets 2024-04-26T04:24:40.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.483+0000 7f09c959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:24:40.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.483+0000 7f09c959a640 1 -- 172.21.15.63:0/3801538972 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09b40ab600 con 0x7f09cc138c40 2024-04-26T04:24:40.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.483+0000 7f09d0fcc640 5 monclient: authenticate success, global_id 5382 2024-04-26T04:24:40.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.483+0000 7f09d0fcc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T04:24:40.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.483+0000 7f09c959a640 10 monclient: handle_config config(0 keys) 2024-04-26T04:24:40.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.483+0000 7f09d0fcc640 10 monclient: get_monmap_and_config success 2024-04-26T04:24:40.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.483+0000 7f09d0fcc640 4 set_mon_vals no callback set 2024-04-26T04:24:40.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.483+0000 7f09c959a640 1 -- 172.21.15.63:0/3801538972 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b40ab9d0 con 0x7f09cc138c40 2024-04-26T04:24:40.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.483+0000 7f09c959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:40.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.483+0000 7f09c959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:24:40.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.483+0000 7f09c959a640 10 monclient: dump: 2024-04-26T04:24:40.494 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:40.494 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:40.494 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:40.494 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:40.495 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:40.495 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:40.495 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:40.495 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:40.495 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:40.495 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:40.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.483+0000 7f09d0fcc640 10 monclient: shutdown 2024-04-26T04:24:40.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.483+0000 7f09d0fcc640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:24:40.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.483+0000 7f09d0fcc640 1 -- 172.21.15.63:0/3801538972 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f09cc138c40 msgr2=0x7f09cc13b030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:40.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.483+0000 7f09d0fcc640 1 --2- 172.21.15.63:0/3801538972 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f09cc138c40 0x7f09cc13b030 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f09b4009870 tx=0x7f09b401c370 comp rx=0 tx=0).stop 2024-04-26T04:24:40.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.483+0000 7f09d0fcc640 1 -- 172.21.15.63:0/3801538972 shutdown_connections 2024-04-26T04:24:40.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.483+0000 7f09d0fcc640 1 --2- 172.21.15.63:0/3801538972 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f09cc138c40 0x7f09cc13b030 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:40.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.483+0000 7f09d0fcc640 1 --2- 172.21.15.63:0/3801538972 >> v2:172.21.15.186:3300/0 conn(0x7f09cc136310 0x7f09cc138700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:40.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.483+0000 7f09d0fcc640 1 -- 172.21.15.63:0/3801538972 >> 172.21.15.63:0/3801538972 conn(0x7f09cc0a1e70 msgr2=0x7f09cc0a2280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:24:40.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.483+0000 7f09d0fcc640 1 -- 172.21.15.63:0/3801538972 shutdown_connections 2024-04-26T04:24:40.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.483+0000 7f09d0fcc640 1 -- 172.21.15.63:0/3801538972 wait complete. 2024-04-26T04:24:40.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.487+0000 7f09d0fcc640 10 monclient: build_initial_monmap 2024-04-26T04:24:40.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.487+0000 7f09d0fcc640 10 monclient: monmap: 2024-04-26T04:24:40.496 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:24:40.496 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:24:40.496 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:24:40.496 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:24:40.496 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:24:40.496 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:40.496 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:24:40.496 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:24:40.496 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:24:40.496 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:40.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.487+0000 7f09d0fcc640 1 Processor -- start 2024-04-26T04:24:40.497 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.487+0000 7f09d0fcc640 1 -- start start 2024-04-26T04:24:40.497 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.487+0000 7f09d0fcc640 10 monclient: init 2024-04-26T04:24:40.497 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.487+0000 7f09d0fcc640 10 monclient: _reopen_session rank -1 2024-04-26T04:24:40.497 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.487+0000 7f09d0fcc640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T04:24:40.497 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.487+0000 7f09d0fcc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09cc0a0bb0 0x7f09cc134770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:40.497 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.487+0000 7f09d0fcc640 10 monclient(hunting): picked mon.noname-b con 0x7f09cc0a0bb0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:24:40.497 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.487+0000 7f09d0fcc640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f09cc136310 0x7f09cc134cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:40.497 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.487+0000 7f09d0fcc640 10 monclient(hunting): picked mon.noname-c con 0x7f09cc136310 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:24:40.497 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.487+0000 7f09d0fcc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f09cc138c40 0x7f09cc12a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:40.497 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.487+0000 7f09d0fcc640 10 monclient(hunting): picked mon.noname-a con 0x7f09cc138c40 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:24:40.497 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.487+0000 7f09d0fcc640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:40.497 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.487+0000 7f09d0fcc640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f09cc13e240 con 0x7f09cc138c40 2024-04-26T04:24:40.497 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.487+0000 7f09d0fcc640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:40.497 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.487+0000 7f09d0fcc640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f09cc13e0c0 con 0x7f09cc0a0bb0 2024-04-26T04:24:40.497 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.487+0000 7f09d0fcc640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:40.498 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.487+0000 7f09d0fcc640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f09cc13e3c0 con 0x7f09cc136310 2024-04-26T04:24:40.498 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.487+0000 7f09d0fcc640 10 monclient(hunting): _renew_subs 2024-04-26T04:24:40.498 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.487+0000 7f09d0fcc640 10 monclient(hunting): authenticate will time out at 1312.197224s 2024-04-26T04:24:40.498 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.487+0000 7f09cad9d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f09cc138c40 0x7f09cc12a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:40.498 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.487+0000 7f09cad9d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f09cc138c40 0x7f09cc12a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:53012/0 (socket says 172.21.15.63:53012) 2024-04-26T04:24:40.498 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.487+0000 7f09cad9d640 1 -- 172.21.15.63:0/3667411445 learned_addr learned my addr 172.21.15.63:0/3667411445 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:24:40.498 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.487+0000 7f09cad9d640 10 monclient(hunting): get_auth_request con 0x7f09cc138c40 auth_method 0 2024-04-26T04:24:40.498 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.487+0000 7f09cad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:40.498 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.487+0000 7f09cad9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:40.498 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.487+0000 7f09c9d9b640 1 --2- 172.21.15.63:0/3667411445 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f09cc136310 0x7f09cc134cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:40.498 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.487+0000 7f09cad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:40.498 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.487+0000 7f09c9d9b640 10 monclient(hunting): get_auth_request con 0x7f09cc136310 auth_method 0 2024-04-26T04:24:40.498 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.487+0000 7f09c9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:40.499 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.487+0000 7f09c9d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:40.499 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.487+0000 7f09c9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:40.500 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.487+0000 7f09cad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:40.500 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.487+0000 7f09cad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:40.500 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.487+0000 7f09cad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:24:40.500 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.487+0000 7f09c9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:40.500 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.487+0000 7f09c9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:40.500 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.487+0000 7f09c9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:24:40.500 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.487+0000 7f09ca59c640 1 --2- 172.21.15.63:0/3667411445 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09cc0a0bb0 0x7f09cc134770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:40.500 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.487+0000 7f09ca59c640 10 monclient(hunting): get_auth_request con 0x7f09cc0a0bb0 auth_method 0 2024-04-26T04:24:40.500 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.487+0000 7f09ca59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:40.500 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.487+0000 7f09ca59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:40.500 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.487+0000 7f09ca59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:40.500 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.487+0000 7f09cad9d640 10 monclient(hunting): handle_auth_done global_id 5385 payload 824 2024-04-26T04:24:40.500 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.487+0000 7f09cad9d640 1 -- 172.21.15.63:0/3667411445 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f09cc136310 msgr2=0x7f09cc134cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:40.500 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.487+0000 7f09cad9d640 1 --2- 172.21.15.63:0/3667411445 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f09cc136310 0x7f09cc134cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:40.501 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.487+0000 7f09cad9d640 1 -- 172.21.15.63:0/3667411445 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09cc0a0bb0 msgr2=0x7f09cc134770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:40.501 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.487+0000 7f09cad9d640 1 --2- 172.21.15.63:0/3667411445 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09cc0a0bb0 0x7f09cc134770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:40.501 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.487+0000 7f09cad9d640 10 monclient: _finish_hunting 0 2024-04-26T04:24:40.501 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.487+0000 7f09cad9d640 1 monclient: found mon.noname-a 2024-04-26T04:24:40.501 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.487+0000 7f09cad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:24:40.501 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.487+0000 7f09cad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:24:40.501 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.487+0000 7f09cad9d640 1 -- 172.21.15.63:0/3667411445 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09cc12c030 con 0x7f09cc138c40 2024-04-26T04:24:40.501 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.487+0000 7f09c9d9b640 1 --2- 172.21.15.63:0/3667411445 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f09cc136310 0x7f09cc134cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T04:24:40.501 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.487+0000 7f09cad9d640 1 -- 172.21.15.63:0/3667411445 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f09cc138c40 msgr2=0x7f09cc12a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T04:24:40.501 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.487+0000 7f09cad9d640 1 -- 172.21.15.63:0/3667411445 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f09cc138c40 msgr2=0x7f09cc12a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T04:24:40.501 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.487+0000 7f09cad9d640 1 --2- 172.21.15.63:0/3667411445 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f09cc138c40 0x7f09cc12a260 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7f09b40ae040 tx=0x7f09b4019110 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:24:40.501 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.487+0000 7f09cad9d640 1 --2- 172.21.15.63:0/3667411445 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f09cc138c40 0x7f09cc12a260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:24:40.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.691+0000 7f09cad9d640 1 --2- 172.21.15.63:0/3667411445 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f09cc138c40 0x7f09cc12a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:40.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.691+0000 7f09cad9d640 10 monclient: get_auth_request con 0x7f09cc138c40 auth_method 0 2024-04-26T04:24:40.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.691+0000 7f09cad9d640 0 --2- 172.21.15.63:0/3667411445 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f09cc138c40 0x7f09cc12a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-04-26T04:24:40.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.691+0000 7f09cad9d640 1 --2- 172.21.15.63:0/3667411445 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f09cc138c40 0x7f09cc12a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:40.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.691+0000 7f09c2ffd640 10 monclient: ms_handle_reset current mon [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:24:40.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.691+0000 7f09c2ffd640 10 monclient: _reopen_session rank -1 2024-04-26T04:24:40.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.691+0000 7f09c2ffd640 1 -- 172.21.15.63:0/3667411445 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f09cc138c40 msgr2=0x7f09cc12a260 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:24:40.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.691+0000 7f09c2ffd640 1 --2- 172.21.15.63:0/3667411445 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f09cc138c40 0x7f09cc12a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:40.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.691+0000 7f09c2ffd640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T04:24:40.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.691+0000 7f09c2ffd640 1 --2- 172.21.15.63:0/3667411445 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f09cc136310 0x7f09ac0054f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:40.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.691+0000 7f09c2ffd640 10 monclient(hunting): picked mon.noname-c con 0x7f09cc136310 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:24:40.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.691+0000 7f09c2ffd640 1 --2- 172.21.15.63:0/3667411445 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09cc0a0bb0 0x7f09ac007fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:40.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.691+0000 7f09c2ffd640 10 monclient(hunting): picked mon.noname-b con 0x7f09cc0a0bb0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:24:40.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.691+0000 7f09c2ffd640 1 --2- 172.21.15.63:0/3667411445 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f09ac008a00 0x7f09ac00ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:40.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.691+0000 7f09c2ffd640 10 monclient(hunting): picked mon.noname-a con 0x7f09ac008a00 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:24:40.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.691+0000 7f09c2ffd640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:40.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.691+0000 7f09c2ffd640 1 -- 172.21.15.63:0/3667411445 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f09ac00b990 con 0x7f09ac008a00 2024-04-26T04:24:40.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.691+0000 7f09c2ffd640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:40.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.691+0000 7f09c2ffd640 1 -- 172.21.15.63:0/3667411445 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f09ac00bb10 con 0x7f09cc0a0bb0 2024-04-26T04:24:40.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.691+0000 7f09c2ffd640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:40.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.691+0000 7f09c2ffd640 1 -- 172.21.15.63:0/3667411445 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f09ac00bd10 con 0x7f09cc136310 2024-04-26T04:24:40.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.691+0000 7f09c2ffd640 10 monclient(hunting): _renew_subs 2024-04-26T04:24:40.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.691+0000 7f09ca59c640 1 --2- 172.21.15.63:0/3667411445 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f09cc136310 0x7f09ac0054f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:40.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.691+0000 7f09ca59c640 10 monclient(hunting): get_auth_request con 0x7f09cc136310 auth_method 0 2024-04-26T04:24:40.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.691+0000 7f09c9d9b640 1 --2- 172.21.15.63:0/3667411445 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09cc0a0bb0 0x7f09ac007fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:40.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.691+0000 7f09ca59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:40.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.691+0000 7f09ca59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:40.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.691+0000 7f09ca59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:24:40.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.691+0000 7f09cad9d640 1 --2- 172.21.15.63:0/3667411445 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f09ac008a00 0x7f09ac00ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:40.702 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.691+0000 7f09c9d9b640 10 monclient(hunting): get_auth_request con 0x7f09cc0a0bb0 auth_method 0 2024-04-26T04:24:40.702 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.691+0000 7f09c9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:40.702 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.691+0000 7f09c9d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:40.702 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.691+0000 7f09c9d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:24:40.702 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.691+0000 7f09cad9d640 10 monclient(hunting): get_auth_request con 0x7f09ac008a00 auth_method 0 2024-04-26T04:24:40.702 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.691+0000 7f09cad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:40.702 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.691+0000 7f09cad9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:40.702 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.691+0000 7f09cad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:24:40.703 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.691+0000 7f09c9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:40.703 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.691+0000 7f09c9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:40.703 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.691+0000 7f09c9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T04:24:40.703 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.691+0000 7f09ca59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:40.703 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.691+0000 7f09ca59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:40.703 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.691+0000 7f09ca59c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T04:24:40.703 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.691+0000 7f09cad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:40.704 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.691+0000 7f09cad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:40.704 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.691+0000 7f09cad9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T04:24:40.704 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.691+0000 7f09c9d9b640 10 monclient(hunting): handle_auth_done global_id 5385 payload 293 2024-04-26T04:24:40.704 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.691+0000 7f09c9d9b640 1 -- 172.21.15.63:0/3667411445 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f09cc136310 msgr2=0x7f09ac0054f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:40.704 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.691+0000 7f09c9d9b640 1 --2- 172.21.15.63:0/3667411445 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f09cc136310 0x7f09ac0054f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:40.704 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.691+0000 7f09c9d9b640 1 -- 172.21.15.63:0/3667411445 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f09ac008a00 msgr2=0x7f09ac00ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:40.704 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.691+0000 7f09c9d9b640 1 --2- 172.21.15.63:0/3667411445 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f09ac008a00 0x7f09ac00ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:40.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.691+0000 7f09c9d9b640 10 monclient: _finish_hunting 0 2024-04-26T04:24:40.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.691+0000 7f09c9d9b640 1 monclient: found mon.noname-b 2024-04-26T04:24:40.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.691+0000 7f09c9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:24:40.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.691+0000 7f09c9d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.186:3300/0 2024-04-26T04:24:40.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.691+0000 7f09c9d9b640 1 -- 172.21.15.63:0/3667411445 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09ac00bfd0 con 0x7f09cc0a0bb0 2024-04-26T04:24:40.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.691+0000 7f09ca59c640 1 --2- 172.21.15.63:0/3667411445 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f09cc136310 0x7f09ac0054f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T04:24:40.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.695+0000 7f09c9d9b640 1 -- 172.21.15.63:0/3667411445 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09cc0a0bb0 msgr2=0x7f09ac007fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:24:40.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.695+0000 7f09c9d9b640 1 --2- 172.21.15.63:0/3667411445 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09cc0a0bb0 0x7f09ac007fa0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f09cc09c0d0 tx=0x7f09b40a6000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:40.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.695+0000 7f09c2ffd640 1 -- 172.21.15.63:0/3667411445 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09bc00d5a0 con 0x7f09cc0a0bb0 2024-04-26T04:24:40.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.695+0000 7f09c2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:40.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.695+0000 7f09c2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T04:24:40.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.695+0000 7f09c2ffd640 10 monclient: dump: 2024-04-26T04:24:40.711 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:40.711 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:40.712 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:40.712 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:40.712 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:40.712 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:40.712 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:40.712 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:40.712 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:40.712 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:40.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.695+0000 7f09c2ffd640 10 monclient: _finish_auth 0 2024-04-26T04:24:40.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.695+0000 7f09c2ffd640 10 monclient: _check_auth_tickets 2024-04-26T04:24:40.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.695+0000 7f09c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:24:40.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.695+0000 7f09c2ffd640 1 -- 172.21.15.63:0/3667411445 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09bc00cc10 con 0x7f09cc0a0bb0 2024-04-26T04:24:40.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.695+0000 7f09c2ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T04:24:40.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.695+0000 7f09c2ffd640 1 -- 172.21.15.63:0/3667411445 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09bc00a440 con 0x7f09cc0a0bb0 2024-04-26T04:24:40.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.695+0000 7f09d0fcc640 5 monclient: authenticate success, global_id 5385 2024-04-26T04:24:40.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.695+0000 7f09d0fcc640 10 monclient: _renew_subs 2024-04-26T04:24:40.714 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.695+0000 7f09d0fcc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-04-26T04:24:40.714 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.695+0000 7f09d0fcc640 1 -- 172.21.15.63:0/3667411445 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09cc09ba60 con 0x7f09cc0a0bb0 2024-04-26T04:24:40.714 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.695+0000 7f09c2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:40.714 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.695+0000 7f09c2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T04:24:40.714 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.695+0000 7f09c8d99640 4 set_mon_vals no callback set 2024-04-26T04:24:40.714 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.695+0000 7f09c2ffd640 10 monclient: dump: 2024-04-26T04:24:40.714 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:40.715 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:40.715 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:40.715 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:40.715 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:40.715 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:40.715 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:40.715 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:40.716 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:40.716 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:40.716 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.695+0000 7f09d0fcc640 10 monclient: _renew_subs 2024-04-26T04:24:40.716 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.695+0000 7f09d0fcc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-04-26T04:24:40.716 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.695+0000 7f09d0fcc640 1 -- 172.21.15.63:0/3667411445 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09cc12cf00 con 0x7f09cc0a0bb0 2024-04-26T04:24:40.716 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.695+0000 7f09d0fcc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:24:40.716 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.695+0000 7f09d0fcc640 1 --2- 172.21.15.63:0/3667411445 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09cc12a260 0x7f09cc12a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:40.716 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.695+0000 7f09d0fcc640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:40.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.695+0000 7f09d0fcc640 1 -- 172.21.15.63:0/3667411445 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f09cc12cb80 con 0x7f09cc12a260 2024-04-26T04:24:40.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.695+0000 7f09ca59c640 1 --2- 172.21.15.63:0/3667411445 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09cc12a260 0x7f09cc12a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:40.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.695+0000 7f09ca59c640 10 monclient: get_auth_request con 0x7f09cc12a260 auth_method 0 2024-04-26T04:24:40.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.695+0000 7f09ca59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:40.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.695+0000 7f09ca59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:40.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.695+0000 7f09ca59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:40.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.695+0000 7f09c2ffd640 1 -- 172.21.15.63:0/3667411445 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f09bc00a5e0 con 0x7f09cc0a0bb0 2024-04-26T04:24:40.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.695+0000 7f09ca59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:40.718 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.695+0000 7f09ca59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:40.718 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.695+0000 7f09ca59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:24:40.718 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.695+0000 7f09c2ffd640 1 --2- 172.21.15.63:0/3667411445 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f09cc138c40 0x7f09ac0407c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:40.718 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.695+0000 7f09c2ffd640 1 -- 172.21.15.63:0/3667411445 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(222..222 src has 1..222) ==== 4738+0+0 (secure 0 0 0) 0x7f09bc0497f0 con 0x7f09cc0a0bb0 2024-04-26T04:24:40.718 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.695+0000 7f09ca59c640 10 monclient(hunting): handle_auth_done global_id 34144 payload 824 2024-04-26T04:24:40.718 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.695+0000 7f09ca59c640 1 -- 172.21.15.63:0/3667411445 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f09cc12c940 con 0x7f09cc12a260 2024-04-26T04:24:40.718 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.695+0000 7f09cad9d640 1 --2- 172.21.15.63:0/3667411445 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f09cc138c40 0x7f09ac0407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:40.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.695+0000 7f09cad9d640 10 monclient: get_auth_request con 0x7f09cc138c40 auth_method 0 2024-04-26T04:24:40.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.699+0000 7f09ca59c640 1 -- 172.21.15.63:0/3667411445 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09cc12a260 msgr2=0x7f09cc12a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:24:40.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.699+0000 7f09ca59c640 1 --2- 172.21.15.63:0/3667411445 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09cc12a260 0x7f09cc12a640 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f09c4011890 tx=0x7f09c4011d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:40.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.699+0000 7f09c2ffd640 1 -- 172.21.15.63:0/3667411445 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c4019020 con 0x7f09cc12a260 2024-04-26T04:24:40.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.699+0000 7f09c2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:40.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.699+0000 7f09c2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T04:24:40.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.699+0000 7f09c2ffd640 10 monclient: dump: 2024-04-26T04:24:40.719 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:40.720 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:40.720 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:40.720 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:40.720 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:40.720 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:40.720 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:40.720 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:40.720 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:40.721 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:40.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.699+0000 7f09cad9d640 1 --2- 172.21.15.63:0/3667411445 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f09cc138c40 0x7f09ac0407c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f09b401cc10 tx=0x7f09b4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:40.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.699+0000 7f09c2ffd640 1 -- 172.21.15.63:0/3667411445 <== mon.1 v2:172.21.15.186:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f09c4007470 con 0x7f09cc12a260 2024-04-26T04:24:40.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.699+0000 7f09c2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:24:40.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.699+0000 7f09c2ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:24:40.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.699+0000 7f09c2ffd640 1 -- 172.21.15.63:0/3667411445 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09cc12a260 msgr2=0x7f09cc12a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:40.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.699+0000 7f09c2ffd640 1 --2- 172.21.15.63:0/3667411445 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09cc12a260 0x7f09cc12a640 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f09c4011890 tx=0x7f09c4011d50 comp rx=0 tx=0).stop 2024-04-26T04:24:40.722 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.699+0000 7f09c2ffd640 1 -- 172.21.15.63:0/3667411445 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09cc12a260 msgr2=0x7f09cc12a640 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:24:40.722 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.699+0000 7f09c2ffd640 1 --2- 172.21.15.63:0/3667411445 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09cc12a260 0x7f09cc12a640 secure :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f09c4011890 tx=0x7f09c4011d50 comp rx=0 tx=0).stop 2024-04-26T04:24:40.724 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.707+0000 7f09d0fcc640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T04:24:40.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.707+0000 7f09d0fcc640 1 --2- 172.21.15.63:0/3667411445 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09cc12ab80 0x7f09cc02bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:40.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.707+0000 7f09d0fcc640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:40.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.707+0000 7f09d0fcc640 1 -- 172.21.15.63:0/3667411445 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f09cc0504d0 con 0x7f09cc12ab80 2024-04-26T04:24:40.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.707+0000 7f09ca59c640 1 --2- 172.21.15.63:0/3667411445 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09cc12ab80 0x7f09cc02bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:40.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.707+0000 7f09ca59c640 10 monclient: get_auth_request con 0x7f09cc12ab80 auth_method 0 2024-04-26T04:24:40.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.707+0000 7f09ca59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:40.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.707+0000 7f09ca59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:40.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.707+0000 7f09ca59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:40.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.707+0000 7f09ca59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:40.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.707+0000 7f09ca59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:40.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.707+0000 7f09ca59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:24:40.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.707+0000 7f09ca59c640 10 monclient(hunting): handle_auth_done global_id 34147 payload 824 2024-04-26T04:24:40.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.707+0000 7f09ca59c640 1 -- 172.21.15.63:0/3667411445 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f09cc062e10 con 0x7f09cc12ab80 2024-04-26T04:24:40.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.711+0000 7f09ca59c640 1 -- 172.21.15.63:0/3667411445 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09cc12ab80 msgr2=0x7f09cc02bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:24:40.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.711+0000 7f09ca59c640 1 --2- 172.21.15.63:0/3667411445 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09cc12ab80 0x7f09cc02bfc0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f09c4011890 tx=0x7f09c4003770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:40.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.711+0000 7f09c2ffd640 1 -- 172.21.15.63:0/3667411445 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09c4019030 con 0x7f09cc12ab80 2024-04-26T04:24:40.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.711+0000 7f09c2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:40.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.711+0000 7f09c2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T04:24:40.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.711+0000 7f09c2ffd640 10 monclient: dump: 2024-04-26T04:24:40.727 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:40.727 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:40.727 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:40.728 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:40.728 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:40.728 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:40.728 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:40.728 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:40.728 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:40.728 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:40.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.711+0000 7f09c2ffd640 1 -- 172.21.15.63:0/3667411445 <== mon.1 v2:172.21.15.186:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4720 (secure 0 0 0) 0x7f09c400f7c0 con 0x7f09cc12ab80 2024-04-26T04:24:40.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.711+0000 7f09c2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T04:24:40.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.711+0000 7f09c2ffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:24:40.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.711+0000 7f09c2ffd640 1 -- 172.21.15.63:0/3667411445 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09cc12ab80 msgr2=0x7f09cc02bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:40.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.711+0000 7f09c2ffd640 1 --2- 172.21.15.63:0/3667411445 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09cc12ab80 0x7f09cc02bfc0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f09c4011890 tx=0x7f09c4003770 comp rx=0 tx=0).stop 2024-04-26T04:24:40.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.711+0000 7f09c2ffd640 1 -- 172.21.15.63:0/3667411445 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09cc12ab80 msgr2=0x7f09cc02bfc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:24:40.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.711+0000 7f09c2ffd640 1 --2- 172.21.15.63:0/3667411445 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09cc12ab80 0x7f09cc02bfc0 secure :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f09c4011890 tx=0x7f09c4003770 comp rx=0 tx=0).stop 2024-04-26T04:24:40.730 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-26T04:24:40.730 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b", 2024-04-26T04:24:40.730 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 1, 2024-04-26T04:24:40.730 INFO:teuthology.orchestra.run.smithi063.stdout: "state": "peon", 2024-04-26T04:24:40.730 INFO:teuthology.orchestra.run.smithi063.stdout: "election_epoch": 26, 2024-04-26T04:24:40.730 INFO:teuthology.orchestra.run.smithi063.stdout: "uptime": 13491, 2024-04-26T04:24:40.730 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum": [ 2024-04-26T04:24:40.731 INFO:teuthology.orchestra.run.smithi063.stdout: 0, 2024-04-26T04:24:40.731 INFO:teuthology.orchestra.run.smithi063.stdout: 1, 2024-04-26T04:24:40.731 INFO:teuthology.orchestra.run.smithi063.stdout: 2 2024-04-26T04:24:40.731 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:24:40.731 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_age": 3, 2024-04-26T04:24:40.731 INFO:teuthology.orchestra.run.smithi063.stdout: "features": { 2024-04-26T04:24:40.731 INFO:teuthology.orchestra.run.smithi063.stdout: "required_con": "2450521708007866372", 2024-04-26T04:24:40.731 INFO:teuthology.orchestra.run.smithi063.stdout: "required_mon": [ 2024-04-26T04:24:40.732 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:24:40.732 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:24:40.732 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:24:40.732 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:24:40.732 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:24:40.732 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:24:40.732 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:24:40.733 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:24:40.733 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:24:40.733 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:24:40.733 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:24:40.733 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:24:40.733 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_con": "4540701547738038271", 2024-04-26T04:24:40.733 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_mon": [ 2024-04-26T04:24:40.733 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:24:40.734 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:24:40.734 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:24:40.734 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:24:40.734 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:24:40.734 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:24:40.734 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:24:40.734 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:24:40.735 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:24:40.735 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:24:40.735 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:24:40.735 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:24:40.735 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:40.735 INFO:teuthology.orchestra.run.smithi063.stdout: "outside_quorum": [], 2024-04-26T04:24:40.735 INFO:teuthology.orchestra.run.smithi063.stdout: "extra_probe_peers": [], 2024-04-26T04:24:40.736 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_provider": [], 2024-04-26T04:24:40.736 INFO:teuthology.orchestra.run.smithi063.stdout: "monmap": { 2024-04-26T04:24:40.736 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 1, 2024-04-26T04:24:40.736 INFO:teuthology.orchestra.run.smithi063.stdout: "fsid": "cd1c6a0c-4e4b-4014-89c0-2e535479ef39", 2024-04-26T04:24:40.736 INFO:teuthology.orchestra.run.smithi063.stdout: "modified": "2024-04-26T04:18:03.702339Z", 2024-04-26T04:24:40.736 INFO:teuthology.orchestra.run.smithi063.stdout: "created": "2024-04-26T04:18:03.702339Z", 2024-04-26T04:24:40.736 INFO:teuthology.orchestra.run.smithi063.stdout: "min_mon_release": 19, 2024-04-26T04:24:40.736 INFO:teuthology.orchestra.run.smithi063.stdout: "min_mon_release_name": "squid", 2024-04-26T04:24:40.737 INFO:teuthology.orchestra.run.smithi063.stdout: "election_strategy": 1, 2024-04-26T04:24:40.737 INFO:teuthology.orchestra.run.smithi063.stdout: "disallowed_leaders: ": "", 2024-04-26T04:24:40.737 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": false, 2024-04-26T04:24:40.737 INFO:teuthology.orchestra.run.smithi063.stdout: "tiebreaker_mon": "", 2024-04-26T04:24:40.737 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_ranks: ": "", 2024-04-26T04:24:40.737 INFO:teuthology.orchestra.run.smithi063.stdout: "features": { 2024-04-26T04:24:40.737 INFO:teuthology.orchestra.run.smithi063.stdout: "persistent": [ 2024-04-26T04:24:40.737 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:24:40.738 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:24:40.738 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:24:40.738 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:24:40.738 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:24:40.738 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:24:40.738 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:24:40.738 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:24:40.739 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:24:40.739 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:24:40.739 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:24:40.739 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:24:40.739 INFO:teuthology.orchestra.run.smithi063.stdout: "optional": [] 2024-04-26T04:24:40.739 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:40.739 INFO:teuthology.orchestra.run.smithi063.stdout: "mons": [ 2024-04-26T04:24:40.739 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:40.740 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 0, 2024-04-26T04:24:40.740 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a", 2024-04-26T04:24:40.740 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:24:40.740 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:24:40.740 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:40.740 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:24:40.740 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:3300", 2024-04-26T04:24:40.741 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:24:40.741 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:40.741 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:40.741 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:24:40.741 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6789", 2024-04-26T04:24:40.741 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:24:40.741 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:24:40.741 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:24:40.742 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:40.742 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6789/0", 2024-04-26T04:24:40.742 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6789/0", 2024-04-26T04:24:40.742 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:24:40.742 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:24:40.742 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:24:40.743 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:40.743 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:40.744 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 1, 2024-04-26T04:24:40.744 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b", 2024-04-26T04:24:40.744 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:24:40.744 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:24:40.744 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:40.744 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:24:40.744 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:3300", 2024-04-26T04:24:40.744 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:24:40.744 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:40.744 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:40.744 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:24:40.744 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:6789", 2024-04-26T04:24:40.744 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:24:40.744 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:24:40.745 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:24:40.745 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:40.745 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:6789/0", 2024-04-26T04:24:40.745 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.186:6789/0", 2024-04-26T04:24:40.745 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:24:40.745 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:24:40.745 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:24:40.745 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:40.745 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:40.745 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 2, 2024-04-26T04:24:40.745 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "c", 2024-04-26T04:24:40.745 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:24:40.745 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:24:40.745 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:40.745 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:24:40.746 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:3301", 2024-04-26T04:24:40.746 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:24:40.746 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:40.746 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:40.746 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:24:40.746 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6790", 2024-04-26T04:24:40.746 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:24:40.746 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:24:40.746 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:24:40.746 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:40.746 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6790/0", 2024-04-26T04:24:40.746 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6790/0", 2024-04-26T04:24:40.746 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:24:40.746 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:24:40.747 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:24:40.747 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:24:40.747 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:24:40.747 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:40.747 INFO:teuthology.orchestra.run.smithi063.stdout: "feature_map": { 2024-04-26T04:24:40.748 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": [ 2024-04-26T04:24:40.748 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:40.748 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:24:40.748 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:24:40.748 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 1 2024-04-26T04:24:40.749 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:24:40.749 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:24:40.749 INFO:teuthology.orchestra.run.smithi063.stdout: "client": [ 2024-04-26T04:24:40.749 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:40.749 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:24:40.750 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:24:40.750 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 2 2024-04-26T04:24:40.750 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:24:40.750 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:24:40.750 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": [ 2024-04-26T04:24:40.750 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:40.750 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:24:40.750 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:24:40.750 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 1 2024-04-26T04:24:40.750 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:24:40.750 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:24:40.750 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:40.751 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": false 2024-04-26T04:24:40.751 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-26T04:24:40.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.711+0000 7f09d0fcc640 1 -- 172.21.15.63:0/3667411445 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f09cc138c40 msgr2=0x7f09ac0407c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:40.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.711+0000 7f09d0fcc640 1 --2- 172.21.15.63:0/3667411445 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f09cc138c40 0x7f09ac0407c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f09b401cc10 tx=0x7f09b4008040 comp rx=0 tx=0).stop 2024-04-26T04:24:40.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.711+0000 7f09d0fcc640 10 monclient: shutdown 2024-04-26T04:24:40.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.711+0000 7f09d0fcc640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:24:40.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.711+0000 7f09d0fcc640 1 -- 172.21.15.63:0/3667411445 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09cc0a0bb0 msgr2=0x7f09ac007fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:40.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.711+0000 7f09d0fcc640 1 --2- 172.21.15.63:0/3667411445 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09cc0a0bb0 0x7f09ac007fa0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f09cc09c0d0 tx=0x7f09b40a6000 comp rx=0 tx=0).stop 2024-04-26T04:24:40.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.711+0000 7f09cad9d640 1 -- 172.21.15.63:0/3667411445 reap_dead start 2024-04-26T04:24:40.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.711+0000 7f09cad9d640 1 -- 172.21.15.63:0/3667411445 reap_dead start 2024-04-26T04:24:40.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.711+0000 7f09d0fcc640 1 -- 172.21.15.63:0/3667411445 shutdown_connections 2024-04-26T04:24:40.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.711+0000 7f09d0fcc640 1 -- 172.21.15.63:0/3667411445 >> 172.21.15.63:0/3667411445 conn(0x7f09cc0a1e70 msgr2=0x7f09cc135520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:24:40.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.711+0000 7f09d0fcc640 1 -- 172.21.15.63:0/3667411445 shutdown_connections 2024-04-26T04:24:40.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.711+0000 7f09d0fcc640 1 -- 172.21.15.63:0/3667411445 wait complete. 2024-04-26T04:24:40.756 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T04:24:40.756 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-04-26T04:24:40.756 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-04-26T04:24:40.756 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-04-26T04:24:40.756 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-04-26T04:24:40.757 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-26T04:24:40.757 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T04:24:40.775 INFO:tasks.ceph.mon.a:Stopped 2024-04-26T04:24:40.775 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-04-26T04:24:40.775 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T04:24:40.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.943+0000 7f8d08fe2640 10 monclient: get_monmap_and_config 2024-04-26T04:24:40.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.943+0000 7f8d08fe2640 10 monclient: build_initial_monmap 2024-04-26T04:24:40.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.943+0000 7f8d08fe2640 10 monclient: monmap: 2024-04-26T04:24:40.948 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:24:40.948 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:40.948 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:24:40.948099+0000 2024-04-26T04:24:40.948 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:24:40.948099+0000 2024-04-26T04:24:40.948 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:24:40.948 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:40.948 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:24:40.948 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:24:40.949 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:24:40.949 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:24:40.949 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:24:40.949 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:40.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.943+0000 7f8d08fe2640 1 Processor -- start 2024-04-26T04:24:40.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.943+0000 7f8d08fe2640 1 -- start start 2024-04-26T04:24:40.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.943+0000 7f8d08fe2640 10 monclient: init 2024-04-26T04:24:40.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.943+0000 7f8d08fe2640 10 monclient: _reopen_session rank -1 2024-04-26T04:24:40.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.943+0000 7f8d08fe2640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-04-26T04:24:40.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.943+0000 7f8d08fe2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8d040a0bb0 addr v1:172.21.15.63:6789/0 2024-04-26T04:24:40.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.943+0000 7f8d08fe2640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f8d04136310 0x7f8d04138700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:40.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.943+0000 7f8d08fe2640 10 monclient(hunting): picked mon.noname-c con 0x7f8d04136310 addr v2:172.21.15.186:3300/0 2024-04-26T04:24:40.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.943+0000 7f8d08fe2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8d04138c40 addr v1:172.21.15.186:6789/0 2024-04-26T04:24:40.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.943+0000 7f8d08fe2640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d0409f270 con 0x7f8d040a0bb0 2024-04-26T04:24:40.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.943+0000 7f8d08fe2640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d040a0f90 con 0x7f8d04138c40 2024-04-26T04:24:40.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.943+0000 7f8d08fe2640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:40.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.943+0000 7f8d08fe2640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f8d04099240 con 0x7f8d04136310 2024-04-26T04:24:40.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.943+0000 7f8d08fe2640 10 monclient(hunting): _renew_subs 2024-04-26T04:24:40.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.943+0000 7f8d08fe2640 10 monclient(hunting): authenticate will time out at 1312.655873s 2024-04-26T04:24:40.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.943+0000 7f8d0259c640 1 -- >> v1:172.21.15.63:6789/0 conn(0x7f8d040a0bb0 legacy=0x7f8d0409eb60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-04-26T04:24:40.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.947+0000 7f8d01d9b640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f8d04136310 0x7f8d04138700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:40.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.947+0000 7f8d01d9b640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f8d04136310 0x7f8d04138700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.63:43640/0 (socket says 172.21.15.63:43640) 2024-04-26T04:24:40.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.947+0000 7f8d01d9b640 1 -- 172.21.15.63:0/46683949 learned_addr learned my addr 172.21.15.63:0/46683949 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:24:40.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.947+0000 7f8d01d9b640 10 monclient(hunting): get_auth_request con 0x7f8d04136310 auth_method 0 2024-04-26T04:24:40.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.947+0000 7f8d01d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:40.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.947+0000 7f8d01d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:40.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.947+0000 7f8d01d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:40.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.947+0000 7f8d01d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:40.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.947+0000 7f8d01d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:40.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.947+0000 7f8d01d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:24:40.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.947+0000 7f8d01d9b640 10 monclient(hunting): handle_auth_done global_id 34150 payload 274 2024-04-26T04:24:40.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.947+0000 7f8d01d9b640 1 -- 172.21.15.63:0/46683949 >> v1:172.21.15.186:6789/0 conn(0x7f8d04138c40 legacy=0x7f8d0413b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:24:40.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.947+0000 7f8d01d9b640 1 -- 172.21.15.63:0/46683949 >> v1:172.21.15.63:6789/0 conn(0x7f8d040a0bb0 legacy=0x7f8d0409eb60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T04:24:40.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.947+0000 7f8d01d9b640 10 monclient: _finish_hunting 0 2024-04-26T04:24:40.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.947+0000 7f8d01d9b640 1 monclient: found mon.noname-c 2024-04-26T04:24:40.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.947+0000 7f8d01d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:24:40.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.947+0000 7f8d01d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.186:3300/0 2024-04-26T04:24:40.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.947+0000 7f8d01d9b640 1 -- 172.21.15.63:0/46683949 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d0409b820 con 0x7f8d04136310 2024-04-26T04:24:40.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.951+0000 7f8d01d9b640 1 -- 172.21.15.63:0/46683949 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d04136310 msgr2=0x7f8d04138700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:24:40.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.951+0000 7f8d01d9b640 1 --2- 172.21.15.63:0/46683949 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d04136310 0x7f8d04138700 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8cec00bb40 tx=0x7f8cec01e840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cdf58284aac4eb8d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:40.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.951+0000 7f8d0159a640 1 -- 172.21.15.63:0/46683949 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cec0b4070 con 0x7f8d04136310 2024-04-26T04:24:40.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.951+0000 7f8d0159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:40.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.951+0000 7f8d0159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:24:40.963 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.951+0000 7f8d0159a640 10 monclient: dump: 2024-04-26T04:24:40.963 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:40.963 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:40.963 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:40.963 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:40.963 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:40.963 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:40.963 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:40.964 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:40.964 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:40.964 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:40.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.951+0000 7f8d0159a640 10 monclient: _finish_auth 0 2024-04-26T04:24:40.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.951+0000 7f8d0159a640 10 monclient: _check_auth_tickets 2024-04-26T04:24:40.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.951+0000 7f8d0159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:24:40.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.951+0000 7f8d0159a640 1 -- 172.21.15.63:0/46683949 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8cec0a78f0 con 0x7f8d04136310 2024-04-26T04:24:40.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.951+0000 7f8d0159a640 10 monclient: handle_config config(0 keys) 2024-04-26T04:24:40.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.951+0000 7f8d0159a640 1 -- 172.21.15.63:0/46683949 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cec0a7cc0 con 0x7f8d04136310 2024-04-26T04:24:40.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.951+0000 7f8d0159a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:40.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.951+0000 7f8d0159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T04:24:40.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.951+0000 7f8d0159a640 10 monclient: dump: 2024-04-26T04:24:40.964 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:40.965 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:40.965 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:40.965 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:40.965 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:40.965 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:40.965 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:40.965 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:40.965 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:40.965 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:40.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.951+0000 7f8d08fe2640 5 monclient: authenticate success, global_id 34150 2024-04-26T04:24:40.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.951+0000 7f8d08fe2640 10 monclient: get_monmap_and_config success 2024-04-26T04:24:40.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.951+0000 7f8d08fe2640 4 set_mon_vals no callback set 2024-04-26T04:24:40.965 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.951+0000 7f8d08fe2640 10 monclient: shutdown 2024-04-26T04:24:40.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.951+0000 7f8d08fe2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:24:40.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.951+0000 7f8d08fe2640 1 -- 172.21.15.63:0/46683949 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d04136310 msgr2=0x7f8d04138700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:40.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.951+0000 7f8d08fe2640 1 --2- 172.21.15.63:0/46683949 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d04136310 0x7f8d04138700 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8cec00bb40 tx=0x7f8cec01e840 comp rx=0 tx=0).stop 2024-04-26T04:24:40.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.951+0000 7f8d08fe2640 1 -- 172.21.15.63:0/46683949 shutdown_connections 2024-04-26T04:24:40.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.951+0000 7f8d08fe2640 1 --2- 172.21.15.63:0/46683949 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d04136310 0x7f8d04138700 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:40.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.951+0000 7f8d08fe2640 1 -- 172.21.15.63:0/46683949 >> 172.21.15.63:0/46683949 conn(0x7f8d040a1e70 msgr2=0x7f8d040a2280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:24:40.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.951+0000 7f8d08fe2640 1 -- 172.21.15.63:0/46683949 shutdown_connections 2024-04-26T04:24:40.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.951+0000 7f8d08fe2640 1 -- 172.21.15.63:0/46683949 wait complete. 2024-04-26T04:24:40.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.951+0000 7f8d08fe2640 10 monclient: build_initial_monmap 2024-04-26T04:24:40.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.951+0000 7f8d08fe2640 10 monclient: monmap: 2024-04-26T04:24:40.966 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:24:40.966 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:24:40.966 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:24:40.967 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:24:40.967 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:24:40.967 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:40.967 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:24:40.967 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:24:40.967 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:24:40.967 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:40.967 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.951+0000 7f8d08fe2640 1 Processor -- start 2024-04-26T04:24:40.967 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.951+0000 7f8d08fe2640 1 -- start start 2024-04-26T04:24:40.967 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.951+0000 7f8d08fe2640 10 monclient: init 2024-04-26T04:24:40.967 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.955+0000 7f8d08fe2640 10 monclient: _reopen_session rank -1 2024-04-26T04:24:40.967 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.955+0000 7f8d08fe2640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T04:24:40.967 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.955+0000 7f8d08fe2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d040a0bb0 0x7f8d041ca860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:40.967 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.955+0000 7f8d08fe2640 10 monclient(hunting): picked mon.noname-b con 0x7f8d040a0bb0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:24:40.967 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.955+0000 7f8d08fe2640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f8d04136310 0x7f8d041353a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:40.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.955+0000 7f8d08fe2640 10 monclient(hunting): picked mon.noname-c con 0x7f8d04136310 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:24:40.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.955+0000 7f8d08fe2640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d04138c40 0x7f8d04070ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:40.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.955+0000 7f8d08fe2640 10 monclient(hunting): picked mon.noname-a con 0x7f8d04138c40 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:24:40.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.955+0000 7f8d08fe2640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:40.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.955+0000 7f8d08fe2640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f8d0409d3d0 con 0x7f8d04138c40 2024-04-26T04:24:40.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.955+0000 7f8d08fe2640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:40.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.955+0000 7f8d08fe2640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8d0409d250 con 0x7f8d040a0bb0 2024-04-26T04:24:40.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.955+0000 7f8d08fe2640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:40.969 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.955+0000 7f8d08fe2640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f8d0413e760 con 0x7f8d04136310 2024-04-26T04:24:40.969 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.955+0000 7f8d08fe2640 10 monclient(hunting): _renew_subs 2024-04-26T04:24:40.969 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.955+0000 7f8d08fe2640 10 monclient(hunting): authenticate will time out at 1312.664430s 2024-04-26T04:24:40.969 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.955+0000 7f8d02d9d640 1 -- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d04138c40 msgr2=0x7f8d04070ac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:3300/0 2024-04-26T04:24:40.969 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.955+0000 7f8d02d9d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d04138c40 0x7f8d04070ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:24:40.969 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.955+0000 7f8d01d9b640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f8d04136310 0x7f8d041353a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:40.969 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.955+0000 7f8d01d9b640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f8d04136310 0x7f8d041353a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:59342/0 (socket says 172.21.15.63:59342) 2024-04-26T04:24:40.970 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.955+0000 7f8d01d9b640 1 -- 172.21.15.63:0/174975418 learned_addr learned my addr 172.21.15.63:0/174975418 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:24:40.970 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.955+0000 7f8d01d9b640 10 monclient(hunting): get_auth_request con 0x7f8d04136310 auth_method 0 2024-04-26T04:24:40.970 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.955+0000 7f8d01d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:40.970 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.955+0000 7f8d01d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:40.970 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.955+0000 7f8d01d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:40.970 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.955+0000 7f8d01d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:40.970 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.955+0000 7f8d01d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:40.970 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.955+0000 7f8d01d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:24:40.970 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.955+0000 7f8d0259c640 1 --2- 172.21.15.63:0/174975418 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d040a0bb0 0x7f8d041ca860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:40.970 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.955+0000 7f8d0259c640 10 monclient(hunting): get_auth_request con 0x7f8d040a0bb0 auth_method 0 2024-04-26T04:24:40.970 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.955+0000 7f8d0259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:40.970 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.955+0000 7f8d0259c640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:40.970 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.955+0000 7f8d0259c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:40.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.955+0000 7f8d01d9b640 10 monclient(hunting): handle_auth_done global_id 24278 payload 824 2024-04-26T04:24:40.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.955+0000 7f8d01d9b640 1 -- 172.21.15.63:0/174975418 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d040a0bb0 msgr2=0x7f8d041ca860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:40.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.955+0000 7f8d01d9b640 1 --2- 172.21.15.63:0/174975418 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d040a0bb0 0x7f8d041ca860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:40.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.955+0000 7f8d01d9b640 1 -- 172.21.15.63:0/174975418 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d04138c40 msgr2=0x7f8d04070ac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T04:24:40.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.955+0000 7f8d01d9b640 1 --2- 172.21.15.63:0/174975418 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d04138c40 0x7f8d04070ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:40.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.955+0000 7f8d01d9b640 10 monclient: _finish_hunting 0 2024-04-26T04:24:40.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.955+0000 7f8d01d9b640 1 monclient: found mon.noname-c 2024-04-26T04:24:40.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.955+0000 7f8d01d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:24:40.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.955+0000 7f8d01d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.63:3301/0 2024-04-26T04:24:40.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.955+0000 7f8d01d9b640 1 -- 172.21.15.63:0/174975418 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d0412b5c0 con 0x7f8d04136310 2024-04-26T04:24:40.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.955+0000 7f8d01d9b640 1 -- 172.21.15.63:0/174975418 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f8d04136310 msgr2=0x7f8d041353a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:24:40.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.955+0000 7f8d01d9b640 1 --2- 172.21.15.63:0/174975418 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f8d04136310 0x7f8d041353a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8cec00bb40 tx=0x7f8cec01b5e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:40.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.955+0000 7f8cfaffd640 1 -- 172.21.15.63:0/174975418 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cec0b4040 con 0x7f8d04136310 2024-04-26T04:24:40.971 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.955+0000 7f8cfaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:40.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.955+0000 7f8cfaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:24:40.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.955+0000 7f8cfaffd640 10 monclient: dump: 2024-04-26T04:24:40.972 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:40.972 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:40.972 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:40.972 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:40.972 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:40.972 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:40.972 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:40.972 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:40.972 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:40.972 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:40.972 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.955+0000 7f8cfaffd640 10 monclient: _finish_auth 0 2024-04-26T04:24:40.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.955+0000 7f8cfaffd640 10 monclient: _check_auth_tickets 2024-04-26T04:24:40.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.955+0000 7f8cfaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:24:40.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.955+0000 7f8d08fe2640 5 monclient: authenticate success, global_id 24278 2024-04-26T04:24:40.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.955+0000 7f8cfaffd640 1 -- 172.21.15.63:0/174975418 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8cec0c0070 con 0x7f8d04136310 2024-04-26T04:24:40.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.955+0000 7f8cfaffd640 10 monclient: handle_config config(0 keys) 2024-04-26T04:24:40.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.955+0000 7f8cfaffd640 1 -- 172.21.15.63:0/174975418 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cec0afbd0 con 0x7f8d04136310 2024-04-26T04:24:40.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.955+0000 7f8cfaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:40.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.955+0000 7f8cfaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:24:40.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.955+0000 7f8cfbfff640 4 set_mon_vals no callback set 2024-04-26T04:24:40.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.955+0000 7f8cfaffd640 10 monclient: dump: 2024-04-26T04:24:40.973 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:40.973 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:40.973 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:40.973 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:40.973 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:40.974 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:40.974 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:40.974 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:40.974 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:40.974 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:40.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.955+0000 7f8d08fe2640 10 monclient: _renew_subs 2024-04-26T04:24:40.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.955+0000 7f8d08fe2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:24:40.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.955+0000 7f8d08fe2640 1 -- 172.21.15.63:0/174975418 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d0412bb10 con 0x7f8d04136310 2024-04-26T04:24:40.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.959+0000 7f8d08fe2640 10 monclient: _renew_subs 2024-04-26T04:24:40.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.959+0000 7f8d08fe2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:24:40.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.959+0000 7f8d08fe2640 1 -- 172.21.15.63:0/174975418 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d041cada0 con 0x7f8d04136310 2024-04-26T04:24:40.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.959+0000 7f8d08fe2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:24:40.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.959+0000 7f8d08fe2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:24:40.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.959+0000 7f8d08fe2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:24:40.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.959+0000 7f8d08fe2640 1 -- 172.21.15.63:0/174975418 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d04062e10 con 0x7f8d04136310 2024-04-26T04:24:40.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.959+0000 7f8cfaffd640 1 -- 172.21.15.63:0/174975418 <== mon.2 v2:172.21.15.63:3301/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f8cec0c5460 con 0x7f8d04136310 2024-04-26T04:24:40.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.959+0000 7f8cfaffd640 1 --2- 172.21.15.63:0/174975418 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f8ce4035e70 0x7f8ce4038330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:40.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.959+0000 7f8cfaffd640 1 -- 172.21.15.63:0/174975418 <== mon.2 v2:172.21.15.63:3301/0 5 ==== osd_map(222..222 src has 1..222) ==== 4738+0+0 (secure 0 0 0) 0x7f8cec0e5aa0 con 0x7f8d04136310 2024-04-26T04:24:40.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.959+0000 7f8d0259c640 1 --2- 172.21.15.63:0/174975418 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f8ce4035e70 0x7f8ce4038330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:40.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.959+0000 7f8d0259c640 10 monclient: get_auth_request con 0x7f8ce4035e70 auth_method 0 2024-04-26T04:24:40.976 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.963+0000 7f8d0259c640 1 --2- 172.21.15.63:0/174975418 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f8ce4035e70 0x7f8ce4038330 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f8cf4006fd0 tx=0x7f8cf4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:40.976 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.963+0000 7f8cfaffd640 1 -- 172.21.15.63:0/174975418 <== mon.2 v2:172.21.15.63:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f8d0412bb10 con 0x7f8d04136310 2024-04-26T04:24:40.976 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.963+0000 7f8cfaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:24:40.976 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:40.963+0000 7f8cfaffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:24:41.188 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:41.183+0000 7f8d08fe2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T04:24:41.189 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:41.183+0000 7f8d08fe2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T04:24:41.189 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:41.183+0000 7f8d08fe2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:24:41.189 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:41.183+0000 7f8d08fe2640 1 -- 172.21.15.63:0/174975418 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f8d0412bd50 con 0x7f8d04136310 2024-04-26T04:24:41.189 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:41.183+0000 7f8cfaffd640 1 -- 172.21.15.63:0/174975418 <== mon.2 v2:172.21.15.63:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f8d040a0f90 con 0x7f8d04136310 2024-04-26T04:24:41.189 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:41.183+0000 7f8cfaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T04:24:41.189 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:41.183+0000 7f8cfaffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:24:41.190 INFO:teuthology.orchestra.run.smithi063.stdout:{"election_epoch":26,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cd1c6a0c-4e4b-4014-89c0-2e535479ef39","modified":"2024-04-26T04:18:03.702339Z","created":"2024-04-26T04:18:03.702339Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3300","nonce":0},{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:3300","nonce":0},{"type":"v1","addr":"172.21.15.186:6789","nonce":0}]},"addr":"172.21.15.186:6789/0","public_addr":"172.21.15.186:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3301","nonce":0},{"type":"v1","addr":"172.21.15.63:6790","nonce":0}]},"addr":"172.21.15.63:6790/0","public_addr":"172.21.15.63:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T04:24:41.191 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:41.187+0000 7f8d08fe2640 1 -- 172.21.15.63:0/174975418 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f8ce4035e70 msgr2=0x7f8ce4038330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:41.191 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:41.187+0000 7f8d08fe2640 1 --2- 172.21.15.63:0/174975418 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f8ce4035e70 0x7f8ce4038330 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f8cf4006fd0 tx=0x7f8cf4008040 comp rx=0 tx=0).stop 2024-04-26T04:24:41.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:41.187+0000 7f8d08fe2640 10 monclient: shutdown 2024-04-26T04:24:41.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:41.187+0000 7f8d08fe2640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:24:41.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:41.187+0000 7f8d08fe2640 1 -- 172.21.15.63:0/174975418 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f8d04136310 msgr2=0x7f8d041353a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:41.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:41.187+0000 7f8d08fe2640 1 --2- 172.21.15.63:0/174975418 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f8d04136310 0x7f8d041353a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8cec00bb40 tx=0x7f8cec01b5e0 comp rx=0 tx=0).stop 2024-04-26T04:24:41.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:41.187+0000 7f8d08fe2640 1 -- 172.21.15.63:0/174975418 shutdown_connections 2024-04-26T04:24:41.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:41.187+0000 7f8d08fe2640 1 --2- 172.21.15.63:0/174975418 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f8ce4035e70 0x7f8ce4038330 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:41.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:41.187+0000 7f8d08fe2640 1 --2- 172.21.15.63:0/174975418 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d04138c40 0x7f8d04070ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:41.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:41.187+0000 7f8d08fe2640 1 --2- 172.21.15.63:0/174975418 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f8d04136310 0x7f8d041353a0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:41.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:41.187+0000 7f8d08fe2640 1 --2- 172.21.15.63:0/174975418 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d040a0bb0 0x7f8d041ca860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:41.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:41.187+0000 7f8d08fe2640 1 -- 172.21.15.63:0/174975418 >> 172.21.15.63:0/174975418 conn(0x7f8d040a1e70 msgr2=0x7f8d0409d7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:24:41.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:41.187+0000 7f8d08fe2640 1 -- 172.21.15.63:0/174975418 shutdown_connections 2024-04-26T04:24:41.192 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:41.187+0000 7f8d08fe2640 1 -- 172.21.15.63:0/174975418 wait complete. 2024-04-26T04:24:44.210 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T04:24:44.344 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.339+0000 7f8a862b4640 10 monclient: get_monmap_and_config 2024-04-26T04:24:44.344 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.339+0000 7f8a862b4640 10 monclient: build_initial_monmap 2024-04-26T04:24:44.344 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.339+0000 7f8a862b4640 10 monclient: monmap: 2024-04-26T04:24:44.344 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:24:44.345 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:44.345 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:24:44.344392+0000 2024-04-26T04:24:44.345 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:24:44.344392+0000 2024-04-26T04:24:44.345 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:24:44.345 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:44.345 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:24:44.345 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:24:44.345 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:24:44.345 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:24:44.345 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:24:44.345 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:44.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.339+0000 7f8a862b4640 1 Processor -- start 2024-04-26T04:24:44.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.339+0000 7f8a862b4640 1 -- start start 2024-04-26T04:24:44.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.339+0000 7f8a862b4640 10 monclient: init 2024-04-26T04:24:44.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.343+0000 7f8a862b4640 10 monclient: _reopen_session rank -1 2024-04-26T04:24:44.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.343+0000 7f8a862b4640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-04-26T04:24:44.349 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.343+0000 7f8a862b4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8a80072ec0 addr v1:172.21.15.186:6789/0 2024-04-26T04:24:44.349 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.343+0000 7f8a862b4640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f8a800732a0 0x7f8a800998b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:44.349 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.343+0000 7f8a862b4640 10 monclient(hunting): picked mon.noname-a con 0x7f8a800732a0 addr v2:172.21.15.63:3300/0 2024-04-26T04:24:44.349 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.343+0000 7f8a862b4640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f8a801413b0 0x7f8a801437a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:44.349 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.343+0000 7f8a862b4640 10 monclient(hunting): picked mon.noname-b con 0x7f8a801413b0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:24:44.349 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.343+0000 7f8a862b4640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a8009bba0 con 0x7f8a80072ec0 2024-04-26T04:24:44.349 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.343+0000 7f8a862b4640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:44.349 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.343+0000 7f8a862b4640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7f8a8009b5e0 con 0x7f8a800732a0 2024-04-26T04:24:44.349 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.343+0000 7f8a862b4640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:44.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.343+0000 7f8a862b4640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f8a800a18c0 con 0x7f8a801413b0 2024-04-26T04:24:44.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.343+0000 7f8a862b4640 10 monclient(hunting): _renew_subs 2024-04-26T04:24:44.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.343+0000 7f8a862b4640 10 monclient(hunting): authenticate will time out at 1316.052168s 2024-04-26T04:24:44.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.343+0000 7f8a7f7fe640 1 -- >> v2:172.21.15.63:3300/0 conn(0x7f8a800732a0 msgr2=0x7f8a800998b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.63:3300/0 2024-04-26T04:24:44.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.343+0000 7f8a7f7fe640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f8a800732a0 0x7f8a800998b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:24:44.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.343+0000 7f8a84851640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f8a801413b0 0x7f8a801437a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:44.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.343+0000 7f8a84851640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f8a801413b0 0x7f8a801437a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:59348/0 (socket says 172.21.15.63:59348) 2024-04-26T04:24:44.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.343+0000 7f8a84851640 1 -- 172.21.15.63:0/435589928 learned_addr learned my addr 172.21.15.63:0/435589928 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:24:44.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.343+0000 7f8a84851640 10 monclient(hunting): get_auth_request con 0x7f8a801413b0 auth_method 0 2024-04-26T04:24:44.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.343+0000 7f8a84851640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:44.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.343+0000 7f8a84851640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:44.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.343+0000 7f8a84851640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:44.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.343+0000 7f8a84851640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:44.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.343+0000 7f8a84851640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:44.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.343+0000 7f8a84851640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:24:44.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.343+0000 7f8a84851640 10 monclient(hunting): handle_auth_done global_id 24281 payload 274 2024-04-26T04:24:44.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.343+0000 7f8a84851640 1 -- 172.21.15.63:0/435589928 >> v2:172.21.15.63:3300/0 conn(0x7f8a800732a0 msgr2=0x7f8a800998b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T04:24:44.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.343+0000 7f8a84851640 1 --2- 172.21.15.63:0/435589928 >> v2:172.21.15.63:3300/0 conn(0x7f8a800732a0 0x7f8a800998b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:44.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.343+0000 7f8a84851640 1 -- 172.21.15.63:0/435589928 >> v1:172.21.15.186:6789/0 conn(0x7f8a80072ec0 legacy=0x7f8a800991a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:24:44.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.343+0000 7f8a84851640 10 monclient: _finish_hunting 0 2024-04-26T04:24:44.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.343+0000 7f8a84851640 1 monclient: found mon.noname-b 2024-04-26T04:24:44.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.343+0000 7f8a84851640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:24:44.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.343+0000 7f8a84851640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.63:3301/0 2024-04-26T04:24:44.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.343+0000 7f8a84851640 1 -- 172.21.15.63:0/435589928 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a800a1640 con 0x7f8a801413b0 2024-04-26T04:24:44.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.347+0000 7f8a84851640 1 -- 172.21.15.63:0/435589928 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f8a801413b0 msgr2=0x7f8a801437a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:24:44.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.347+0000 7f8a84851640 1 --2- 172.21.15.63:0/435589928 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f8a801413b0 0x7f8a801437a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8a70009870 tx=0x7f8a7001c3d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=665b3ad4457b2e84 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:44.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.347+0000 7f8a7effd640 1 -- 172.21.15.63:0/435589928 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a700b1070 con 0x7f8a801413b0 2024-04-26T04:24:44.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.347+0000 7f8a7effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:44.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.347+0000 7f8a7effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T04:24:44.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.347+0000 7f8a7effd640 10 monclient: dump: 2024-04-26T04:24:44.357 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:44.357 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:44.358 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:44.358 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:44.358 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:44.358 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:44.358 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:44.358 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:44.358 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:44.358 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:44.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.347+0000 7f8a7effd640 10 monclient: _finish_auth 0 2024-04-26T04:24:44.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.347+0000 7f8a7effd640 10 monclient: _check_auth_tickets 2024-04-26T04:24:44.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.347+0000 7f8a7effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:24:44.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.347+0000 7f8a862b4640 5 monclient: authenticate success, global_id 24281 2024-04-26T04:24:44.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.347+0000 7f8a7effd640 1 -- 172.21.15.63:0/435589928 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a700ac410 con 0x7f8a801413b0 2024-04-26T04:24:44.359 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.347+0000 7f8a7effd640 10 monclient: handle_config config(0 keys) 2024-04-26T04:24:44.359 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.347+0000 7f8a862b4640 10 monclient: get_monmap_and_config success 2024-04-26T04:24:44.359 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.347+0000 7f8a7effd640 1 -- 172.21.15.63:0/435589928 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a700ac7e0 con 0x7f8a801413b0 2024-04-26T04:24:44.359 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.347+0000 7f8a862b4640 4 set_mon_vals no callback set 2024-04-26T04:24:44.359 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.347+0000 7f8a7effd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:44.359 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.347+0000 7f8a7effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:24:44.359 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.347+0000 7f8a7effd640 10 monclient: dump: 2024-04-26T04:24:44.359 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:44.359 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:44.359 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:44.359 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:44.359 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:44.359 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:44.359 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:44.360 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:44.360 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:44.360 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:44.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.347+0000 7f8a862b4640 10 monclient: shutdown 2024-04-26T04:24:44.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.347+0000 7f8a862b4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:24:44.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.347+0000 7f8a862b4640 1 -- 172.21.15.63:0/435589928 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f8a801413b0 msgr2=0x7f8a801437a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:44.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.347+0000 7f8a862b4640 1 --2- 172.21.15.63:0/435589928 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f8a801413b0 0x7f8a801437a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8a70009870 tx=0x7f8a7001c3d0 comp rx=0 tx=0).stop 2024-04-26T04:24:44.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.347+0000 7f8a862b4640 1 -- 172.21.15.63:0/435589928 shutdown_connections 2024-04-26T04:24:44.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.347+0000 7f8a862b4640 1 --2- 172.21.15.63:0/435589928 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f8a801413b0 0x7f8a801437a0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:44.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.347+0000 7f8a862b4640 1 --2- 172.21.15.63:0/435589928 >> v2:172.21.15.63:3300/0 conn(0x7f8a800732a0 0x7f8a800998b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:44.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.347+0000 7f8a862b4640 1 -- 172.21.15.63:0/435589928 >> 172.21.15.63:0/435589928 conn(0x7f8a800a1e70 msgr2=0x7f8a800a2280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:24:44.361 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.347+0000 7f8a862b4640 1 -- 172.21.15.63:0/435589928 shutdown_connections 2024-04-26T04:24:44.361 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.347+0000 7f8a862b4640 1 -- 172.21.15.63:0/435589928 wait complete. 2024-04-26T04:24:44.361 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.347+0000 7f8a862b4640 10 monclient: build_initial_monmap 2024-04-26T04:24:44.361 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.347+0000 7f8a862b4640 10 monclient: monmap: 2024-04-26T04:24:44.361 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:24:44.361 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:24:44.361 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:24:44.361 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:24:44.361 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:24:44.361 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:44.361 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:24:44.361 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:24:44.361 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:24:44.362 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:44.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.347+0000 7f8a862b4640 1 Processor -- start 2024-04-26T04:24:44.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.347+0000 7f8a862b4640 1 -- start start 2024-04-26T04:24:44.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.347+0000 7f8a862b4640 10 monclient: init 2024-04-26T04:24:44.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.347+0000 7f8a862b4640 10 monclient: _reopen_session rank -1 2024-04-26T04:24:44.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.347+0000 7f8a862b4640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T04:24:44.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.347+0000 7f8a862b4640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f8a80072ec0 0x7f8a8013f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:44.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.347+0000 7f8a862b4640 10 monclient(hunting): picked mon.noname-c con 0x7f8a80072ec0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:24:44.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.347+0000 7f8a862b4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8a800732a0 0x7f8a8013ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:44.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.347+0000 7f8a862b4640 10 monclient(hunting): picked mon.noname-b con 0x7f8a800732a0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:24:44.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.347+0000 7f8a862b4640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8a801413b0 0x7f8a80140470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:44.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.347+0000 7f8a862b4640 10 monclient(hunting): picked mon.noname-a con 0x7f8a801413b0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:24:44.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.347+0000 7f8a862b4640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:44.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.347+0000 7f8a862b4640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f8a800a0f90 con 0x7f8a801413b0 2024-04-26T04:24:44.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.347+0000 7f8a862b4640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:44.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.347+0000 7f8a862b4640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8a800a0e10 con 0x7f8a800732a0 2024-04-26T04:24:44.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.347+0000 7f8a862b4640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:44.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.347+0000 7f8a862b4640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f8a80099df0 con 0x7f8a80072ec0 2024-04-26T04:24:44.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.347+0000 7f8a862b4640 10 monclient(hunting): _renew_subs 2024-04-26T04:24:44.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.347+0000 7f8a862b4640 10 monclient(hunting): authenticate will time out at 1316.059820s 2024-04-26T04:24:44.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.347+0000 7f8a84851640 1 -- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8a801413b0 msgr2=0x7f8a80140470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:3300/0 2024-04-26T04:24:44.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.347+0000 7f8a84851640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8a801413b0 0x7f8a80140470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:24:44.363 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.347+0000 7f8a7ffff640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f8a80072ec0 0x7f8a8013f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:44.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.351+0000 7f8a7ffff640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f8a80072ec0 0x7f8a8013f9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:59354/0 (socket says 172.21.15.63:59354) 2024-04-26T04:24:44.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.351+0000 7f8a7ffff640 1 -- 172.21.15.63:0/243362984 learned_addr learned my addr 172.21.15.63:0/243362984 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:24:44.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.351+0000 7f8a7ffff640 10 monclient(hunting): get_auth_request con 0x7f8a80072ec0 auth_method 0 2024-04-26T04:24:44.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.351+0000 7f8a7ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:44.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.351+0000 7f8a7ffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:44.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.351+0000 7f8a7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:44.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.351+0000 7f8a7ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:44.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.351+0000 7f8a7ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:44.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.351+0000 7f8a7ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:24:44.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.351+0000 7f8a7f7fe640 1 --2- 172.21.15.63:0/243362984 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8a800732a0 0x7f8a8013ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:44.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.351+0000 7f8a7f7fe640 10 monclient(hunting): get_auth_request con 0x7f8a800732a0 auth_method 0 2024-04-26T04:24:44.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.351+0000 7f8a7f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:44.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.351+0000 7f8a7f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:44.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.351+0000 7f8a7f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:44.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.351+0000 7f8a7ffff640 10 monclient(hunting): handle_auth_done global_id 24284 payload 824 2024-04-26T04:24:44.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.351+0000 7f8a7ffff640 1 -- 172.21.15.63:0/243362984 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8a800732a0 msgr2=0x7f8a8013ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:44.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.351+0000 7f8a7ffff640 1 --2- 172.21.15.63:0/243362984 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8a800732a0 0x7f8a8013ff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:44.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.351+0000 7f8a7ffff640 1 -- 172.21.15.63:0/243362984 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8a801413b0 msgr2=0x7f8a80140470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T04:24:44.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.351+0000 7f8a7ffff640 1 --2- 172.21.15.63:0/243362984 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8a801413b0 0x7f8a80140470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:44.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.351+0000 7f8a7ffff640 10 monclient: _finish_hunting 0 2024-04-26T04:24:44.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.351+0000 7f8a7ffff640 1 monclient: found mon.noname-c 2024-04-26T04:24:44.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.351+0000 7f8a7ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:24:44.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.351+0000 7f8a7ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.63:3301/0 2024-04-26T04:24:44.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.351+0000 7f8a7ffff640 1 -- 172.21.15.63:0/243362984 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a80096ad0 con 0x7f8a80072ec0 2024-04-26T04:24:44.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.351+0000 7f8a7ffff640 1 -- 172.21.15.63:0/243362984 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f8a80072ec0 msgr2=0x7f8a8013f9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:24:44.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.351+0000 7f8a7ffff640 1 --2- 172.21.15.63:0/243362984 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f8a80072ec0 0x7f8a8013f9f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8a7800c7f0 tx=0x7f8a7800ccb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:44.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.351+0000 7f8a7cff9640 1 -- 172.21.15.63:0/243362984 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a78016020 con 0x7f8a80072ec0 2024-04-26T04:24:44.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.351+0000 7f8a7cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:44.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.351+0000 7f8a7cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:24:44.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.351+0000 7f8a7cff9640 10 monclient: dump: 2024-04-26T04:24:44.366 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:44.366 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:44.366 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:44.366 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:44.366 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:44.366 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:44.366 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:44.366 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:44.367 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:44.367 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:44.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.351+0000 7f8a7cff9640 10 monclient: _finish_auth 0 2024-04-26T04:24:44.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.351+0000 7f8a7cff9640 10 monclient: _check_auth_tickets 2024-04-26T04:24:44.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.351+0000 7f8a7cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:24:44.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.351+0000 7f8a7cff9640 1 -- 172.21.15.63:0/243362984 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a7800ac60 con 0x7f8a80072ec0 2024-04-26T04:24:44.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.351+0000 7f8a862b4640 5 monclient: authenticate success, global_id 24284 2024-04-26T04:24:44.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.351+0000 7f8a862b4640 10 monclient: _renew_subs 2024-04-26T04:24:44.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.351+0000 7f8a862b4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:24:44.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.351+0000 7f8a862b4640 1 -- 172.21.15.63:0/243362984 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a8009de90 con 0x7f8a80072ec0 2024-04-26T04:24:44.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.351+0000 7f8a7cff9640 10 monclient: handle_config config(0 keys) 2024-04-26T04:24:44.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.351+0000 7f8a7cff9640 1 -- 172.21.15.63:0/243362984 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a78012660 con 0x7f8a80072ec0 2024-04-26T04:24:44.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.351+0000 7f8a7cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:44.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.351+0000 7f8a7cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:24:44.368 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.351+0000 7f8a7cff9640 10 monclient: dump: 2024-04-26T04:24:44.368 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:44.368 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:44.368 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:44.368 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:44.368 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:44.368 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:44.368 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:44.368 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:44.368 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:44.368 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:44.368 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.351+0000 7f8a7dffb640 4 set_mon_vals no callback set 2024-04-26T04:24:44.368 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.351+0000 7f8a862b4640 10 monclient: _renew_subs 2024-04-26T04:24:44.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.351+0000 7f8a862b4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:24:44.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.351+0000 7f8a862b4640 1 -- 172.21.15.63:0/243362984 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a800967f0 con 0x7f8a80072ec0 2024-04-26T04:24:44.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.351+0000 7f8a862b4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:24:44.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.351+0000 7f8a862b4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:24:44.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.351+0000 7f8a862b4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:24:44.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.351+0000 7f8a862b4640 1 -- 172.21.15.63:0/243362984 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a80062e10 con 0x7f8a80072ec0 2024-04-26T04:24:44.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.355+0000 7f8a7cff9640 1 -- 172.21.15.63:0/243362984 <== mon.2 v2:172.21.15.63:3301/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f8a78019020 con 0x7f8a80072ec0 2024-04-26T04:24:44.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.355+0000 7f8a7cff9640 1 --2- 172.21.15.63:0/243362984 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f8a60035e70 0x7f8a60038330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:44.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.355+0000 7f8a7cff9640 1 -- 172.21.15.63:0/243362984 <== mon.2 v2:172.21.15.63:3301/0 5 ==== osd_map(222..222 src has 1..222) ==== 4738+0+0 (secure 0 0 0) 0x7f8a78049700 con 0x7f8a80072ec0 2024-04-26T04:24:44.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.355+0000 7f8a7f7fe640 1 --2- 172.21.15.63:0/243362984 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f8a60035e70 0x7f8a60038330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:44.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.355+0000 7f8a7f7fe640 10 monclient: get_auth_request con 0x7f8a60035e70 auth_method 0 2024-04-26T04:24:44.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.359+0000 7f8a7f7fe640 1 --2- 172.21.15.63:0/243362984 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f8a60035e70 0x7f8a60038330 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f8a68006fd0 tx=0x7f8a68008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:44.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.359+0000 7f8a7cff9640 1 -- 172.21.15.63:0/243362984 <== mon.2 v2:172.21.15.63:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f8a78020020 con 0x7f8a80072ec0 2024-04-26T04:24:44.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.359+0000 7f8a7cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:24:44.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.359+0000 7f8a7cff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:24:44.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.571+0000 7f8a862b4640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T04:24:44.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.571+0000 7f8a862b4640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T04:24:44.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.571+0000 7f8a862b4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:24:44.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.571+0000 7f8a862b4640 1 -- 172.21.15.63:0/243362984 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f8a80068430 con 0x7f8a80072ec0 2024-04-26T04:24:44.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.571+0000 7f8a7cff9640 1 -- 172.21.15.63:0/243362984 <== mon.2 v2:172.21.15.63:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f8a78023350 con 0x7f8a80072ec0 2024-04-26T04:24:44.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.571+0000 7f8a7cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T04:24:44.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.571+0000 7f8a7cff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:24:44.577 INFO:teuthology.orchestra.run.smithi063.stdout:{"election_epoch":26,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cd1c6a0c-4e4b-4014-89c0-2e535479ef39","modified":"2024-04-26T04:18:03.702339Z","created":"2024-04-26T04:18:03.702339Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3300","nonce":0},{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:3300","nonce":0},{"type":"v1","addr":"172.21.15.186:6789","nonce":0}]},"addr":"172.21.15.186:6789/0","public_addr":"172.21.15.186:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3301","nonce":0},{"type":"v1","addr":"172.21.15.63:6790","nonce":0}]},"addr":"172.21.15.63:6790/0","public_addr":"172.21.15.63:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T04:24:44.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.575+0000 7f8a862b4640 1 -- 172.21.15.63:0/243362984 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f8a60035e70 msgr2=0x7f8a60038330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:44.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.575+0000 7f8a862b4640 1 --2- 172.21.15.63:0/243362984 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f8a60035e70 0x7f8a60038330 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f8a68006fd0 tx=0x7f8a68008040 comp rx=0 tx=0).stop 2024-04-26T04:24:44.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.575+0000 7f8a862b4640 10 monclient: shutdown 2024-04-26T04:24:44.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.575+0000 7f8a862b4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:24:44.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.575+0000 7f8a862b4640 1 -- 172.21.15.63:0/243362984 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f8a80072ec0 msgr2=0x7f8a8013f9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:44.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.575+0000 7f8a862b4640 1 --2- 172.21.15.63:0/243362984 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f8a80072ec0 0x7f8a8013f9f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8a7800c7f0 tx=0x7f8a7800ccb0 comp rx=0 tx=0).stop 2024-04-26T04:24:44.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.575+0000 7f8a862b4640 1 -- 172.21.15.63:0/243362984 shutdown_connections 2024-04-26T04:24:44.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.575+0000 7f8a862b4640 1 --2- 172.21.15.63:0/243362984 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f8a60035e70 0x7f8a60038330 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:44.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.575+0000 7f8a862b4640 1 --2- 172.21.15.63:0/243362984 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8a801413b0 0x7f8a80140470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:44.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.575+0000 7f8a862b4640 1 --2- 172.21.15.63:0/243362984 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8a800732a0 0x7f8a8013ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:44.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.575+0000 7f8a862b4640 1 --2- 172.21.15.63:0/243362984 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f8a80072ec0 0x7f8a8013f9f0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:44.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.575+0000 7f8a862b4640 1 -- 172.21.15.63:0/243362984 >> 172.21.15.63:0/243362984 conn(0x7f8a800a1e70 msgr2=0x7f8a8009e4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:24:44.580 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.575+0000 7f8a862b4640 1 -- 172.21.15.63:0/243362984 shutdown_connections 2024-04-26T04:24:44.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:44.575+0000 7f8a862b4640 1 -- 172.21.15.63:0/243362984 wait complete. 2024-04-26T04:24:47.596 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T04:24:47.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.727+0000 7fadf6ab7640 10 monclient: get_monmap_and_config 2024-04-26T04:24:47.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.727+0000 7fadf6ab7640 10 monclient: build_initial_monmap 2024-04-26T04:24:47.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.727+0000 7fadf6ab7640 10 monclient: monmap: 2024-04-26T04:24:47.731 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:24:47.731 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:47.731 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:24:47.731291+0000 2024-04-26T04:24:47.731 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:24:47.731291+0000 2024-04-26T04:24:47.732 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:24:47.732 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:47.732 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:24:47.732 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:24:47.732 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:24:47.732 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:24:47.732 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:24:47.732 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:47.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.727+0000 7fadf6ab7640 1 Processor -- start 2024-04-26T04:24:47.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.727+0000 7fadf6ab7640 1 -- start start 2024-04-26T04:24:47.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.727+0000 7fadf6ab7640 10 monclient: init 2024-04-26T04:24:47.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.727+0000 7fadf6ab7640 10 monclient: _reopen_session rank -1 2024-04-26T04:24:47.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.727+0000 7fadf6ab7640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-04-26T04:24:47.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.727+0000 7fadf6ab7640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fadf009fe60 0x7fadf01372f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:47.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.727+0000 7fadf6ab7640 10 monclient(hunting): picked mon.noname-c con 0x7fadf009fe60 addr v2:172.21.15.186:3300/0 2024-04-26T04:24:47.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.727+0000 7fadf6ab7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fadf00a0240 addr v1:172.21.15.186:6789/0 2024-04-26T04:24:47.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.727+0000 7fadf6ab7640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fadf013c100 0x7fadf013e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:47.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.727+0000 7fadf6ab7640 10 monclient(hunting): picked mon.noname-b con 0x7fadf013c100 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:24:47.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.727+0000 7fadf6ab7640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fadf0137830 con 0x7fadf00a0240 2024-04-26T04:24:47.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.727+0000 7fadf6ab7640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:47.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.727+0000 7fadf6ab7640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fadf00a3730 con 0x7fadf009fe60 2024-04-26T04:24:47.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.727+0000 7fadf6ab7640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:47.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.727+0000 7fadf6ab7640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7fadf00a0c60 con 0x7fadf013c100 2024-04-26T04:24:47.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.727+0000 7fadf6ab7640 10 monclient(hunting): _renew_subs 2024-04-26T04:24:47.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.727+0000 7fadf6ab7640 10 monclient(hunting): authenticate will time out at 1319.439006s 2024-04-26T04:24:47.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.727+0000 7fadf5054640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fadf013c100 0x7fadf013e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:47.736 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.727+0000 7fadf5054640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fadf013c100 0x7fadf013e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:59368/0 (socket says 172.21.15.63:59368) 2024-04-26T04:24:47.736 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.727+0000 7fadf5054640 1 -- 172.21.15.63:0/982951914 learned_addr learned my addr 172.21.15.63:0/982951914 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:24:47.736 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.727+0000 7fadf5054640 10 monclient(hunting): get_auth_request con 0x7fadf013c100 auth_method 0 2024-04-26T04:24:47.736 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.727+0000 7fadf5054640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:47.736 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.727+0000 7fadf5054640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:47.736 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.727+0000 7fadf5054640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:47.736 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.727+0000 7fadf5054640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:47.736 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.727+0000 7fadf5054640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:47.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.727+0000 7fadf5054640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:24:47.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.727+0000 7fadf4853640 1 --2- 172.21.15.63:0/982951914 >> v2:172.21.15.186:3300/0 conn(0x7fadf009fe60 0x7fadf01372f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:47.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.727+0000 7fadf4853640 10 monclient(hunting): get_auth_request con 0x7fadf009fe60 auth_method 0 2024-04-26T04:24:47.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.727+0000 7fadf4853640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:47.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.727+0000 7fadf4853640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:47.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.727+0000 7fadf4853640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:47.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.727+0000 7fadf5054640 10 monclient(hunting): handle_auth_done global_id 24287 payload 274 2024-04-26T04:24:47.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.727+0000 7fadf5054640 1 -- 172.21.15.63:0/982951914 >> v2:172.21.15.186:3300/0 conn(0x7fadf009fe60 msgr2=0x7fadf01372f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:24:47.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.727+0000 7fadf5054640 1 --2- 172.21.15.63:0/982951914 >> v2:172.21.15.186:3300/0 conn(0x7fadf009fe60 0x7fadf01372f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:47.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.727+0000 7fadf5054640 1 -- 172.21.15.63:0/982951914 >> v1:172.21.15.186:6789/0 conn(0x7fadf00a0240 legacy=0x7fadf00a2a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:24:47.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.727+0000 7fadf5054640 10 monclient: _finish_hunting 0 2024-04-26T04:24:47.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.727+0000 7fadf5054640 1 monclient: found mon.noname-b 2024-04-26T04:24:47.738 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.727+0000 7fadf5054640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:24:47.738 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.727+0000 7fadf5054640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.63:3301/0 2024-04-26T04:24:47.738 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.727+0000 7fadf5054640 1 -- 172.21.15.63:0/982951914 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadf00a0f40 con 0x7fadf013c100 2024-04-26T04:24:47.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.731+0000 7fadf5054640 1 -- 172.21.15.63:0/982951914 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fadf013c100 msgr2=0x7fadf013e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:24:47.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.731+0000 7fadf5054640 1 --2- 172.21.15.63:0/982951914 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fadf013c100 0x7fadf013e4f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fade8009e50 tx=0x7fade801c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8cc0e19ee337f0e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:47.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.731+0000 7fadef7fe640 1 -- 172.21.15.63:0/982951914 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fade80a5720 con 0x7fadf013c100 2024-04-26T04:24:47.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.731+0000 7fadef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:47.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.731+0000 7fadef7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T04:24:47.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.731+0000 7fadef7fe640 10 monclient: dump: 2024-04-26T04:24:47.742 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:47.742 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:47.742 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:47.742 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:47.742 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:47.742 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:47.742 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:47.742 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:47.743 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:47.743 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:47.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.731+0000 7fadef7fe640 10 monclient: _finish_auth 0 2024-04-26T04:24:47.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.731+0000 7fadef7fe640 10 monclient: _check_auth_tickets 2024-04-26T04:24:47.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.731+0000 7fadef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:24:47.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.731+0000 7fadef7fe640 1 -- 172.21.15.63:0/982951914 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fade80a58c0 con 0x7fadf013c100 2024-04-26T04:24:47.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.731+0000 7fadef7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T04:24:47.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.731+0000 7fadef7fe640 1 -- 172.21.15.63:0/982951914 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fade80a5c90 con 0x7fadf013c100 2024-04-26T04:24:47.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.731+0000 7fadf6ab7640 5 monclient: authenticate success, global_id 24287 2024-04-26T04:24:47.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.731+0000 7fadf6ab7640 10 monclient: get_monmap_and_config success 2024-04-26T04:24:47.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.731+0000 7fadf6ab7640 4 set_mon_vals no callback set 2024-04-26T04:24:47.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.731+0000 7fadef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:47.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.731+0000 7fadef7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:24:47.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.731+0000 7fadef7fe640 10 monclient: dump: 2024-04-26T04:24:47.744 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:47.744 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:47.744 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:47.744 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:47.744 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:47.744 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:47.744 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:47.745 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:47.745 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:47.745 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:47.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.735+0000 7fadf6ab7640 10 monclient: shutdown 2024-04-26T04:24:47.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.735+0000 7fadf6ab7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:24:47.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.735+0000 7fadf6ab7640 1 -- 172.21.15.63:0/982951914 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fadf013c100 msgr2=0x7fadf013e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:47.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.735+0000 7fadf6ab7640 1 --2- 172.21.15.63:0/982951914 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fadf013c100 0x7fadf013e4f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fade8009e50 tx=0x7fade801c880 comp rx=0 tx=0).stop 2024-04-26T04:24:47.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.735+0000 7fadf6ab7640 1 -- 172.21.15.63:0/982951914 shutdown_connections 2024-04-26T04:24:47.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.735+0000 7fadf6ab7640 1 --2- 172.21.15.63:0/982951914 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fadf013c100 0x7fadf013e4f0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:47.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.735+0000 7fadf6ab7640 1 --2- 172.21.15.63:0/982951914 >> v2:172.21.15.186:3300/0 conn(0x7fadf009fe60 0x7fadf01372f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:47.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.735+0000 7fadf6ab7640 1 -- 172.21.15.63:0/982951914 >> 172.21.15.63:0/982951914 conn(0x7fadf00a5fb0 msgr2=0x7fadf00a63c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:24:47.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.735+0000 7fadf6ab7640 1 -- 172.21.15.63:0/982951914 shutdown_connections 2024-04-26T04:24:47.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.735+0000 7fadf6ab7640 1 -- 172.21.15.63:0/982951914 wait complete. 2024-04-26T04:24:47.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.735+0000 7fadf6ab7640 10 monclient: build_initial_monmap 2024-04-26T04:24:47.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.735+0000 7fadf6ab7640 10 monclient: monmap: 2024-04-26T04:24:47.746 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:24:47.746 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:24:47.746 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:24:47.746 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:24:47.746 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:24:47.746 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:47.747 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:24:47.747 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:24:47.747 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:24:47.747 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:47.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.735+0000 7fadf6ab7640 1 Processor -- start 2024-04-26T04:24:47.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.735+0000 7fadf6ab7640 1 -- start start 2024-04-26T04:24:47.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.735+0000 7fadf6ab7640 10 monclient: init 2024-04-26T04:24:47.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.735+0000 7fadf6ab7640 10 monclient: _reopen_session rank -1 2024-04-26T04:24:47.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.735+0000 7fadf6ab7640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T04:24:47.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.735+0000 7fadf6ab7640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fadf009fe60 0x7fadf0072500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:47.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.735+0000 7fadf6ab7640 10 monclient(hunting): picked mon.noname-a con 0x7fadf009fe60 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:24:47.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.735+0000 7fadf6ab7640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fadf00a0240 0x7fadf0072a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:47.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.735+0000 7fadf6ab7640 10 monclient(hunting): picked mon.noname-c con 0x7fadf00a0240 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:24:47.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.735+0000 7fadf6ab7640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fadf013c100 0x7fadf006e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:47.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.735+0000 7fadf6ab7640 10 monclient(hunting): picked mon.noname-b con 0x7fadf013c100 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:24:47.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.735+0000 7fadf6ab7640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:47.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.735+0000 7fadf6ab7640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fadf01416a0 con 0x7fadf009fe60 2024-04-26T04:24:47.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.735+0000 7fadf6ab7640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:47.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.735+0000 7fadf6ab7640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fadf0141520 con 0x7fadf013c100 2024-04-26T04:24:47.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.735+0000 7fadf6ab7640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:47.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.735+0000 7fadf6ab7640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7fadf0141820 con 0x7fadf00a0240 2024-04-26T04:24:47.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.735+0000 7fadf6ab7640 10 monclient(hunting): _renew_subs 2024-04-26T04:24:47.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.735+0000 7fadf6ab7640 10 monclient(hunting): authenticate will time out at 1319.446180s 2024-04-26T04:24:47.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.735+0000 7fadf4853640 1 -- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fadf009fe60 msgr2=0x7fadf0072500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:3300/0 2024-04-26T04:24:47.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.735+0000 7fadeffff640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fadf00a0240 0x7fadf0072a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:47.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.735+0000 7fadf4853640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fadf009fe60 0x7fadf0072500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:24:47.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.735+0000 7fadeffff640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fadf00a0240 0x7fadf0072a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:59376/0 (socket says 172.21.15.63:59376) 2024-04-26T04:24:47.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.735+0000 7fadeffff640 1 -- 172.21.15.63:0/2365925872 learned_addr learned my addr 172.21.15.63:0/2365925872 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:24:47.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.735+0000 7fadeffff640 10 monclient(hunting): get_auth_request con 0x7fadf00a0240 auth_method 0 2024-04-26T04:24:47.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.735+0000 7fadeffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:47.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.735+0000 7fadeffff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:47.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.735+0000 7fadeffff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:47.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.735+0000 7fadeffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:47.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.735+0000 7fadeffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:47.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.735+0000 7fadeffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:24:47.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.735+0000 7fadeffff640 10 monclient(hunting): handle_auth_done global_id 24290 payload 824 2024-04-26T04:24:47.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.735+0000 7fadf5054640 1 --2- 172.21.15.63:0/2365925872 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fadf013c100 0x7fadf006e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:47.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.735+0000 7fadeffff640 1 -- 172.21.15.63:0/2365925872 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fadf013c100 msgr2=0x7fadf006e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:47.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.735+0000 7fadeffff640 1 --2- 172.21.15.63:0/2365925872 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fadf013c100 0x7fadf006e320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:47.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.735+0000 7fadeffff640 1 -- 172.21.15.63:0/2365925872 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fadf009fe60 msgr2=0x7fadf0072500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T04:24:47.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.735+0000 7fadeffff640 1 --2- 172.21.15.63:0/2365925872 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fadf009fe60 0x7fadf0072500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:47.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.735+0000 7fadeffff640 10 monclient: _finish_hunting 0 2024-04-26T04:24:47.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.735+0000 7fadeffff640 1 monclient: found mon.noname-c 2024-04-26T04:24:47.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.735+0000 7fadeffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:24:47.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.735+0000 7fadeffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.63:3301/0 2024-04-26T04:24:47.751 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.735+0000 7fadeffff640 1 -- 172.21.15.63:0/2365925872 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadf013b970 con 0x7fadf00a0240 2024-04-26T04:24:47.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.739+0000 7fadeffff640 1 -- 172.21.15.63:0/2365925872 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fadf00a0240 msgr2=0x7fadf0072a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:24:47.753 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.739+0000 7fadeffff640 1 --2- 172.21.15.63:0/2365925872 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fadf00a0240 0x7fadf0072a40 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fadd800c410 tx=0x7fadd800c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:47.753 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.739+0000 7faded7fa640 1 -- 172.21.15.63:0/2365925872 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadd80119a0 con 0x7fadf00a0240 2024-04-26T04:24:47.753 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.739+0000 7faded7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:47.753 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.739+0000 7faded7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:24:47.753 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.739+0000 7faded7fa640 10 monclient: dump: 2024-04-26T04:24:47.753 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:47.753 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:47.753 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:47.753 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:47.753 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:47.754 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:47.754 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:47.754 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:47.754 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:47.754 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:47.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.739+0000 7faded7fa640 10 monclient: _finish_auth 0 2024-04-26T04:24:47.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.739+0000 7faded7fa640 10 monclient: _check_auth_tickets 2024-04-26T04:24:47.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.739+0000 7faded7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:24:47.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.739+0000 7faded7fa640 1 -- 172.21.15.63:0/2365925872 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fadd8017070 con 0x7fadf00a0240 2024-04-26T04:24:47.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.739+0000 7fadf6ab7640 5 monclient: authenticate success, global_id 24290 2024-04-26T04:24:47.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.739+0000 7fadf6ab7640 10 monclient: _renew_subs 2024-04-26T04:24:47.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.739+0000 7fadf6ab7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:24:47.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.739+0000 7fadf6ab7640 1 -- 172.21.15.63:0/2365925872 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadf0070fd0 con 0x7fadf00a0240 2024-04-26T04:24:47.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.739+0000 7faded7fa640 10 monclient: handle_config config(0 keys) 2024-04-26T04:24:47.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.739+0000 7faded7fa640 1 -- 172.21.15.63:0/2365925872 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadd8011d80 con 0x7fadf00a0240 2024-04-26T04:24:47.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.739+0000 7faded7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:47.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.739+0000 7faded7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:24:47.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.739+0000 7faded7fa640 10 monclient: dump: 2024-04-26T04:24:47.755 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:47.755 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:47.756 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:47.756 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:47.756 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:47.756 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:47.756 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:47.756 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:47.756 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:47.756 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:47.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.739+0000 7fadee7fc640 4 set_mon_vals no callback set 2024-04-26T04:24:47.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.739+0000 7fadf6ab7640 10 monclient: _renew_subs 2024-04-26T04:24:47.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.739+0000 7fadf6ab7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:24:47.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.739+0000 7fadf6ab7640 1 -- 172.21.15.63:0/2365925872 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fadf0070cf0 con 0x7fadf00a0240 2024-04-26T04:24:47.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.739+0000 7fadf6ab7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:24:47.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.739+0000 7fadf6ab7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:24:47.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.739+0000 7fadf6ab7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:24:47.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.739+0000 7faded7fa640 1 -- 172.21.15.63:0/2365925872 <== mon.2 v2:172.21.15.63:3301/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fadd801c460 con 0x7fadf00a0240 2024-04-26T04:24:47.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.739+0000 7fadf6ab7640 1 -- 172.21.15.63:0/2365925872 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fadf006f600 con 0x7fadf00a0240 2024-04-26T04:24:47.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.739+0000 7faded7fa640 1 --2- 172.21.15.63:0/2365925872 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fadd0035e70 0x7fadd0038330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:47.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.739+0000 7faded7fa640 1 -- 172.21.15.63:0/2365925872 <== mon.2 v2:172.21.15.63:3301/0 5 ==== osd_map(222..222 src has 1..222) ==== 4738+0+0 (secure 0 0 0) 0x7fadd8048e20 con 0x7fadf00a0240 2024-04-26T04:24:47.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.739+0000 7fadf4853640 1 --2- 172.21.15.63:0/2365925872 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fadd0035e70 0x7fadd0038330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:47.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.739+0000 7fadf4853640 10 monclient: get_auth_request con 0x7fadd0035e70 auth_method 0 2024-04-26T04:24:47.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.743+0000 7fadf4853640 1 --2- 172.21.15.63:0/2365925872 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fadd0035e70 0x7fadd0038330 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fade0006e90 tx=0x7fade0006e20 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:47.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.743+0000 7faded7fa640 1 -- 172.21.15.63:0/2365925872 <== mon.2 v2:172.21.15.63:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fadd801c9e0 con 0x7fadf00a0240 2024-04-26T04:24:47.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.743+0000 7faded7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:24:47.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.743+0000 7faded7fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:24:47.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.967+0000 7fadf6ab7640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T04:24:47.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.967+0000 7fadf6ab7640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T04:24:47.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.967+0000 7fadf6ab7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:24:47.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.967+0000 7fadf6ab7640 1 -- 172.21.15.63:0/2365925872 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fadf0062e10 con 0x7fadf00a0240 2024-04-26T04:24:47.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.971+0000 7faded7fa640 1 -- 172.21.15.63:0/2365925872 <== mon.2 v2:172.21.15.63:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7fadd8047840 con 0x7fadf00a0240 2024-04-26T04:24:47.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.971+0000 7faded7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T04:24:47.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.971+0000 7faded7fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:24:47.974 INFO:teuthology.orchestra.run.smithi063.stdout:{"election_epoch":26,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cd1c6a0c-4e4b-4014-89c0-2e535479ef39","modified":"2024-04-26T04:18:03.702339Z","created":"2024-04-26T04:18:03.702339Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3300","nonce":0},{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:3300","nonce":0},{"type":"v1","addr":"172.21.15.186:6789","nonce":0}]},"addr":"172.21.15.186:6789/0","public_addr":"172.21.15.186:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3301","nonce":0},{"type":"v1","addr":"172.21.15.63:6790","nonce":0}]},"addr":"172.21.15.63:6790/0","public_addr":"172.21.15.63:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T04:24:47.977 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.971+0000 7fadf6ab7640 1 -- 172.21.15.63:0/2365925872 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fadd0035e70 msgr2=0x7fadd0038330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:47.977 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.971+0000 7fadf6ab7640 1 --2- 172.21.15.63:0/2365925872 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fadd0035e70 0x7fadd0038330 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fade0006e90 tx=0x7fade0006e20 comp rx=0 tx=0).stop 2024-04-26T04:24:47.977 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.971+0000 7fadf6ab7640 10 monclient: shutdown 2024-04-26T04:24:47.977 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.971+0000 7fadf6ab7640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:24:47.977 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.971+0000 7fadf6ab7640 1 -- 172.21.15.63:0/2365925872 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fadf00a0240 msgr2=0x7fadf0072a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:47.977 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.971+0000 7fadf6ab7640 1 --2- 172.21.15.63:0/2365925872 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fadf00a0240 0x7fadf0072a40 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fadd800c410 tx=0x7fadd800c8d0 comp rx=0 tx=0).stop 2024-04-26T04:24:47.977 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.971+0000 7fadf6ab7640 1 -- 172.21.15.63:0/2365925872 shutdown_connections 2024-04-26T04:24:47.977 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.971+0000 7fadf6ab7640 1 --2- 172.21.15.63:0/2365925872 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fadd0035e70 0x7fadd0038330 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:47.977 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.971+0000 7fadf6ab7640 1 --2- 172.21.15.63:0/2365925872 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fadf013c100 0x7fadf006e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:47.978 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.971+0000 7fadf6ab7640 1 --2- 172.21.15.63:0/2365925872 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fadf00a0240 0x7fadf0072a40 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:47.978 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.971+0000 7fadf6ab7640 1 --2- 172.21.15.63:0/2365925872 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fadf009fe60 0x7fadf0072500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:47.978 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.971+0000 7fadf6ab7640 1 -- 172.21.15.63:0/2365925872 >> 172.21.15.63:0/2365925872 conn(0x7fadf00a5fb0 msgr2=0x7fadf0135ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:24:47.978 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.971+0000 7fadf6ab7640 1 -- 172.21.15.63:0/2365925872 shutdown_connections 2024-04-26T04:24:47.978 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:47.971+0000 7fadf6ab7640 1 -- 172.21.15.63:0/2365925872 wait complete. 2024-04-26T04:24:50.994 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T04:24:51.127 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:51.123+0000 7f9ccf818640 10 monclient: get_monmap_and_config 2024-04-26T04:24:51.127 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:51.123+0000 7f9ccf818640 10 monclient: build_initial_monmap 2024-04-26T04:24:51.127 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:51.123+0000 7f9ccf818640 10 monclient: monmap: 2024-04-26T04:24:51.127 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:24:51.127 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:51.127 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:24:51.127165+0000 2024-04-26T04:24:51.127 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:24:51.127165+0000 2024-04-26T04:24:51.127 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:24:51.127 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:51.127 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:24:51.127 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:24:51.128 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:24:51.128 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:24:51.128 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:24:51.128 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:51.130 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:51.123+0000 7f9ccf818640 1 Processor -- start 2024-04-26T04:24:51.130 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:51.123+0000 7f9ccf818640 1 -- start start 2024-04-26T04:24:51.130 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:51.123+0000 7f9ccf818640 10 monclient: init 2024-04-26T04:24:51.130 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:51.123+0000 7f9ccf818640 10 monclient: _reopen_session rank -1 2024-04-26T04:24:51.130 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:51.123+0000 7f9ccf818640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-04-26T04:24:51.130 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:51.123+0000 7f9ccf818640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f9cc80a2c60 0x7f9cc80a3040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:51.130 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:51.123+0000 7f9ccf818640 10 monclient(hunting): picked mon.noname-c con 0x7f9cc80a2c60 addr v2:172.21.15.186:3300/0 2024-04-26T04:24:51.130 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:51.123+0000 7f9ccf818640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f9cc80a20c0 0x7f9cc80a3db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:51.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:51.123+0000 7f9ccf818640 10 monclient(hunting): picked mon.noname-a con 0x7f9cc80a20c0 addr v2:172.21.15.63:3300/0 2024-04-26T04:24:51.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:51.123+0000 7f9ccf818640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9cc80a24a0 addr v1:172.21.15.186:6789/0 2024-04-26T04:24:51.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:51.123+0000 7f9ccf818640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9cc80a5010 con 0x7f9cc80a24a0 2024-04-26T04:24:51.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:51.123+0000 7f9ccf818640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:51.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:51.123+0000 7f9ccf818640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7f9cc80a0df0 con 0x7f9cc80a20c0 2024-04-26T04:24:51.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:51.123+0000 7f9ccf818640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:51.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:51.123+0000 7f9ccf818640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f9cc80a13b0 con 0x7f9cc80a2c60 2024-04-26T04:24:51.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:51.123+0000 7f9ccf818640 10 monclient(hunting): _renew_subs 2024-04-26T04:24:51.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:51.123+0000 7f9ccf818640 10 monclient(hunting): authenticate will time out at 1322.834897s 2024-04-26T04:24:51.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:51.123+0000 7f9cccdb3640 1 -- >> v2:172.21.15.63:3300/0 conn(0x7f9cc80a20c0 msgr2=0x7f9cc80a3db0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.63:3300/0 2024-04-26T04:24:51.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:51.123+0000 7f9cccdb3640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f9cc80a20c0 0x7f9cc80a3db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:24:51.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:51.123+0000 7f9ccd5b4640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f9cc80a2c60 0x7f9cc80a3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:51.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:51.123+0000 7f9ccd5b4640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f9cc80a2c60 0x7f9cc80a3040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.63:39426/0 (socket says 172.21.15.63:39426) 2024-04-26T04:24:51.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:51.123+0000 7f9ccd5b4640 1 -- 172.21.15.63:0/898483349 learned_addr learned my addr 172.21.15.63:0/898483349 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:24:51.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:51.123+0000 7f9ccd5b4640 10 monclient(hunting): get_auth_request con 0x7f9cc80a2c60 auth_method 0 2024-04-26T04:24:51.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:51.123+0000 7f9ccd5b4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:51.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:51.123+0000 7f9ccd5b4640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:51.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:51.123+0000 7f9ccd5b4640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:51.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:51.123+0000 7f9ccddb5640 1 --1- 172.21.15.63:0/898483349 >> v1:172.21.15.186:6789/0 conn(0x7f9cc80a24a0 0x7f9cc80a42f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.63:38074/0 (socket says 172.21.15.63:38074) 2024-04-26T04:24:51.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:51.123+0000 7f9ccd5b4640 1 -- 172.21.15.63:0/898483349 >> v2:172.21.15.186:3300/0 conn(0x7f9cc80a2c60 msgr2=0x7f9cc80a3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T04:24:51.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:51.123+0000 7f9ccd5b4640 1 -- 172.21.15.63:0/898483349 >> v2:172.21.15.186:3300/0 conn(0x7f9cc80a2c60 msgr2=0x7f9cc80a3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:24:51.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:51.123+0000 7f9ccd5b4640 1 --2- 172.21.15.63:0/898483349 >> v2:172.21.15.186:3300/0 conn(0x7f9cc80a2c60 0x7f9cc80a3040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:24:51.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:51.123+0000 7f9ccd5b4640 1 --2- 172.21.15.63:0/898483349 >> v2:172.21.15.186:3300/0 conn(0x7f9cc80a2c60 0x7f9cc80a3040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:24:51.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:51.127+0000 7f9ccddb5640 1 -- 172.21.15.63:0/898483349 >> v1:172.21.15.186:6789/0 conn(0x7f9cc80a24a0 legacy=0x7f9cc80a42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:24:51.329 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:51.327+0000 7f9cccdb3640 1 -- 172.21.15.63:0/898483349 >> v2:172.21.15.63:3300/0 conn(0x7f9cc80a20c0 msgr2=0x7f9cc80a3db0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.63:3300/0 2024-04-26T04:24:51.329 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:51.327+0000 7f9cccdb3640 1 --2- 172.21.15.63:0/898483349 >> v2:172.21.15.63:3300/0 conn(0x7f9cc80a20c0 0x7f9cc80a3db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T04:24:51.330 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:51.327+0000 7f9ccd5b4640 1 --2- 172.21.15.63:0/898483349 >> v2:172.21.15.186:3300/0 conn(0x7f9cc80a2c60 0x7f9cc80a3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:51.330 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:51.327+0000 7f9ccd5b4640 10 monclient(hunting): get_auth_request con 0x7f9cc80a2c60 auth_method 0 2024-04-26T04:24:51.330 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:51.327+0000 7f9ccd5b4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:51.330 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:51.327+0000 7f9ccd5b4640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:51.330 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:51.327+0000 7f9ccd5b4640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:24:51.330 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:51.327+0000 7f9ccd5b4640 1 -- 172.21.15.63:0/898483349 >> v2:172.21.15.186:3300/0 conn(0x7f9cc80a2c60 msgr2=0x7f9cc80a3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T04:24:51.330 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:51.327+0000 7f9ccd5b4640 1 -- 172.21.15.63:0/898483349 >> v2:172.21.15.186:3300/0 conn(0x7f9cc80a2c60 msgr2=0x7f9cc80a3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:24:51.331 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:51.327+0000 7f9ccd5b4640 1 --2- 172.21.15.63:0/898483349 >> v2:172.21.15.186:3300/0 conn(0x7f9cc80a2c60 0x7f9cc80a3040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:24:51.331 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:51.327+0000 7f9ccd5b4640 1 --2- 172.21.15.63:0/898483349 >> v2:172.21.15.186:3300/0 conn(0x7f9cc80a2c60 0x7f9cc80a3040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T04:24:51.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:51.731+0000 7f9cccdb3640 1 -- 172.21.15.63:0/898483349 >> v2:172.21.15.63:3300/0 conn(0x7f9cc80a20c0 msgr2=0x7f9cc80a3db0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.63:3300/0 2024-04-26T04:24:51.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:51.731+0000 7f9cccdb3640 1 --2- 172.21.15.63:0/898483349 >> v2:172.21.15.63:3300/0 conn(0x7f9cc80a20c0 0x7f9cc80a3db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T04:24:51.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:51.731+0000 7f9ccd5b4640 1 --2- 172.21.15.63:0/898483349 >> v2:172.21.15.186:3300/0 conn(0x7f9cc80a2c60 0x7f9cc80a3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:51.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:51.731+0000 7f9ccd5b4640 10 monclient(hunting): get_auth_request con 0x7f9cc80a2c60 auth_method 0 2024-04-26T04:24:51.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:51.731+0000 7f9ccd5b4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:51.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:51.731+0000 7f9ccd5b4640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:51.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:51.731+0000 7f9ccd5b4640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:24:51.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:51.731+0000 7f9ccd5b4640 1 -- 172.21.15.63:0/898483349 >> v2:172.21.15.186:3300/0 conn(0x7f9cc80a2c60 msgr2=0x7f9cc80a3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T04:24:51.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:51.731+0000 7f9ccd5b4640 1 -- 172.21.15.63:0/898483349 >> v2:172.21.15.186:3300/0 conn(0x7f9cc80a2c60 msgr2=0x7f9cc80a3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:24:51.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:51.731+0000 7f9ccd5b4640 1 --2- 172.21.15.63:0/898483349 >> v2:172.21.15.186:3300/0 conn(0x7f9cc80a2c60 0x7f9cc80a3040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:24:51.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:51.731+0000 7f9ccd5b4640 1 --2- 172.21.15.63:0/898483349 >> v2:172.21.15.186:3300/0 conn(0x7f9cc80a2c60 0x7f9cc80a3040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T04:24:52.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:52.535+0000 7f9cccdb3640 1 -- 172.21.15.63:0/898483349 >> v2:172.21.15.63:3300/0 conn(0x7f9cc80a20c0 msgr2=0x7f9cc80a3db0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.63:3300/0 2024-04-26T04:24:52.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:52.535+0000 7f9cccdb3640 1 --2- 172.21.15.63:0/898483349 >> v2:172.21.15.63:3300/0 conn(0x7f9cc80a20c0 0x7f9cc80a3db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T04:24:52.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:52.535+0000 7f9ccd5b4640 1 --2- 172.21.15.63:0/898483349 >> v2:172.21.15.186:3300/0 conn(0x7f9cc80a2c60 0x7f9cc80a3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:52.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:52.535+0000 7f9ccd5b4640 10 monclient(hunting): get_auth_request con 0x7f9cc80a2c60 auth_method 0 2024-04-26T04:24:52.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:52.535+0000 7f9ccd5b4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:52.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:52.535+0000 7f9ccd5b4640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:52.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:52.535+0000 7f9ccd5b4640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:24:52.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:52.535+0000 7f9ccd5b4640 1 -- 172.21.15.63:0/898483349 >> v2:172.21.15.186:3300/0 conn(0x7f9cc80a2c60 msgr2=0x7f9cc80a3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T04:24:52.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:52.535+0000 7f9ccd5b4640 1 -- 172.21.15.63:0/898483349 >> v2:172.21.15.186:3300/0 conn(0x7f9cc80a2c60 msgr2=0x7f9cc80a3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:24:52.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:52.535+0000 7f9ccd5b4640 1 --2- 172.21.15.63:0/898483349 >> v2:172.21.15.186:3300/0 conn(0x7f9cc80a2c60 0x7f9cc80a3040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:24:52.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:52.535+0000 7f9ccd5b4640 1 --2- 172.21.15.63:0/898483349 >> v2:172.21.15.186:3300/0 conn(0x7f9cc80a2c60 0x7f9cc80a3040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T04:24:54.129 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:54.123+0000 7f9cbeffd640 10 monclient(hunting): tick 2024-04-26T04:24:54.129 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:54.123+0000 7f9cbeffd640 10 monclient(hunting): _check_auth_tickets 2024-04-26T04:24:54.130 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:54.123+0000 7f9cbeffd640 1 monclient(hunting): continuing hunt 2024-04-26T04:24:54.130 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:54.123+0000 7f9cbeffd640 10 monclient(hunting): _reopen_session rank -1 2024-04-26T04:24:54.130 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:54.123+0000 7f9cbeffd640 1 -- 172.21.15.63:0/898483349 >> v2:172.21.15.186:3300/0 conn(0x7f9cc80a2c60 msgr2=0x7f9cc80a3040 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T04:24:54.130 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:54.123+0000 7f9cbeffd640 1 --2- 172.21.15.63:0/898483349 >> v2:172.21.15.186:3300/0 conn(0x7f9cc80a2c60 0x7f9cc80a3040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:54.130 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:54.123+0000 7f9cbeffd640 1 -- 172.21.15.63:0/898483349 >> v2:172.21.15.63:3300/0 conn(0x7f9cc80a20c0 msgr2=0x7f9cc80a3db0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T04:24:54.130 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:54.123+0000 7f9cbeffd640 1 --2- 172.21.15.63:0/898483349 >> v2:172.21.15.63:3300/0 conn(0x7f9cc80a20c0 0x7f9cc80a3db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:54.130 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:54.123+0000 7f9cbeffd640 1 -- 172.21.15.63:0/898483349 >> v1:172.21.15.186:6789/0 conn(0x7f9cc80a24a0 legacy=0x7f9cc80a42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:54.130 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:54.123+0000 7f9cbeffd640 10 monclient: _add_conns ranks=[0,2] 2024-04-26T04:24:54.130 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:54.123+0000 7f9cbeffd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9cac003740 addr v1:172.21.15.63:6789/0 2024-04-26T04:24:54.130 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:54.123+0000 7f9cbeffd640 1 --2- 172.21.15.63:0/898483349 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9cac007410 0x7f9cac0098d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:54.130 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:54.123+0000 7f9cbeffd640 10 monclient(hunting): picked mon.noname-b con 0x7f9cac007410 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:24:54.130 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:54.123+0000 7f9cbeffd640 1 -- 172.21.15.63:0/898483349 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9cc80a10d0 con 0x7f9cac003740 2024-04-26T04:24:54.130 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:54.123+0000 7f9cbeffd640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:54.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:54.123+0000 7f9cbeffd640 1 -- 172.21.15.63:0/898483349 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f9cc80a0df0 con 0x7f9cac007410 2024-04-26T04:24:54.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:54.123+0000 7f9cbeffd640 10 monclient(hunting): _renew_subs 2024-04-26T04:24:54.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:54.123+0000 7f9ccd5b4640 1 -- 172.21.15.63:0/898483349 >> v1:172.21.15.63:6789/0 conn(0x7f9cac003740 legacy=0x7f9cac005ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-04-26T04:24:54.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:54.123+0000 7f9cccdb3640 1 --2- 172.21.15.63:0/898483349 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9cac007410 0x7f9cac0098d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:54.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:54.123+0000 7f9cccdb3640 10 monclient(hunting): get_auth_request con 0x7f9cac007410 auth_method 0 2024-04-26T04:24:54.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:54.123+0000 7f9cccdb3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:54.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:54.123+0000 7f9cccdb3640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:54.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:54.123+0000 7f9cccdb3640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:54.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:54.123+0000 7f9cccdb3640 1 -- 172.21.15.63:0/898483349 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9cac007410 msgr2=0x7f9cac0098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T04:24:54.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:54.123+0000 7f9cccdb3640 1 -- 172.21.15.63:0/898483349 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9cac007410 msgr2=0x7f9cac0098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:24:54.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:54.123+0000 7f9cccdb3640 1 --2- 172.21.15.63:0/898483349 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9cac007410 0x7f9cac0098d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:24:54.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:54.123+0000 7f9cccdb3640 1 --2- 172.21.15.63:0/898483349 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9cac007410 0x7f9cac0098d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:24:54.329 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:54.327+0000 7f9ccd5b4640 1 -- 172.21.15.63:0/898483349 >> v1:172.21.15.63:6789/0 conn(0x7f9cac003740 legacy=0x7f9cac005ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-04-26T04:24:54.330 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:54.327+0000 7f9cccdb3640 1 --2- 172.21.15.63:0/898483349 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9cac007410 0x7f9cac0098d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:54.330 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:54.327+0000 7f9cccdb3640 10 monclient(hunting): get_auth_request con 0x7f9cac007410 auth_method 0 2024-04-26T04:24:54.331 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:54.327+0000 7f9cccdb3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:54.331 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:54.327+0000 7f9cccdb3640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:54.331 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:54.327+0000 7f9cccdb3640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:24:54.331 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:54.327+0000 7f9cccdb3640 1 -- 172.21.15.63:0/898483349 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9cac007410 msgr2=0x7f9cac0098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T04:24:54.331 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:54.327+0000 7f9cccdb3640 1 -- 172.21.15.63:0/898483349 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9cac007410 msgr2=0x7f9cac0098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:24:54.331 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:54.327+0000 7f9cccdb3640 1 --2- 172.21.15.63:0/898483349 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9cac007410 0x7f9cac0098d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:24:54.331 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:54.327+0000 7f9cccdb3640 1 --2- 172.21.15.63:0/898483349 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9cac007410 0x7f9cac0098d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T04:24:54.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:54.731+0000 7f9ccd5b4640 1 -- 172.21.15.63:0/898483349 >> v1:172.21.15.63:6789/0 conn(0x7f9cac003740 legacy=0x7f9cac005ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-04-26T04:24:54.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:54.731+0000 7f9cccdb3640 1 --2- 172.21.15.63:0/898483349 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9cac007410 0x7f9cac0098d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:54.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:54.731+0000 7f9cccdb3640 10 monclient(hunting): get_auth_request con 0x7f9cac007410 auth_method 0 2024-04-26T04:24:54.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:54.731+0000 7f9cccdb3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:54.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:54.731+0000 7f9cccdb3640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:54.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:54.731+0000 7f9cccdb3640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:24:54.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:54.731+0000 7f9cccdb3640 1 -- 172.21.15.63:0/898483349 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9cac007410 msgr2=0x7f9cac0098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T04:24:54.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:54.731+0000 7f9cccdb3640 1 -- 172.21.15.63:0/898483349 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9cac007410 msgr2=0x7f9cac0098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:24:54.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:54.731+0000 7f9cccdb3640 1 --2- 172.21.15.63:0/898483349 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9cac007410 0x7f9cac0098d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:24:54.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:54.731+0000 7f9cccdb3640 1 --2- 172.21.15.63:0/898483349 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9cac007410 0x7f9cac0098d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T04:24:55.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:55.535+0000 7f9ccd5b4640 1 -- 172.21.15.63:0/898483349 >> v1:172.21.15.63:6789/0 conn(0x7f9cac003740 legacy=0x7f9cac005ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-04-26T04:24:55.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:55.535+0000 7f9cccdb3640 1 --2- 172.21.15.63:0/898483349 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9cac007410 0x7f9cac0098d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:55.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:55.535+0000 7f9cccdb3640 10 monclient(hunting): get_auth_request con 0x7f9cac007410 auth_method 0 2024-04-26T04:24:55.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:55.535+0000 7f9cccdb3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:55.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:55.535+0000 7f9cccdb3640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:55.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:55.535+0000 7f9cccdb3640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:24:55.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:55.535+0000 7f9cccdb3640 1 -- 172.21.15.63:0/898483349 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9cac007410 msgr2=0x7f9cac0098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T04:24:55.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:55.535+0000 7f9cccdb3640 1 -- 172.21.15.63:0/898483349 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9cac007410 msgr2=0x7f9cac0098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:24:55.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:55.535+0000 7f9cccdb3640 1 --2- 172.21.15.63:0/898483349 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9cac007410 0x7f9cac0098d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:24:55.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:55.535+0000 7f9cccdb3640 1 --2- 172.21.15.63:0/898483349 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9cac007410 0x7f9cac0098d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T04:24:56.527 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[30636]: starting. 2024-04-26T04:24:56.527 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[30636]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-30634 2024-04-26T04:24:56.527 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[30636]: shutting down. 2024-04-26T04:24:57.130 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.123+0000 7f9cbeffd640 10 monclient(hunting): tick 2024-04-26T04:24:57.130 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.123+0000 7f9cbeffd640 10 monclient(hunting): _check_auth_tickets 2024-04-26T04:24:57.130 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.123+0000 7f9cbeffd640 1 monclient(hunting): continuing hunt 2024-04-26T04:24:57.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.123+0000 7f9cbeffd640 10 monclient(hunting): _reopen_session rank -1 2024-04-26T04:24:57.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.123+0000 7f9cbeffd640 1 -- 172.21.15.63:0/898483349 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9cac007410 msgr2=0x7f9cac0098d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T04:24:57.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.123+0000 7f9cbeffd640 1 --2- 172.21.15.63:0/898483349 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9cac007410 0x7f9cac0098d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:57.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.123+0000 7f9cbeffd640 1 -- 172.21.15.63:0/898483349 >> v1:172.21.15.63:6789/0 conn(0x7f9cac003740 legacy=0x7f9cac005ba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T04:24:57.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.123+0000 7f9ccddb5640 1 -- 172.21.15.63:0/898483349 reap_dead start 2024-04-26T04:24:57.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.123+0000 7f9cbeffd640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-04-26T04:24:57.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.123+0000 7f9cbeffd640 1 --2- 172.21.15.63:0/898483349 >> v2:172.21.15.63:3300/0 conn(0x7f9cac00a000 0x7f9cac0098d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:57.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.123+0000 7f9cbeffd640 10 monclient(hunting): picked mon.noname-a con 0x7f9cac00a000 addr v2:172.21.15.63:3300/0 2024-04-26T04:24:57.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.123+0000 7f9cbeffd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9cac00a480 addr v1:172.21.15.63:6789/0 2024-04-26T04:24:57.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.123+0000 7f9cbeffd640 1 --2- 172.21.15.63:0/898483349 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9cac00a920 0x7f9cac00d580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:57.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.123+0000 7f9cbeffd640 10 monclient(hunting): picked mon.noname-b con 0x7f9cac00a920 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:24:57.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.123+0000 7f9cbeffd640 1 -- 172.21.15.63:0/898483349 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9cac00b160 con 0x7f9cac00a480 2024-04-26T04:24:57.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.123+0000 7f9cbeffd640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:57.131 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.123+0000 7f9cbeffd640 1 -- 172.21.15.63:0/898483349 --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7f9cc80a0df0 con 0x7f9cac00a000 2024-04-26T04:24:57.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.123+0000 7f9cbeffd640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:57.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.123+0000 7f9cbeffd640 1 -- 172.21.15.63:0/898483349 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f9cc80a13b0 con 0x7f9cac00a920 2024-04-26T04:24:57.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.123+0000 7f9cbeffd640 10 monclient(hunting): _renew_subs 2024-04-26T04:24:57.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.123+0000 7f9ccd5b4640 1 -- 172.21.15.63:0/898483349 >> v2:172.21.15.63:3300/0 conn(0x7f9cac00a000 msgr2=0x7f9cac0098d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.63:3300/0 2024-04-26T04:24:57.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.123+0000 7f9cccdb3640 1 -- 172.21.15.63:0/898483349 >> v1:172.21.15.63:6789/0 conn(0x7f9cac00a480 legacy=0x7f9cac005ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.63:6789/0 2024-04-26T04:24:57.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.123+0000 7f9ccd5b4640 1 --2- 172.21.15.63:0/898483349 >> v2:172.21.15.63:3300/0 conn(0x7f9cac00a000 0x7f9cac0098d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:24:57.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.123+0000 7f9ccddb5640 1 --2- 172.21.15.63:0/898483349 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9cac00a920 0x7f9cac00d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:57.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.123+0000 7f9ccddb5640 10 monclient(hunting): get_auth_request con 0x7f9cac00a920 auth_method 0 2024-04-26T04:24:57.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.123+0000 7f9ccddb5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:57.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.123+0000 7f9ccddb5640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:57.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.123+0000 7f9ccddb5640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:57.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.127+0000 7f9ccddb5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:57.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.127+0000 7f9ccddb5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:57.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.127+0000 7f9ccddb5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:24:57.132 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.127+0000 7f9ccddb5640 10 monclient(hunting): handle_auth_done global_id 24305 payload 274 2024-04-26T04:24:57.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.127+0000 7f9ccddb5640 1 -- 172.21.15.63:0/898483349 >> v2:172.21.15.63:3300/0 conn(0x7f9cac00a000 msgr2=0x7f9cac0098d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T04:24:57.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.127+0000 7f9ccddb5640 1 --2- 172.21.15.63:0/898483349 >> v2:172.21.15.63:3300/0 conn(0x7f9cac00a000 0x7f9cac0098d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:57.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.127+0000 7f9ccddb5640 1 -- 172.21.15.63:0/898483349 >> v1:172.21.15.63:6789/0 conn(0x7f9cac00a480 legacy=0x7f9cac005ba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T04:24:57.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.127+0000 7f9ccddb5640 10 monclient: _finish_hunting 0 2024-04-26T04:24:57.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.127+0000 7f9ccddb5640 1 monclient: found mon.noname-b 2024-04-26T04:24:57.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.127+0000 7f9ccddb5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:24:57.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.127+0000 7f9ccddb5640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.63:3301/0 2024-04-26T04:24:57.133 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.127+0000 7f9ccddb5640 1 -- 172.21.15.63:0/898483349 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cc80a10d0 con 0x7f9cac00a920 2024-04-26T04:24:57.138 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.131+0000 7f9ccddb5640 1 -- 172.21.15.63:0/898483349 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9cac00a920 msgr2=0x7f9cac00d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:24:57.138 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.131+0000 7f9ccddb5640 1 --2- 172.21.15.63:0/898483349 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9cac00a920 0x7f9cac00d580 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9cc806d7c0 tx=0x7f9cc001c200 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40eaf62affe5fc9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:57.138 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.135+0000 7f9cbffff640 1 -- 172.21.15.63:0/898483349 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cc0007350 con 0x7f9cac00a920 2024-04-26T04:24:57.138 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.135+0000 7f9cbffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:57.138 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.135+0000 7f9cbffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T04:24:57.138 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.135+0000 7f9cbffff640 10 monclient: dump: 2024-04-26T04:24:57.138 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:57.138 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:57.138 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:57.139 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:57.139 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:57.139 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:57.139 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:57.139 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:57.139 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:57.139 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:57.139 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.135+0000 7f9cbffff640 10 monclient: _finish_auth 0 2024-04-26T04:24:57.139 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.135+0000 7f9cbffff640 10 monclient: _check_auth_tickets 2024-04-26T04:24:57.139 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.135+0000 7f9cbffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:24:57.139 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.135+0000 7f9cbffff640 1 -- 172.21.15.63:0/898483349 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9cc00074f0 con 0x7f9cac00a920 2024-04-26T04:24:57.139 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.135+0000 7f9cbffff640 10 monclient: handle_config config(0 keys) 2024-04-26T04:24:57.139 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.135+0000 7f9cbffff640 1 -- 172.21.15.63:0/898483349 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cc00049f0 con 0x7f9cac00a920 2024-04-26T04:24:57.139 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.135+0000 7f9ccf818640 5 monclient: authenticate success, global_id 24305 2024-04-26T04:24:57.139 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.135+0000 7f9ccf818640 10 monclient: get_monmap_and_config success 2024-04-26T04:24:57.140 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.135+0000 7f9ccf818640 4 set_mon_vals no callback set 2024-04-26T04:24:57.140 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.135+0000 7f9cbffff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:57.140 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.135+0000 7f9cbffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:24:57.140 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.135+0000 7f9cbffff640 10 monclient: dump: 2024-04-26T04:24:57.140 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:57.140 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:57.140 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:57.140 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:57.140 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:57.140 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:57.140 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:57.140 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:57.140 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:57.140 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:57.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.135+0000 7f9ccf818640 10 monclient: shutdown 2024-04-26T04:24:57.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.135+0000 7f9ccf818640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:24:57.142 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.135+0000 7f9ccf818640 1 -- 172.21.15.63:0/898483349 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9cac00a920 msgr2=0x7f9cac00d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:57.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.135+0000 7f9ccf818640 1 --2- 172.21.15.63:0/898483349 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9cac00a920 0x7f9cac00d580 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9cc806d7c0 tx=0x7f9cc001c200 comp rx=0 tx=0).stop 2024-04-26T04:24:57.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.135+0000 7f9ccf818640 1 -- 172.21.15.63:0/898483349 shutdown_connections 2024-04-26T04:24:57.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.135+0000 7f9ccf818640 1 --2- 172.21.15.63:0/898483349 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9cac00a920 0x7f9cac00d580 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:57.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.135+0000 7f9ccf818640 1 --2- 172.21.15.63:0/898483349 >> v2:172.21.15.63:3300/0 conn(0x7f9cac00a000 0x7f9cac0098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:57.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.135+0000 7f9ccf818640 1 -- 172.21.15.63:0/898483349 >> 172.21.15.63:0/898483349 conn(0x7f9cc80a5fb0 msgr2=0x7f9cc80a63c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:24:57.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.135+0000 7f9ccf818640 1 -- 172.21.15.63:0/898483349 shutdown_connections 2024-04-26T04:24:57.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.135+0000 7f9ccf818640 1 -- 172.21.15.63:0/898483349 wait complete. 2024-04-26T04:24:57.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.135+0000 7f9ccf818640 10 monclient: build_initial_monmap 2024-04-26T04:24:57.143 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.135+0000 7f9ccf818640 10 monclient: monmap: 2024-04-26T04:24:57.143 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:24:57.143 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:24:57.143 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:24:57.144 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:24:57.144 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:24:57.144 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:57.144 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:24:57.144 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:24:57.144 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:24:57.144 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:57.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.139+0000 7f9ccf818640 1 Processor -- start 2024-04-26T04:24:57.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.139+0000 7f9ccf818640 1 -- start start 2024-04-26T04:24:57.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.139+0000 7f9ccf818640 10 monclient: init 2024-04-26T04:24:57.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.139+0000 7f9ccf818640 10 monclient: _reopen_session rank -1 2024-04-26T04:24:57.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.139+0000 7f9ccf818640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T04:24:57.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.139+0000 7f9ccf818640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9cac00a000 0x7f9cc81cf500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:57.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.139+0000 7f9ccf818640 10 monclient(hunting): picked mon.noname-b con 0x7f9cac00a000 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:24:57.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.139+0000 7f9ccf818640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9cac00a480 0x7f9cc81cfa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:57.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.139+0000 7f9ccf818640 10 monclient(hunting): picked mon.noname-a con 0x7f9cac00a480 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:24:57.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.139+0000 7f9ccf818640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9cac00a920 0x7f9cc81cff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:57.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.139+0000 7f9ccf818640 10 monclient(hunting): picked mon.noname-c con 0x7f9cac00a920 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:24:57.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.139+0000 7f9ccf818640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:57.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.139+0000 7f9ccf818640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f9cc80a4950 con 0x7f9cac00a480 2024-04-26T04:24:57.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.139+0000 7f9ccf818640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:57.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.139+0000 7f9ccf818640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9cc80a47d0 con 0x7f9cac00a000 2024-04-26T04:24:57.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.139+0000 7f9ccf818640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:57.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.139+0000 7f9ccf818640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f9cc80a4ad0 con 0x7f9cac00a920 2024-04-26T04:24:57.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.139+0000 7f9ccf818640 10 monclient(hunting): _renew_subs 2024-04-26T04:24:57.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.139+0000 7f9ccf818640 10 monclient(hunting): authenticate will time out at 1328.850017s 2024-04-26T04:24:57.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.139+0000 7f9cccdb3640 1 -- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9cac00a480 msgr2=0x7f9cc81cfa40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:3300/0 2024-04-26T04:24:57.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.139+0000 7f9cccdb3640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9cac00a480 0x7f9cc81cfa40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:24:57.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.139+0000 7f9ccddb5640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9cac00a920 0x7f9cc81cff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:57.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.139+0000 7f9ccddb5640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9cac00a920 0x7f9cc81cff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:34054/0 (socket says 172.21.15.63:34054) 2024-04-26T04:24:57.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.139+0000 7f9ccddb5640 1 -- 172.21.15.63:0/3452229412 learned_addr learned my addr 172.21.15.63:0/3452229412 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:24:57.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.139+0000 7f9ccddb5640 10 monclient(hunting): get_auth_request con 0x7f9cac00a920 auth_method 0 2024-04-26T04:24:57.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.139+0000 7f9ccddb5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:57.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.139+0000 7f9ccddb5640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:57.150 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.139+0000 7f9ccddb5640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:57.150 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.139+0000 7f9ccddb5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:57.150 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.139+0000 7f9ccddb5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:57.150 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.139+0000 7f9ccddb5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:24:57.150 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.139+0000 7f9ccd5b4640 1 --2- 172.21.15.63:0/3452229412 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9cac00a000 0x7f9cc81cf500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:57.150 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.139+0000 7f9ccd5b4640 10 monclient(hunting): get_auth_request con 0x7f9cac00a000 auth_method 0 2024-04-26T04:24:57.150 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.139+0000 7f9ccd5b4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:57.150 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.139+0000 7f9ccd5b4640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:57.150 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.139+0000 7f9ccd5b4640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:57.150 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.139+0000 7f9ccddb5640 10 monclient(hunting): handle_auth_done global_id 24308 payload 824 2024-04-26T04:24:57.150 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.139+0000 7f9ccddb5640 1 -- 172.21.15.63:0/3452229412 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9cac00a000 msgr2=0x7f9cc81cf500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:57.150 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.139+0000 7f9ccddb5640 1 --2- 172.21.15.63:0/3452229412 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9cac00a000 0x7f9cc81cf500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:57.150 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.139+0000 7f9ccddb5640 1 -- 172.21.15.63:0/3452229412 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9cac00a480 msgr2=0x7f9cc81cfa40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T04:24:57.150 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.139+0000 7f9ccddb5640 1 --2- 172.21.15.63:0/3452229412 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9cac00a480 0x7f9cc81cfa40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:57.151 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.139+0000 7f9ccddb5640 10 monclient: _finish_hunting 0 2024-04-26T04:24:57.151 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.139+0000 7f9ccddb5640 1 monclient: found mon.noname-c 2024-04-26T04:24:57.151 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.139+0000 7f9ccddb5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:24:57.151 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.139+0000 7f9ccddb5640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.63:3301/0 2024-04-26T04:24:57.151 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.139+0000 7f9ccddb5640 1 -- 172.21.15.63:0/3452229412 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cc811fba0 con 0x7f9cac00a920 2024-04-26T04:24:57.151 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.143+0000 7f9ccddb5640 1 -- 172.21.15.63:0/3452229412 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9cac00a920 msgr2=0x7f9cc81cff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:24:57.151 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.143+0000 7f9ccddb5640 1 --2- 172.21.15.63:0/3452229412 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9cac00a920 0x7f9cc81cff80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f9cc001c6e0 tx=0x7f9cc0018ff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:57.151 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.143+0000 7f9cbdffb640 1 -- 172.21.15.63:0/3452229412 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cc0007350 con 0x7f9cac00a920 2024-04-26T04:24:57.151 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.143+0000 7f9cbdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:57.151 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.143+0000 7f9cbdffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:24:57.151 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.143+0000 7f9cbdffb640 10 monclient: dump: 2024-04-26T04:24:57.151 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:57.151 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:57.151 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:57.151 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:57.152 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:57.152 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:57.152 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:57.152 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:57.152 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:57.152 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:57.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.143+0000 7f9cbdffb640 10 monclient: _finish_auth 0 2024-04-26T04:24:57.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.143+0000 7f9cbdffb640 10 monclient: _check_auth_tickets 2024-04-26T04:24:57.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.143+0000 7f9cbdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:24:57.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.143+0000 7f9cbdffb640 1 -- 172.21.15.63:0/3452229412 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9cc00ac420 con 0x7f9cac00a920 2024-04-26T04:24:57.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.143+0000 7f9cbdffb640 10 monclient: handle_config config(0 keys) 2024-04-26T04:24:57.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.143+0000 7f9cbdffb640 1 -- 172.21.15.63:0/3452229412 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cc00ac7d0 con 0x7f9cac00a920 2024-04-26T04:24:57.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.143+0000 7f9ccf818640 5 monclient: authenticate success, global_id 24308 2024-04-26T04:24:57.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.143+0000 7f9ccf818640 10 monclient: _renew_subs 2024-04-26T04:24:57.153 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.143+0000 7f9ccf818640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:24:57.153 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.143+0000 7f9ccf818640 1 -- 172.21.15.63:0/3452229412 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cc8120660 con 0x7f9cac00a920 2024-04-26T04:24:57.153 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.143+0000 7f9cbeffd640 4 set_mon_vals no callback set 2024-04-26T04:24:57.153 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.143+0000 7f9cbdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:57.153 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.143+0000 7f9cbdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:24:57.153 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.143+0000 7f9cbdffb640 10 monclient: dump: 2024-04-26T04:24:57.153 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:57.153 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:57.153 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:57.153 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:57.153 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:57.153 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:57.153 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:57.153 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:57.153 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:57.153 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:57.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.143+0000 7f9ccf818640 10 monclient: _renew_subs 2024-04-26T04:24:57.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.143+0000 7f9ccf818640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:24:57.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.143+0000 7f9ccf818640 1 -- 172.21.15.63:0/3452229412 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cc8120280 con 0x7f9cac00a920 2024-04-26T04:24:57.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.143+0000 7f9cbdffb640 1 -- 172.21.15.63:0/3452229412 <== mon.2 v2:172.21.15.63:3301/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f9cc00acbc0 con 0x7f9cac00a920 2024-04-26T04:24:57.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.143+0000 7f9ccf818640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:24:57.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.143+0000 7f9ccf818640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:24:57.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.143+0000 7f9ccf818640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:24:57.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.143+0000 7f9ccf818640 1 -- 172.21.15.63:0/3452229412 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9cc8062e10 con 0x7f9cac00a920 2024-04-26T04:24:57.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.143+0000 7f9cbdffb640 1 --2- 172.21.15.63:0/3452229412 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f9cac037a40 0x7f9cac039f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:57.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.143+0000 7f9cbdffb640 1 -- 172.21.15.63:0/3452229412 <== mon.2 v2:172.21.15.63:3301/0 5 ==== osd_map(223..223 src has 1..223) ==== 4346+0+0 (secure 0 0 0) 0x7f9cc00e2c40 con 0x7f9cac00a920 2024-04-26T04:24:57.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.143+0000 7f9ccd5b4640 1 --2- 172.21.15.63:0/3452229412 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f9cac037a40 0x7f9cac039f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:57.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.143+0000 7f9ccd5b4640 10 monclient: get_auth_request con 0x7f9cac037a40 auth_method 0 2024-04-26T04:24:57.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.147+0000 7f9ccd5b4640 1 --2- 172.21.15.63:0/3452229412 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f9cac037a40 0x7f9cac039f00 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f9cc40096d0 tx=0x7f9cc4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:57.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.147+0000 7f9cbdffb640 1 -- 172.21.15.63:0/3452229412 <== mon.2 v2:172.21.15.63:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f9cc00be060 con 0x7f9cac00a920 2024-04-26T04:24:57.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.147+0000 7f9cbdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:24:57.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.147+0000 7f9cbdffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:24:57.377 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.371+0000 7f9ccf818640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T04:24:57.377 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.371+0000 7f9ccf818640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T04:24:57.378 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.371+0000 7f9ccf818640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:24:57.378 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.371+0000 7f9ccf818640 1 -- 172.21.15.63:0/3452229412 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f9cc8072400 con 0x7f9cac00a920 2024-04-26T04:24:57.378 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.375+0000 7f9cbdffb640 1 -- 172.21.15.63:0/3452229412 <== mon.2 v2:172.21.15.63:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1554 (secure 0 0 0) 0x7f9cc00b3070 con 0x7f9cac00a920 2024-04-26T04:24:57.378 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.375+0000 7f9cbdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T04:24:57.378 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.375+0000 7f9cbdffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:24:57.378 INFO:teuthology.orchestra.run.smithi063.stdout:{"election_epoch":28,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cd1c6a0c-4e4b-4014-89c0-2e535479ef39","modified":"2024-04-26T04:18:03.702339Z","created":"2024-04-26T04:18:03.702339Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3300","nonce":0},{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:3300","nonce":0},{"type":"v1","addr":"172.21.15.186:6789","nonce":0}]},"addr":"172.21.15.186:6789/0","public_addr":"172.21.15.186:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3301","nonce":0},{"type":"v1","addr":"172.21.15.63:6790","nonce":0}]},"addr":"172.21.15.63:6790/0","public_addr":"172.21.15.63:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T04:24:57.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.375+0000 7f9ccf818640 1 -- 172.21.15.63:0/3452229412 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f9cac037a40 msgr2=0x7f9cac039f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:57.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.375+0000 7f9ccf818640 1 --2- 172.21.15.63:0/3452229412 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f9cac037a40 0x7f9cac039f00 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f9cc40096d0 tx=0x7f9cc4009040 comp rx=0 tx=0).stop 2024-04-26T04:24:57.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.375+0000 7f9ccf818640 10 monclient: shutdown 2024-04-26T04:24:57.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.375+0000 7f9ccf818640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:24:57.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.375+0000 7f9ccf818640 1 -- 172.21.15.63:0/3452229412 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9cac00a920 msgr2=0x7f9cc81cff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:57.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.375+0000 7f9ccf818640 1 --2- 172.21.15.63:0/3452229412 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9cac00a920 0x7f9cc81cff80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f9cc001c6e0 tx=0x7f9cc0018ff0 comp rx=0 tx=0).stop 2024-04-26T04:24:57.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.375+0000 7f9ccf818640 1 -- 172.21.15.63:0/3452229412 shutdown_connections 2024-04-26T04:24:57.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.375+0000 7f9ccf818640 1 --2- 172.21.15.63:0/3452229412 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f9cac037a40 0x7f9cac039f00 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:57.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.375+0000 7f9ccf818640 1 --2- 172.21.15.63:0/3452229412 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9cac00a920 0x7f9cc81cff80 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:57.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.375+0000 7f9ccf818640 1 --2- 172.21.15.63:0/3452229412 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9cac00a480 0x7f9cc81cfa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:57.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.375+0000 7f9ccf818640 1 --2- 172.21.15.63:0/3452229412 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9cac00a000 0x7f9cc81cf500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:57.381 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.375+0000 7f9ccf818640 1 -- 172.21.15.63:0/3452229412 >> 172.21.15.63:0/3452229412 conn(0x7f9cc80a5fb0 msgr2=0x7f9cc8134430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:24:57.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.375+0000 7f9ccf818640 1 -- 172.21.15.63:0/3452229412 shutdown_connections 2024-04-26T04:24:57.382 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.375+0000 7f9ccf818640 1 -- 172.21.15.63:0/3452229412 wait complete. 2024-04-26T04:24:57.398 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-04-26T04:24:57.398 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-04-26T04:24:57.399 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T04:24:57.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.511+0000 7f416fa7d640 10 monclient: get_monmap_and_config 2024-04-26T04:24:57.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.511+0000 7f416fa7d640 10 monclient: build_initial_monmap 2024-04-26T04:24:57.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.511+0000 7f416fa7d640 10 monclient: monmap: 2024-04-26T04:24:57.515 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:24:57.515 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:57.515 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:24:57.515272+0000 2024-04-26T04:24:57.515 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:24:57.515272+0000 2024-04-26T04:24:57.515 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:24:57.515 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:57.515 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:24:57.516 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:24:57.516 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:24:57.516 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:24:57.516 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:24:57.516 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:57.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.511+0000 7f416fa7d640 1 Processor -- start 2024-04-26T04:24:57.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.511+0000 7f416fa7d640 1 -- start start 2024-04-26T04:24:57.517 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.511+0000 7f416fa7d640 10 monclient: init 2024-04-26T04:24:57.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.511+0000 7f416fa7d640 10 monclient: _reopen_session rank -1 2024-04-26T04:24:57.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.511+0000 7f416fa7d640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-04-26T04:24:57.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.511+0000 7f416fa7d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4168071db0 addr v1:172.21.15.186:6789/0 2024-04-26T04:24:57.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.511+0000 7f416fa7d640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f4168071250 0x7f4168138500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:57.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.511+0000 7f416fa7d640 10 monclient(hunting): picked mon.noname-c con 0x7f4168071250 addr v2:172.21.15.186:3300/0 2024-04-26T04:24:57.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.511+0000 7f416fa7d640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f4168071660 0x7f416813aa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:57.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.511+0000 7f416fa7d640 10 monclient(hunting): picked mon.noname-a con 0x7f4168071660 addr v2:172.21.15.63:3300/0 2024-04-26T04:24:57.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.511+0000 7f416fa7d640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f416806e8b0 con 0x7f4168071db0 2024-04-26T04:24:57.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.511+0000 7f416fa7d640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:57.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.511+0000 7f416fa7d640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7f41680728a0 con 0x7f4168071660 2024-04-26T04:24:57.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.511+0000 7f416fa7d640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:57.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.511+0000 7f416fa7d640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f41680702b0 con 0x7f4168071250 2024-04-26T04:24:57.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.511+0000 7f416fa7d640 10 monclient(hunting): _renew_subs 2024-04-26T04:24:57.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.511+0000 7f416fa7d640 10 monclient(hunting): authenticate will time out at 1329.223829s 2024-04-26T04:24:57.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.511+0000 7f416e01a640 1 -- >> v2:172.21.15.63:3300/0 conn(0x7f4168071660 msgr2=0x7f416813aa50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.63:3300/0 2024-04-26T04:24:57.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.511+0000 7f416e01a640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f4168071660 0x7f416813aa50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:24:57.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.515+0000 7f416d018640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f4168071250 0x7f4168138500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:57.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.515+0000 7f416d018640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f4168071250 0x7f4168138500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.63:39488/0 (socket says 172.21.15.63:39488) 2024-04-26T04:24:57.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.515+0000 7f416d018640 1 -- 172.21.15.63:0/552438554 learned_addr learned my addr 172.21.15.63:0/552438554 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:24:57.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.515+0000 7f416d018640 10 monclient(hunting): get_auth_request con 0x7f4168071250 auth_method 0 2024-04-26T04:24:57.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.515+0000 7f416d018640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:57.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.515+0000 7f416d018640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:57.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.515+0000 7f416d018640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:57.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.515+0000 7f416d819640 1 --1- 172.21.15.63:0/552438554 >> v1:172.21.15.186:6789/0 conn(0x7f4168071db0 0x7f4168072190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.63:38084/0 (socket says 172.21.15.63:38084) 2024-04-26T04:24:57.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.515+0000 7f416d018640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:57.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.515+0000 7f416d018640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:57.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.515+0000 7f416d018640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:24:57.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.515+0000 7f416d018640 10 monclient(hunting): handle_auth_done global_id 34180 payload 274 2024-04-26T04:24:57.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.515+0000 7f416d018640 1 -- 172.21.15.63:0/552438554 >> v2:172.21.15.63:3300/0 conn(0x7f4168071660 msgr2=0x7f416813aa50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T04:24:57.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.515+0000 7f416d018640 1 --2- 172.21.15.63:0/552438554 >> v2:172.21.15.63:3300/0 conn(0x7f4168071660 0x7f416813aa50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:57.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.515+0000 7f416d018640 1 -- 172.21.15.63:0/552438554 >> v1:172.21.15.186:6789/0 conn(0x7f4168071db0 legacy=0x7f4168072190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:24:57.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.515+0000 7f416d018640 10 monclient: _finish_hunting 0 2024-04-26T04:24:57.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.515+0000 7f416d018640 1 monclient: found mon.noname-c 2024-04-26T04:24:57.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.515+0000 7f416d018640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:24:57.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.515+0000 7f416d018640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.186:3300/0 2024-04-26T04:24:57.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.515+0000 7f416d018640 1 -- 172.21.15.63:0/552438554 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f416806ed20 con 0x7f4168071250 2024-04-26T04:24:57.523 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.519+0000 7f416d018640 1 -- 172.21.15.63:0/552438554 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4168071250 msgr2=0x7f4168138500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:24:57.523 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.519+0000 7f416d018640 1 --2- 172.21.15.63:0/552438554 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4168071250 0x7f4168138500 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f41680a2540 tx=0x7f415801c3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=483cf3fb12794840 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:57.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.523+0000 7f416c817640 1 -- 172.21.15.63:0/552438554 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41580b0070 con 0x7f4168071250 2024-04-26T04:24:57.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.523+0000 7f416c817640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:57.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.523+0000 7f416c817640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:24:57.526 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.523+0000 7f416c817640 10 monclient: dump: 2024-04-26T04:24:57.527 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:57.527 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:57.527 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:57.527 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:57.527 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:57.527 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:57.527 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:57.527 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:57.527 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:57.527 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:57.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.523+0000 7f416c817640 10 monclient: _finish_auth 0 2024-04-26T04:24:57.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.523+0000 7f416c817640 10 monclient: _check_auth_tickets 2024-04-26T04:24:57.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.523+0000 7f416c817640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:24:57.527 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.523+0000 7f416fa7d640 5 monclient: authenticate success, global_id 34180 2024-04-26T04:24:57.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.523+0000 7f416fa7d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T04:24:57.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.523+0000 7f416c817640 1 -- 172.21.15.63:0/552438554 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41580ab600 con 0x7f4168071250 2024-04-26T04:24:57.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.523+0000 7f416c817640 10 monclient: handle_config config(0 keys) 2024-04-26T04:24:57.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.523+0000 7f416c817640 1 -- 172.21.15.63:0/552438554 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41580ab9d0 con 0x7f4168071250 2024-04-26T04:24:57.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.523+0000 7f416c817640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:57.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.523+0000 7f416c817640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T04:24:57.528 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.523+0000 7f416c817640 10 monclient: dump: 2024-04-26T04:24:57.528 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:57.528 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:57.528 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:57.528 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:57.528 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:57.528 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:57.528 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:57.528 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:57.529 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:57.529 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:57.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.523+0000 7f416fa7d640 10 monclient: get_monmap_and_config success 2024-04-26T04:24:57.529 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.523+0000 7f416fa7d640 4 set_mon_vals no callback set 2024-04-26T04:24:57.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.523+0000 7f416fa7d640 10 monclient: shutdown 2024-04-26T04:24:57.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.523+0000 7f416fa7d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:24:57.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.523+0000 7f416fa7d640 1 -- 172.21.15.63:0/552438554 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4168071250 msgr2=0x7f4168138500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:57.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.523+0000 7f416fa7d640 1 --2- 172.21.15.63:0/552438554 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4168071250 0x7f4168138500 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f41680a2540 tx=0x7f415801c3c0 comp rx=0 tx=0).stop 2024-04-26T04:24:57.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.523+0000 7f416fa7d640 1 -- 172.21.15.63:0/552438554 shutdown_connections 2024-04-26T04:24:57.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.523+0000 7f416fa7d640 1 --2- 172.21.15.63:0/552438554 >> v2:172.21.15.63:3300/0 conn(0x7f4168071660 0x7f416813aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:57.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.523+0000 7f416fa7d640 1 --2- 172.21.15.63:0/552438554 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4168071250 0x7f4168138500 secure :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f41680a2540 tx=0x7f415801c3c0 comp rx=0 tx=0).stop 2024-04-26T04:24:57.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.523+0000 7f416fa7d640 1 -- 172.21.15.63:0/552438554 >> 172.21.15.63:0/552438554 conn(0x7f41680a6170 msgr2=0x7f41680a2590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:24:57.530 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.527+0000 7f416fa7d640 1 -- 172.21.15.63:0/552438554 shutdown_connections 2024-04-26T04:24:57.534 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.527+0000 7f416fa7d640 1 -- 172.21.15.63:0/552438554 wait complete. 2024-04-26T04:24:57.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.527+0000 7f416fa7d640 10 monclient: build_initial_monmap 2024-04-26T04:24:57.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.527+0000 7f416fa7d640 10 monclient: monmap: 2024-04-26T04:24:57.539 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:24:57.539 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:24:57.539 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:24:57.539 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:24:57.540 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:24:57.540 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:57.540 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:24:57.540 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:24:57.540 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:24:57.540 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:57.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.527+0000 7f416fa7d640 1 Processor -- start 2024-04-26T04:24:57.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.527+0000 7f416fa7d640 1 -- start start 2024-04-26T04:24:57.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.527+0000 7f416fa7d640 10 monclient: init 2024-04-26T04:24:57.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.527+0000 7f416fa7d640 10 monclient: _reopen_session rank -1 2024-04-26T04:24:57.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.527+0000 7f416fa7d640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T04:24:57.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.527+0000 7f416fa7d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4168071660 0x7f41681ca8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:57.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.527+0000 7f416fa7d640 10 monclient(hunting): picked mon.noname-b con 0x7f4168071660 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:24:57.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.527+0000 7f416fa7d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4168071db0 0x7f4168139560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:57.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.527+0000 7f416fa7d640 10 monclient(hunting): picked mon.noname-a con 0x7f4168071db0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:24:57.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.527+0000 7f416fa7d640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4168139aa0 0x7f416811ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:57.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.527+0000 7f416fa7d640 10 monclient(hunting): picked mon.noname-c con 0x7f4168139aa0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:24:57.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.527+0000 7f416fa7d640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:57.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.527+0000 7f416fa7d640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f416806c230 con 0x7f4168071db0 2024-04-26T04:24:57.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.527+0000 7f416fa7d640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:57.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.527+0000 7f416fa7d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f416806c0b0 con 0x7f4168071660 2024-04-26T04:24:57.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.527+0000 7f416fa7d640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:57.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.527+0000 7f416fa7d640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f416806c3b0 con 0x7f4168139aa0 2024-04-26T04:24:57.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.527+0000 7f416fa7d640 10 monclient(hunting): _renew_subs 2024-04-26T04:24:57.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.527+0000 7f416fa7d640 10 monclient(hunting): authenticate will time out at 1329.238518s 2024-04-26T04:24:57.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.527+0000 7f416d018640 1 -- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4168071db0 msgr2=0x7f4168139560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:3300/0 2024-04-26T04:24:57.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.527+0000 7f416d018640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4168071db0 0x7f4168139560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:24:57.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.527+0000 7f416e01a640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4168139aa0 0x7f416811ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:57.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.527+0000 7f416e01a640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4168139aa0 0x7f416811ffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:34058/0 (socket says 172.21.15.63:34058) 2024-04-26T04:24:57.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.527+0000 7f416e01a640 1 -- 172.21.15.63:0/2216364273 learned_addr learned my addr 172.21.15.63:0/2216364273 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:24:57.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.527+0000 7f416e01a640 10 monclient(hunting): get_auth_request con 0x7f4168139aa0 auth_method 0 2024-04-26T04:24:57.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.527+0000 7f416e01a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:57.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.527+0000 7f416e01a640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:57.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.527+0000 7f416e01a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:57.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.527+0000 7f416e01a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:57.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.527+0000 7f416e01a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:57.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.527+0000 7f416e01a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:24:57.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.527+0000 7f416e01a640 10 monclient(hunting): handle_auth_done global_id 24311 payload 824 2024-04-26T04:24:57.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.527+0000 7f416e01a640 1 -- 172.21.15.63:0/2216364273 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4168071660 msgr2=0x7f41681ca8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:57.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.527+0000 7f416e01a640 1 --2- 172.21.15.63:0/2216364273 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4168071660 0x7f41681ca8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:57.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.527+0000 7f416e01a640 1 -- 172.21.15.63:0/2216364273 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4168071db0 msgr2=0x7f4168139560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T04:24:57.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.527+0000 7f416e01a640 1 --2- 172.21.15.63:0/2216364273 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4168071db0 0x7f4168139560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:57.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.527+0000 7f416e01a640 10 monclient: _finish_hunting 0 2024-04-26T04:24:57.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.527+0000 7f416e01a640 1 monclient: found mon.noname-c 2024-04-26T04:24:57.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.527+0000 7f416e01a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:24:57.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.527+0000 7f416e01a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.63:3301/0 2024-04-26T04:24:57.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.527+0000 7f416e01a640 1 -- 172.21.15.63:0/2216364273 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4168121c00 con 0x7f4168139aa0 2024-04-26T04:24:57.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.531+0000 7f416e01a640 1 -- 172.21.15.63:0/2216364273 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4168139aa0 msgr2=0x7f416811ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:24:57.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.531+0000 7f416e01a640 1 --2- 172.21.15.63:0/2216364273 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4168139aa0 0x7f416811ffb0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f416000ba40 tx=0x7f416000bf00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:57.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.531+0000 7f416e01a640 1 --2- 172.21.15.63:0/2216364273 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4168139aa0 0x7f416811ffb0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f416000ba40 tx=0x7f416000bf00 comp rx=0 tx=0).queue_received will delay after 1.71411e+09 on 0x7f41600156b0 mon_map magic: 0 2024-04-26T04:24:57.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.531+0000 7f415e7fc640 1 -- 172.21.15.63:0/2216364273 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f416000cdf0 con 0x7f4168139aa0 2024-04-26T04:24:57.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.531+0000 7f415e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:57.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.531+0000 7f415e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:24:57.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.531+0000 7f415e7fc640 10 monclient: dump: 2024-04-26T04:24:57.543 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:57.543 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:57.543 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:57.543 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:57.543 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:57.544 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:57.544 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:57.544 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:57.544 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:57.544 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:57.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.531+0000 7f415e7fc640 10 monclient: _finish_auth 0 2024-04-26T04:24:57.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.531+0000 7f415e7fc640 10 monclient: _check_auth_tickets 2024-04-26T04:24:57.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.531+0000 7f415e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:24:57.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.531+0000 7f415e7fc640 1 -- 172.21.15.63:0/2216364273 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4160010070 con 0x7f4168139aa0 2024-04-26T04:24:57.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.531+0000 7f415e7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T04:24:57.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.531+0000 7f415ffff640 4 set_mon_vals no callback set 2024-04-26T04:24:57.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.531+0000 7f416fa7d640 5 monclient: authenticate success, global_id 24311 2024-04-26T04:24:57.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.531+0000 7f416fa7d640 10 monclient: _renew_subs 2024-04-26T04:24:57.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.531+0000 7f416fa7d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:24:57.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.531+0000 7f416fa7d640 1 -- 172.21.15.63:0/2216364273 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4168072a40 con 0x7f4168139aa0 2024-04-26T04:24:57.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.531+0000 7f416fa7d640 10 monclient: _renew_subs 2024-04-26T04:24:57.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.531+0000 7f416fa7d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:24:57.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.531+0000 7f416fa7d640 1 -- 172.21.15.63:0/2216364273 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4168123ef0 con 0x7f4168139aa0 2024-04-26T04:24:57.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.531+0000 7f416fa7d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:24:57.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.531+0000 7f416fa7d640 1 --2- 172.21.15.63:0/2216364273 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4168122980 0x7f41681204f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:57.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.531+0000 7f416fa7d640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:57.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.531+0000 7f416fa7d640 1 -- 172.21.15.63:0/2216364273 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f4168062e10 con 0x7f4168122980 2024-04-26T04:24:57.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.531+0000 7f415e7fc640 1 -- 172.21.15.63:0/2216364273 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41600156b0 con 0x7f4168139aa0 2024-04-26T04:24:57.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.531+0000 7f415e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:57.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.531+0000 7f415e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:24:57.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.531+0000 7f415e7fc640 10 monclient: dump: 2024-04-26T04:24:57.546 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:57.546 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:57.546 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:57.546 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:57.546 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:57.546 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:57.546 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:57.546 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:57.546 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:57.546 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:57.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.531+0000 7f415e7fc640 1 -- 172.21.15.63:0/2216364273 <== mon.2 v2:172.21.15.63:3301/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f4160015850 con 0x7f4168139aa0 2024-04-26T04:24:57.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.531+0000 7f416d819640 1 --2- 172.21.15.63:0/2216364273 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4168122980 0x7f41681204f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:57.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.531+0000 7f416d819640 10 monclient: get_auth_request con 0x7f4168122980 auth_method 0 2024-04-26T04:24:57.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.531+0000 7f416d819640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:57.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.531+0000 7f416d819640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:57.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.531+0000 7f416d819640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:57.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.531+0000 7f416d819640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:57.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.531+0000 7f416d819640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:57.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.531+0000 7f416d819640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:24:57.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.531+0000 7f415e7fc640 1 --2- 172.21.15.63:0/2216364273 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f414c035e70 0x7f414c038330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:57.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.531+0000 7f416d819640 10 monclient(hunting): handle_auth_done global_id 24314 payload 824 2024-04-26T04:24:57.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.531+0000 7f416d819640 1 -- 172.21.15.63:0/2216364273 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4168123a10 con 0x7f4168122980 2024-04-26T04:24:57.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.531+0000 7f416d018640 1 --2- 172.21.15.63:0/2216364273 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f414c035e70 0x7f414c038330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:57.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.531+0000 7f416d018640 10 monclient: get_auth_request con 0x7f414c035e70 auth_method 0 2024-04-26T04:24:57.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.535+0000 7f416d819640 1 -- 172.21.15.63:0/2216364273 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4168122980 msgr2=0x7f41681204f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:24:57.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.535+0000 7f416d819640 1 --2- 172.21.15.63:0/2216364273 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4168122980 0x7f41681204f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f416400c340 tx=0x7f416400c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:57.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.535+0000 7f415e7fc640 1 -- 172.21.15.63:0/2216364273 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4164017070 con 0x7f4168122980 2024-04-26T04:24:57.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.535+0000 7f415e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:57.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.535+0000 7f415e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:24:57.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.535+0000 7f415e7fc640 10 monclient: dump: 2024-04-26T04:24:57.548 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:57.548 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:57.548 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:57.548 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:57.548 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:57.548 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:57.548 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:57.548 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:57.548 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:57.548 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:57.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.535+0000 7f415e7fc640 1 -- 172.21.15.63:0/2216364273 <== mon.2 v2:172.21.15.63:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4164005800 con 0x7f4168122980 2024-04-26T04:24:57.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.535+0000 7f415e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:24:57.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.535+0000 7f415e7fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:24:57.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.535+0000 7f415e7fc640 1 -- 172.21.15.63:0/2216364273 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4168122980 msgr2=0x7f41681204f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:57.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.535+0000 7f415e7fc640 1 --2- 172.21.15.63:0/2216364273 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4168122980 0x7f41681204f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f416400c340 tx=0x7f416400c800 comp rx=0 tx=0).stop 2024-04-26T04:24:57.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.535+0000 7f415e7fc640 1 -- 172.21.15.63:0/2216364273 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4168122980 msgr2=0x7f41681204f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:24:57.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.535+0000 7f415e7fc640 1 --2- 172.21.15.63:0/2216364273 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4168122980 0x7f41681204f0 secure :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f416400c340 tx=0x7f416400c800 comp rx=0 tx=0).stop 2024-04-26T04:24:57.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.535+0000 7f416d018640 1 --2- 172.21.15.63:0/2216364273 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f414c035e70 0x7f414c038330 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f416806ecf0 tx=0x7f4158008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:57.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.539+0000 7f416fa7d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T04:24:57.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.543+0000 7f416fa7d640 1 --2- 172.21.15.63:0/2216364273 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4168120a30 0x7f416811c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:57.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.543+0000 7f416fa7d640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:57.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.543+0000 7f416fa7d640 1 -- 172.21.15.63:0/2216364273 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f4168120e40 con 0x7f4168120a30 2024-04-26T04:24:57.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.543+0000 7f416d819640 1 --2- 172.21.15.63:0/2216364273 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4168120a30 0x7f416811c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:57.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.543+0000 7f416d819640 10 monclient: get_auth_request con 0x7f4168120a30 auth_method 0 2024-04-26T04:24:57.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.543+0000 7f416d819640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:57.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.543+0000 7f416d819640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:57.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.543+0000 7f416d819640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:57.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.543+0000 7f416d819640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:57.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.543+0000 7f416d819640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:57.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.543+0000 7f416d819640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:24:57.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.543+0000 7f416d819640 10 monclient(hunting): handle_auth_done global_id 24317 payload 824 2024-04-26T04:24:57.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.543+0000 7f416d819640 1 -- 172.21.15.63:0/2216364273 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4168068260 con 0x7f4168120a30 2024-04-26T04:24:57.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.543+0000 7f416d819640 1 -- 172.21.15.63:0/2216364273 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4168120a30 msgr2=0x7f416811c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:24:57.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.543+0000 7f416d819640 1 --2- 172.21.15.63:0/2216364273 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4168120a30 0x7f416811c0e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f416400c340 tx=0x7f4164004940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:57.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.543+0000 7f415e7fc640 1 -- 172.21.15.63:0/2216364273 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4164017030 con 0x7f4168120a30 2024-04-26T04:24:57.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.543+0000 7f415e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:57.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.543+0000 7f415e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:24:57.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.543+0000 7f415e7fc640 10 monclient: dump: 2024-04-26T04:24:57.552 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:57.552 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:57.552 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:57.552 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:57.553 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:57.553 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:57.553 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:57.553 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:57.553 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:57.553 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:57.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.543+0000 7f415e7fc640 1 -- 172.21.15.63:0/2216364273 <== mon.2 v2:172.21.15.63:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4710 (secure 0 0 0) 0x7f41640192d0 con 0x7f4168120a30 2024-04-26T04:24:57.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.543+0000 7f415e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T04:24:57.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.543+0000 7f415e7fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:24:57.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.543+0000 7f415e7fc640 1 -- 172.21.15.63:0/2216364273 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4168120a30 msgr2=0x7f416811c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:57.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.543+0000 7f415e7fc640 1 --2- 172.21.15.63:0/2216364273 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4168120a30 0x7f416811c0e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f416400c340 tx=0x7f4164004940 comp rx=0 tx=0).stop 2024-04-26T04:24:57.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.543+0000 7f415e7fc640 1 -- 172.21.15.63:0/2216364273 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4168120a30 msgr2=0x7f416811c0e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:24:57.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.543+0000 7f415e7fc640 1 --2- 172.21.15.63:0/2216364273 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4168120a30 0x7f416811c0e0 secure :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f416400c340 tx=0x7f4164004940 comp rx=0 tx=0).stop 2024-04-26T04:24:57.554 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-26T04:24:57.554 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "c", 2024-04-26T04:24:57.554 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 2, 2024-04-26T04:24:57.554 INFO:teuthology.orchestra.run.smithi063.stdout: "state": "peon", 2024-04-26T04:24:57.554 INFO:teuthology.orchestra.run.smithi063.stdout: "election_epoch": 28, 2024-04-26T04:24:57.554 INFO:teuthology.orchestra.run.smithi063.stdout: "uptime": 145142, 2024-04-26T04:24:57.554 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum": [ 2024-04-26T04:24:57.554 INFO:teuthology.orchestra.run.smithi063.stdout: 1, 2024-04-26T04:24:57.554 INFO:teuthology.orchestra.run.smithi063.stdout: 2 2024-04-26T04:24:57.554 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:24:57.554 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_age": 1, 2024-04-26T04:24:57.554 INFO:teuthology.orchestra.run.smithi063.stdout: "features": { 2024-04-26T04:24:57.555 INFO:teuthology.orchestra.run.smithi063.stdout: "required_con": "2450521708007866372", 2024-04-26T04:24:57.555 INFO:teuthology.orchestra.run.smithi063.stdout: "required_mon": [ 2024-04-26T04:24:57.555 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:24:57.555 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:24:57.555 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:24:57.555 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:24:57.555 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:24:57.555 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:24:57.555 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:24:57.555 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:24:57.555 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:24:57.555 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:24:57.555 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:24:57.555 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:24:57.556 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_con": "4540701547738038271", 2024-04-26T04:24:57.556 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_mon": [ 2024-04-26T04:24:57.556 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:24:57.556 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:24:57.556 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:24:57.556 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:24:57.556 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:24:57.556 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:24:57.556 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:24:57.556 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:24:57.556 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:24:57.556 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:24:57.557 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:24:57.557 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:24:57.557 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:57.557 INFO:teuthology.orchestra.run.smithi063.stdout: "outside_quorum": [], 2024-04-26T04:24:57.557 INFO:teuthology.orchestra.run.smithi063.stdout: "extra_probe_peers": [], 2024-04-26T04:24:57.557 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_provider": [], 2024-04-26T04:24:57.557 INFO:teuthology.orchestra.run.smithi063.stdout: "monmap": { 2024-04-26T04:24:57.557 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 1, 2024-04-26T04:24:57.557 INFO:teuthology.orchestra.run.smithi063.stdout: "fsid": "cd1c6a0c-4e4b-4014-89c0-2e535479ef39", 2024-04-26T04:24:57.557 INFO:teuthology.orchestra.run.smithi063.stdout: "modified": "2024-04-26T04:18:03.702339Z", 2024-04-26T04:24:57.557 INFO:teuthology.orchestra.run.smithi063.stdout: "created": "2024-04-26T04:18:03.702339Z", 2024-04-26T04:24:57.557 INFO:teuthology.orchestra.run.smithi063.stdout: "min_mon_release": 19, 2024-04-26T04:24:57.557 INFO:teuthology.orchestra.run.smithi063.stdout: "min_mon_release_name": "squid", 2024-04-26T04:24:57.557 INFO:teuthology.orchestra.run.smithi063.stdout: "election_strategy": 1, 2024-04-26T04:24:57.557 INFO:teuthology.orchestra.run.smithi063.stdout: "disallowed_leaders: ": "", 2024-04-26T04:24:57.558 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": false, 2024-04-26T04:24:57.558 INFO:teuthology.orchestra.run.smithi063.stdout: "tiebreaker_mon": "", 2024-04-26T04:24:57.558 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_ranks: ": "", 2024-04-26T04:24:57.558 INFO:teuthology.orchestra.run.smithi063.stdout: "features": { 2024-04-26T04:24:57.558 INFO:teuthology.orchestra.run.smithi063.stdout: "persistent": [ 2024-04-26T04:24:57.558 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:24:57.558 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:24:57.558 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:24:57.558 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:24:57.558 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:24:57.558 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:24:57.558 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:24:57.558 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:24:57.558 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:24:57.559 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:24:57.559 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:24:57.559 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:24:57.559 INFO:teuthology.orchestra.run.smithi063.stdout: "optional": [] 2024-04-26T04:24:57.559 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:57.559 INFO:teuthology.orchestra.run.smithi063.stdout: "mons": [ 2024-04-26T04:24:57.559 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:57.559 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 0, 2024-04-26T04:24:57.559 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a", 2024-04-26T04:24:57.559 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:24:57.559 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:24:57.559 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:57.559 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:24:57.559 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:3300", 2024-04-26T04:24:57.559 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:24:57.560 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:57.560 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:57.560 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:24:57.560 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6789", 2024-04-26T04:24:57.560 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:24:57.560 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:24:57.560 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:24:57.560 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:57.560 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6789/0", 2024-04-26T04:24:57.560 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6789/0", 2024-04-26T04:24:57.560 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:24:57.560 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:24:57.560 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:24:57.560 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:57.560 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:57.561 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 1, 2024-04-26T04:24:57.561 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b", 2024-04-26T04:24:57.561 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:24:57.561 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:24:57.561 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:57.561 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:24:57.561 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:3300", 2024-04-26T04:24:57.561 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:24:57.561 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:57.561 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:57.561 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:24:57.561 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:6789", 2024-04-26T04:24:57.561 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:24:57.561 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:24:57.561 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:24:57.562 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:57.562 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:6789/0", 2024-04-26T04:24:57.562 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.186:6789/0", 2024-04-26T04:24:57.562 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:24:57.562 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:24:57.562 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:24:57.562 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:57.562 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:57.562 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 2, 2024-04-26T04:24:57.562 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "c", 2024-04-26T04:24:57.562 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:24:57.562 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:24:57.562 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:57.562 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:24:57.562 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:3301", 2024-04-26T04:24:57.563 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:24:57.563 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:57.563 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:57.563 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:24:57.563 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6790", 2024-04-26T04:24:57.563 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:24:57.563 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:24:57.563 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:24:57.563 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:57.563 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6790/0", 2024-04-26T04:24:57.563 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6790/0", 2024-04-26T04:24:57.563 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:24:57.563 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:24:57.563 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:24:57.563 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:24:57.564 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:24:57.564 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:57.564 INFO:teuthology.orchestra.run.smithi063.stdout: "feature_map": { 2024-04-26T04:24:57.564 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": [ 2024-04-26T04:24:57.564 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:57.564 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:24:57.564 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:24:57.564 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 1 2024-04-26T04:24:57.564 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:24:57.564 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:24:57.564 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": [ 2024-04-26T04:24:57.564 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:57.564 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:24:57.564 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:24:57.564 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 5 2024-04-26T04:24:57.564 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:24:57.565 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:24:57.565 INFO:teuthology.orchestra.run.smithi063.stdout: "client": [ 2024-04-26T04:24:57.565 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:57.565 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:24:57.565 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:24:57.565 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 5 2024-04-26T04:24:57.565 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:24:57.565 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:24:57.565 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:57.565 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": false 2024-04-26T04:24:57.565 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-26T04:24:57.566 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.543+0000 7f416fa7d640 1 -- 172.21.15.63:0/2216364273 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f414c035e70 msgr2=0x7f414c038330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:57.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.543+0000 7f416fa7d640 1 --2- 172.21.15.63:0/2216364273 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f414c035e70 0x7f414c038330 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f416806ecf0 tx=0x7f4158008040 comp rx=0 tx=0).stop 2024-04-26T04:24:57.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.543+0000 7f416fa7d640 10 monclient: shutdown 2024-04-26T04:24:57.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.543+0000 7f416fa7d640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:24:57.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.543+0000 7f416fa7d640 1 -- 172.21.15.63:0/2216364273 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4168139aa0 msgr2=0x7f416811ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:57.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.543+0000 7f416fa7d640 1 --2- 172.21.15.63:0/2216364273 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4168139aa0 0x7f416811ffb0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f416000ba40 tx=0x7f416000bf00 comp rx=0 tx=0).stop 2024-04-26T04:24:57.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.543+0000 7f416e01a640 1 -- 172.21.15.63:0/2216364273 reap_dead start 2024-04-26T04:24:57.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.543+0000 7f415e7fc640 1 -- 172.21.15.63:0/2216364273 <== mon.2 v2:172.21.15.63:3301/0 5 ==== osd_map(224..224 src has 1..224) ==== 4738+0+0 (unknown 0 0 0) 0x7f4160049200 con 0x7f4168139aa0 2024-04-26T04:24:57.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.543+0000 7f416fa7d640 1 -- 172.21.15.63:0/2216364273 shutdown_connections 2024-04-26T04:24:57.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.543+0000 7f416fa7d640 1 --2- 172.21.15.63:0/2216364273 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4168139aa0 0x7f416811ffb0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:57.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.543+0000 7f416fa7d640 1 -- 172.21.15.63:0/2216364273 >> 172.21.15.63:0/2216364273 conn(0x7f41680a6170 msgr2=0x7f41680a0390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:24:57.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.543+0000 7f416fa7d640 1 -- 172.21.15.63:0/2216364273 shutdown_connections 2024-04-26T04:24:57.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.543+0000 7f416fa7d640 1 -- 172.21.15.63:0/2216364273 wait complete. 2024-04-26T04:24:57.568 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-26T04:24:57.568 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-04-26T04:24:57.568 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T04:24:57.682 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.679+0000 7f56b6f80640 10 monclient: get_monmap_and_config 2024-04-26T04:24:57.682 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.679+0000 7f56b6f80640 10 monclient: build_initial_monmap 2024-04-26T04:24:57.682 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.679+0000 7f56b6f80640 10 monclient: monmap: 2024-04-26T04:24:57.682 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:24:57.682 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:57.682 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:24:57.682002+0000 2024-04-26T04:24:57.682 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:24:57.682002+0000 2024-04-26T04:24:57.682 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:24:57.683 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:57.683 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:24:57.683 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:24:57.683 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:24:57.683 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:24:57.683 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:24:57.683 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:57.685 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.679+0000 7f56b6f80640 1 Processor -- start 2024-04-26T04:24:57.685 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.679+0000 7f56b6f80640 1 -- start start 2024-04-26T04:24:57.685 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.679+0000 7f56b6f80640 10 monclient: init 2024-04-26T04:24:57.686 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.679+0000 7f56b6f80640 10 monclient: _reopen_session rank -1 2024-04-26T04:24:57.686 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.679+0000 7f56b6f80640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-04-26T04:24:57.686 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.679+0000 7f56b6f80640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f56b00a2cd0 addr v1:172.21.15.186:6789/0 2024-04-26T04:24:57.686 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.679+0000 7f56b6f80640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f56b00a30b0 0x7f56b013c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:57.686 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.679+0000 7f56b6f80640 10 monclient(hunting): picked mon.noname-b con 0x7f56b00a30b0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:24:57.686 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.679+0000 7f56b6f80640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f56b013c540 0x7f56b013e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:57.686 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.679+0000 7f56b6f80640 10 monclient(hunting): picked mon.noname-c con 0x7f56b013c540 addr v2:172.21.15.186:3300/0 2024-04-26T04:24:57.686 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.679+0000 7f56b6f80640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56b00a4ee0 con 0x7f56b00a2cd0 2024-04-26T04:24:57.686 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.679+0000 7f56b6f80640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:57.686 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.679+0000 7f56b6f80640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f56b00a0fb0 con 0x7f56b013c540 2024-04-26T04:24:57.686 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.679+0000 7f56b6f80640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:57.686 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.679+0000 7f56b6f80640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f56b00a5380 con 0x7f56b00a30b0 2024-04-26T04:24:57.686 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.679+0000 7f56b6f80640 10 monclient(hunting): _renew_subs 2024-04-26T04:24:57.686 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.679+0000 7f56b6f80640 10 monclient(hunting): authenticate will time out at 1329.389655s 2024-04-26T04:24:57.687 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.679+0000 7f56affff640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f56b00a30b0 0x7f56b013c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:57.687 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.679+0000 7f56affff640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f56b00a30b0 0x7f56b013c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:34092/0 (socket says 172.21.15.63:34092) 2024-04-26T04:24:57.687 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.679+0000 7f56affff640 1 -- 172.21.15.63:0/2937616530 learned_addr learned my addr 172.21.15.63:0/2937616530 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:24:57.687 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.679+0000 7f56affff640 10 monclient(hunting): get_auth_request con 0x7f56b00a30b0 auth_method 0 2024-04-26T04:24:57.687 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.679+0000 7f56affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:57.687 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.679+0000 7f56affff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:57.687 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.679+0000 7f56affff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:57.687 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.679+0000 7f56affff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:57.687 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.679+0000 7f56affff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:57.687 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.679+0000 7f56affff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:24:57.687 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.679+0000 7f56b551d640 1 --2- 172.21.15.63:0/2937616530 >> v2:172.21.15.186:3300/0 conn(0x7f56b013c540 0x7f56b013e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:57.687 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.679+0000 7f56b551d640 10 monclient(hunting): get_auth_request con 0x7f56b013c540 auth_method 0 2024-04-26T04:24:57.687 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.679+0000 7f56b551d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:57.687 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.679+0000 7f56b551d640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:57.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.679+0000 7f56b551d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:57.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.679+0000 7f56affff640 10 monclient(hunting): handle_auth_done global_id 24320 payload 274 2024-04-26T04:24:57.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.679+0000 7f56affff640 1 -- 172.21.15.63:0/2937616530 >> v2:172.21.15.186:3300/0 conn(0x7f56b013c540 msgr2=0x7f56b013e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:24:57.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.679+0000 7f56affff640 1 --2- 172.21.15.63:0/2937616530 >> v2:172.21.15.186:3300/0 conn(0x7f56b013c540 0x7f56b013e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:57.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.679+0000 7f56affff640 1 -- 172.21.15.63:0/2937616530 >> v1:172.21.15.186:6789/0 conn(0x7f56b00a2cd0 legacy=0x7f56b00a47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:24:57.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.679+0000 7f56affff640 10 monclient: _finish_hunting 0 2024-04-26T04:24:57.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.679+0000 7f56affff640 1 monclient: found mon.noname-b 2024-04-26T04:24:57.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.679+0000 7f56affff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:24:57.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.679+0000 7f56affff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.63:3301/0 2024-04-26T04:24:57.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.679+0000 7f56affff640 1 -- 172.21.15.63:0/2937616530 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56b00a1290 con 0x7f56b00a30b0 2024-04-26T04:24:57.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.683+0000 7f56affff640 1 -- 172.21.15.63:0/2937616530 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f56b00a30b0 msgr2=0x7f56b013c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:24:57.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.683+0000 7f56affff640 1 --2- 172.21.15.63:0/2937616530 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f56b00a30b0 0x7f56b013c000 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f569c009870 tx=0x7f569c01c470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad23001df5302175 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:57.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.683+0000 7f56af7fe640 1 -- 172.21.15.63:0/2937616530 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f569c0b0070 con 0x7f56b00a30b0 2024-04-26T04:24:57.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.683+0000 7f56af7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:57.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.683+0000 7f56af7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T04:24:57.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.683+0000 7f56af7fe640 10 monclient: dump: 2024-04-26T04:24:57.693 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:57.693 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:57.693 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:57.693 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:57.693 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:57.694 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:57.694 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:57.694 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:57.694 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:57.694 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:57.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.683+0000 7f56af7fe640 10 monclient: _finish_auth 0 2024-04-26T04:24:57.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.683+0000 7f56af7fe640 10 monclient: _check_auth_tickets 2024-04-26T04:24:57.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.683+0000 7f56af7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:24:57.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.683+0000 7f56af7fe640 1 -- 172.21.15.63:0/2937616530 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f569c0ab600 con 0x7f56b00a30b0 2024-04-26T04:24:57.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.683+0000 7f56af7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T04:24:57.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.683+0000 7f56af7fe640 1 -- 172.21.15.63:0/2937616530 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f569c0ab9d0 con 0x7f56b00a30b0 2024-04-26T04:24:57.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.683+0000 7f56b6f80640 5 monclient: authenticate success, global_id 24320 2024-04-26T04:24:57.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.683+0000 7f56b6f80640 10 monclient: get_monmap_and_config success 2024-04-26T04:24:57.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.683+0000 7f56b6f80640 4 set_mon_vals no callback set 2024-04-26T04:24:57.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.683+0000 7f56af7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:57.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.683+0000 7f56af7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:24:57.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.683+0000 7f56af7fe640 10 monclient: dump: 2024-04-26T04:24:57.695 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:57.695 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:57.695 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:57.695 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:57.695 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:57.695 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:57.696 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:57.696 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:57.696 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:57.696 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:57.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.683+0000 7f56b6f80640 10 monclient: shutdown 2024-04-26T04:24:57.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.683+0000 7f56b6f80640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:24:57.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.683+0000 7f56b6f80640 1 -- 172.21.15.63:0/2937616530 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f56b00a30b0 msgr2=0x7f56b013c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:57.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.683+0000 7f56b6f80640 1 --2- 172.21.15.63:0/2937616530 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f56b00a30b0 0x7f56b013c000 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f569c009870 tx=0x7f569c01c470 comp rx=0 tx=0).stop 2024-04-26T04:24:57.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.683+0000 7f56b6f80640 1 -- 172.21.15.63:0/2937616530 shutdown_connections 2024-04-26T04:24:57.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.683+0000 7f56b6f80640 1 --2- 172.21.15.63:0/2937616530 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f56b00a30b0 0x7f56b013c000 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:57.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.683+0000 7f56b6f80640 1 --2- 172.21.15.63:0/2937616530 >> v2:172.21.15.186:3300/0 conn(0x7f56b013c540 0x7f56b013e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:57.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.683+0000 7f56b6f80640 1 -- 172.21.15.63:0/2937616530 >> 172.21.15.63:0/2937616530 conn(0x7f56b00a5f40 msgr2=0x7f56b00a6320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:24:57.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.683+0000 7f56b6f80640 1 -- 172.21.15.63:0/2937616530 shutdown_connections 2024-04-26T04:24:57.697 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.683+0000 7f56b6f80640 1 -- 172.21.15.63:0/2937616530 wait complete. 2024-04-26T04:24:57.697 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.687+0000 7f56b6f80640 10 monclient: build_initial_monmap 2024-04-26T04:24:57.697 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.687+0000 7f56b6f80640 10 monclient: monmap: 2024-04-26T04:24:57.697 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:24:57.697 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:24:57.697 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:24:57.697 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:24:57.697 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:24:57.697 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:57.697 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:24:57.697 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:24:57.697 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:24:57.697 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:57.697 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.687+0000 7f56b6f80640 1 Processor -- start 2024-04-26T04:24:57.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.687+0000 7f56b6f80640 1 -- start start 2024-04-26T04:24:57.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.687+0000 7f56b6f80640 10 monclient: init 2024-04-26T04:24:57.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.687+0000 7f56b6f80640 10 monclient: _reopen_session rank -1 2024-04-26T04:24:57.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.687+0000 7f56b6f80640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T04:24:57.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.687+0000 7f56b6f80640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f56b00a2cd0 0x7f56b01ceee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:57.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.687+0000 7f56b6f80640 10 monclient(hunting): picked mon.noname-a con 0x7f56b00a2cd0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:24:57.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.687+0000 7f56b6f80640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f56b00a30b0 0x7f56b01cf580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:57.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.687+0000 7f56b6f80640 10 monclient(hunting): picked mon.noname-c con 0x7f56b00a30b0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:24:57.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.687+0000 7f56b6f80640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56b013c540 0x7f56b0070100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:57.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.687+0000 7f56b6f80640 10 monclient(hunting): picked mon.noname-b con 0x7f56b013c540 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:24:57.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.687+0000 7f56b6f80640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:57.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.687+0000 7f56b6f80640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f56b0141a20 con 0x7f56b00a2cd0 2024-04-26T04:24:57.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.687+0000 7f56b6f80640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:57.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.687+0000 7f56b6f80640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f56b01418a0 con 0x7f56b013c540 2024-04-26T04:24:57.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.687+0000 7f56b6f80640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:57.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.687+0000 7f56b6f80640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f56b0141ba0 con 0x7f56b00a30b0 2024-04-26T04:24:57.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.687+0000 7f56b6f80640 10 monclient(hunting): _renew_subs 2024-04-26T04:24:57.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.687+0000 7f56b6f80640 10 monclient(hunting): authenticate will time out at 1329.397061s 2024-04-26T04:24:57.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.687+0000 7f56b4d1c640 1 -- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f56b00a2cd0 msgr2=0x7f56b01ceee0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:3300/0 2024-04-26T04:24:57.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.687+0000 7f56b4d1c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f56b00a2cd0 0x7f56b01ceee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:24:57.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.687+0000 7f56affff640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f56b00a30b0 0x7f56b01cf580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:57.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.687+0000 7f56affff640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f56b00a30b0 0x7f56b01cf580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:34106/0 (socket says 172.21.15.63:34106) 2024-04-26T04:24:57.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.687+0000 7f56affff640 1 -- 172.21.15.63:0/220060305 learned_addr learned my addr 172.21.15.63:0/220060305 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:24:57.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.687+0000 7f56affff640 10 monclient(hunting): get_auth_request con 0x7f56b00a30b0 auth_method 0 2024-04-26T04:24:57.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.687+0000 7f56affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:57.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.687+0000 7f56affff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:57.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.687+0000 7f56affff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:57.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.687+0000 7f56affff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:57.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.687+0000 7f56affff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:57.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.687+0000 7f56affff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:24:57.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.687+0000 7f56b551d640 1 --2- 172.21.15.63:0/220060305 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56b013c540 0x7f56b0070100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:57.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.687+0000 7f56b551d640 10 monclient(hunting): get_auth_request con 0x7f56b013c540 auth_method 0 2024-04-26T04:24:57.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.687+0000 7f56b551d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:57.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.687+0000 7f56b551d640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:57.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.687+0000 7f56b551d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:57.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.687+0000 7f56affff640 10 monclient(hunting): handle_auth_done global_id 24323 payload 824 2024-04-26T04:24:57.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.687+0000 7f56affff640 1 -- 172.21.15.63:0/220060305 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56b013c540 msgr2=0x7f56b0070100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:57.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.687+0000 7f56affff640 1 --2- 172.21.15.63:0/220060305 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56b013c540 0x7f56b0070100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:57.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.687+0000 7f56affff640 1 -- 172.21.15.63:0/220060305 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f56b00a2cd0 msgr2=0x7f56b01ceee0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T04:24:57.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.687+0000 7f56affff640 1 --2- 172.21.15.63:0/220060305 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f56b00a2cd0 0x7f56b01ceee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:57.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.687+0000 7f56affff640 10 monclient: _finish_hunting 0 2024-04-26T04:24:57.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.687+0000 7f56affff640 1 monclient: found mon.noname-c 2024-04-26T04:24:57.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.687+0000 7f56affff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:24:57.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.687+0000 7f56affff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.63:3301/0 2024-04-26T04:24:57.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.687+0000 7f56affff640 1 -- 172.21.15.63:0/220060305 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56b00727c0 con 0x7f56b00a30b0 2024-04-26T04:24:57.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.687+0000 7f56affff640 1 -- 172.21.15.63:0/220060305 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f56b00a30b0 msgr2=0x7f56b01cf580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:24:57.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.687+0000 7f56affff640 1 --2- 172.21.15.63:0/220060305 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f56b00a30b0 0x7f56b01cf580 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f569c01cd10 tx=0x7f569c019210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:57.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.691+0000 7f56ad7fa640 1 -- 172.21.15.63:0/220060305 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f569c0b9070 con 0x7f56b00a30b0 2024-04-26T04:24:57.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.691+0000 7f56ad7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:57.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.691+0000 7f56ad7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:24:57.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.691+0000 7f56ad7fa640 10 monclient: dump: 2024-04-26T04:24:57.705 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:57.705 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:57.706 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:57.706 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:57.706 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:57.706 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:57.706 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:57.706 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:57.706 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:57.706 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:57.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.691+0000 7f56ad7fa640 10 monclient: _finish_auth 0 2024-04-26T04:24:57.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.691+0000 7f56ad7fa640 10 monclient: _check_auth_tickets 2024-04-26T04:24:57.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.691+0000 7f56ad7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:24:57.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.691+0000 7f56ad7fa640 1 -- 172.21.15.63:0/220060305 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f569c0aa5b0 con 0x7f56b00a30b0 2024-04-26T04:24:57.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.691+0000 7f56b6f80640 5 monclient: authenticate success, global_id 24323 2024-04-26T04:24:57.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.691+0000 7f56b6f80640 10 monclient: _renew_subs 2024-04-26T04:24:57.706 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.691+0000 7f56b6f80640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:24:57.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.691+0000 7f56b6f80640 1 -- 172.21.15.63:0/220060305 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56b011fa90 con 0x7f56b00a30b0 2024-04-26T04:24:57.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.691+0000 7f56ad7fa640 10 monclient: handle_config config(0 keys) 2024-04-26T04:24:57.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.691+0000 7f56ad7fa640 1 -- 172.21.15.63:0/220060305 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f569c0b0040 con 0x7f56b00a30b0 2024-04-26T04:24:57.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.691+0000 7f56ad7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:57.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.691+0000 7f56ad7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:24:57.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.691+0000 7f56ad7fa640 10 monclient: dump: 2024-04-26T04:24:57.707 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:57.707 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:57.707 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:57.707 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:57.707 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:57.707 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:57.707 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:57.707 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:57.707 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:57.708 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:57.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.691+0000 7f56b6f80640 10 monclient: _renew_subs 2024-04-26T04:24:57.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.691+0000 7f56b6f80640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:24:57.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.691+0000 7f56aeffd640 4 set_mon_vals no callback set 2024-04-26T04:24:57.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.691+0000 7f56b6f80640 1 -- 172.21.15.63:0/220060305 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f56b00731f0 con 0x7f56b00a30b0 2024-04-26T04:24:57.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.691+0000 7f56b6f80640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:24:57.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.691+0000 7f56b6f80640 1 --2- 172.21.15.63:0/220060305 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56b01cfac0 0x7f56b0070640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:57.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.691+0000 7f56b6f80640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:57.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.691+0000 7f56b6f80640 1 -- 172.21.15.63:0/220060305 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f56b0062e10 con 0x7f56b01cfac0 2024-04-26T04:24:57.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.691+0000 7f56ad7fa640 1 -- 172.21.15.63:0/220060305 <== mon.2 v2:172.21.15.63:3301/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f569c0bc070 con 0x7f56b00a30b0 2024-04-26T04:24:57.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.691+0000 7f56b4d1c640 1 --2- 172.21.15.63:0/220060305 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56b01cfac0 0x7f56b0070640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:57.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.691+0000 7f56b4d1c640 10 monclient: get_auth_request con 0x7f56b01cfac0 auth_method 0 2024-04-26T04:24:57.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.691+0000 7f56b4d1c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:57.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.691+0000 7f56b4d1c640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:57.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.691+0000 7f56b4d1c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:57.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.691+0000 7f56ad7fa640 1 --2- 172.21.15.63:0/220060305 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f5694035e70 0x7f5694038330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:57.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.691+0000 7f56ad7fa640 1 -- 172.21.15.63:0/220060305 <== mon.2 v2:172.21.15.63:3301/0 5 ==== osd_map(224..224 src has 1..224) ==== 4738+0+0 (secure 0 0 0) 0x7f569c0e1ee0 con 0x7f56b00a30b0 2024-04-26T04:24:57.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.691+0000 7f56b4d1c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:57.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.691+0000 7f56b4d1c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:57.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.691+0000 7f56b4d1c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:24:57.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.691+0000 7f56b551d640 1 --2- 172.21.15.63:0/220060305 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f5694035e70 0x7f5694038330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:57.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.691+0000 7f56b551d640 10 monclient: get_auth_request con 0x7f5694035e70 auth_method 0 2024-04-26T04:24:57.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.691+0000 7f56b4d1c640 10 monclient(hunting): handle_auth_done global_id 34189 payload 824 2024-04-26T04:24:57.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.691+0000 7f56b4d1c640 1 -- 172.21.15.63:0/220060305 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f56b00008e0 con 0x7f56b01cfac0 2024-04-26T04:24:57.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.695+0000 7f56b4d1c640 1 -- 172.21.15.63:0/220060305 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56b01cfac0 msgr2=0x7f56b0070640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:24:57.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.695+0000 7f56b4d1c640 1 --2- 172.21.15.63:0/220060305 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56b01cfac0 0x7f56b0070640 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f56a4007fd0 tx=0x7f56a400c810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:57.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.695+0000 7f56b551d640 1 --2- 172.21.15.63:0/220060305 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f5694035e70 0x7f5694038330 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f56b00723f0 tx=0x7f56a800b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:57.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.695+0000 7f56ad7fa640 1 -- 172.21.15.63:0/220060305 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56a4017070 con 0x7f56b01cfac0 2024-04-26T04:24:57.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.695+0000 7f56ad7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:57.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.695+0000 7f56ad7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T04:24:57.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.695+0000 7f56ad7fa640 10 monclient: dump: 2024-04-26T04:24:57.710 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:57.710 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:57.710 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:57.710 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:57.710 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:57.710 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:57.710 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:57.710 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:57.710 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:57.710 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:57.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.695+0000 7f56ad7fa640 1 -- 172.21.15.63:0/220060305 <== mon.1 v2:172.21.15.186:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f56a40057d0 con 0x7f56b01cfac0 2024-04-26T04:24:57.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.695+0000 7f56ad7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:24:57.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.695+0000 7f56ad7fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:24:57.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.695+0000 7f56ad7fa640 1 -- 172.21.15.63:0/220060305 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56b01cfac0 msgr2=0x7f56b0070640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:57.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.695+0000 7f56ad7fa640 1 --2- 172.21.15.63:0/220060305 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56b01cfac0 0x7f56b0070640 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f56a4007fd0 tx=0x7f56a400c810 comp rx=0 tx=0).stop 2024-04-26T04:24:57.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.695+0000 7f56ad7fa640 1 -- 172.21.15.63:0/220060305 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56b01cfac0 msgr2=0x7f56b0070640 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:24:57.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.695+0000 7f56ad7fa640 1 --2- 172.21.15.63:0/220060305 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56b01cfac0 0x7f56b0070640 secure :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f56a4007fd0 tx=0x7f56a400c810 comp rx=0 tx=0).stop 2024-04-26T04:24:57.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.699+0000 7f56b6f80640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T04:24:57.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.699+0000 7f56b6f80640 1 --2- 172.21.15.63:0/220060305 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56b0070b80 0x7f56b0139de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:24:57.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.699+0000 7f56b6f80640 10 monclient(hunting): start opening mon connection 2024-04-26T04:24:57.711 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.699+0000 7f56b6f80640 1 -- 172.21.15.63:0/220060305 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f56b0070f90 con 0x7f56b0070b80 2024-04-26T04:24:57.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.703+0000 7f56b4d1c640 1 --2- 172.21.15.63:0/220060305 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56b0070b80 0x7f56b0139de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:24:57.712 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.703+0000 7f56b4d1c640 10 monclient: get_auth_request con 0x7f56b0070b80 auth_method 0 2024-04-26T04:24:57.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.703+0000 7f56b4d1c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:24:57.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.703+0000 7f56b4d1c640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:24:57.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.703+0000 7f56b4d1c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:24:57.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.703+0000 7f56b4d1c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:24:57.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.703+0000 7f56b4d1c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:24:57.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.703+0000 7f56b4d1c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:24:57.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.703+0000 7f56b4d1c640 10 monclient(hunting): handle_auth_done global_id 34192 payload 824 2024-04-26T04:24:57.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.703+0000 7f56b4d1c640 1 -- 172.21.15.63:0/220060305 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f56b0120670 con 0x7f56b0070b80 2024-04-26T04:24:57.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.703+0000 7f56b4d1c640 1 -- 172.21.15.63:0/220060305 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56b0070b80 msgr2=0x7f56b0139de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:24:57.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.703+0000 7f56b4d1c640 1 --2- 172.21.15.63:0/220060305 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56b0070b80 0x7f56b0139de0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f56a4007fd0 tx=0x7f56a40037f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:24:57.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.703+0000 7f56ad7fa640 1 -- 172.21.15.63:0/220060305 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56a4017040 con 0x7f56b0070b80 2024-04-26T04:24:57.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.703+0000 7f56ad7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:24:57.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.703+0000 7f56ad7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T04:24:57.713 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.703+0000 7f56ad7fa640 10 monclient: dump: 2024-04-26T04:24:57.714 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:24:57.714 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:24:57.714 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:57.714 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:24:57.714 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:24:57.714 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:24:57.714 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:24:57.714 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:24:57.714 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:24:57.714 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:24:57.714 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.707+0000 7f56ad7fa640 1 -- 172.21.15.63:0/220060305 <== mon.1 v2:172.21.15.186:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4881 (secure 0 0 0) 0x7f56a4012400 con 0x7f56b0070b80 2024-04-26T04:24:57.714 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.707+0000 7f56ad7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T04:24:57.714 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.707+0000 7f56ad7fa640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:24:57.714 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.707+0000 7f56ad7fa640 1 -- 172.21.15.63:0/220060305 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56b0070b80 msgr2=0x7f56b0139de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:57.715 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.707+0000 7f56ad7fa640 1 --2- 172.21.15.63:0/220060305 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56b0070b80 0x7f56b0139de0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f56a4007fd0 tx=0x7f56a40037f0 comp rx=0 tx=0).stop 2024-04-26T04:24:57.715 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.707+0000 7f56ad7fa640 1 -- 172.21.15.63:0/220060305 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56b0070b80 msgr2=0x7f56b0139de0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:24:57.715 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.707+0000 7f56ad7fa640 1 --2- 172.21.15.63:0/220060305 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56b0070b80 0x7f56b0139de0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:24:57.715 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-26T04:24:57.715 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b", 2024-04-26T04:24:57.715 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 1, 2024-04-26T04:24:57.715 INFO:teuthology.orchestra.run.smithi063.stdout: "state": "leader", 2024-04-26T04:24:57.715 INFO:teuthology.orchestra.run.smithi063.stdout: "election_epoch": 28, 2024-04-26T04:24:57.715 INFO:teuthology.orchestra.run.smithi063.stdout: "uptime": 30487, 2024-04-26T04:24:57.715 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum": [ 2024-04-26T04:24:57.715 INFO:teuthology.orchestra.run.smithi063.stdout: 1, 2024-04-26T04:24:57.716 INFO:teuthology.orchestra.run.smithi063.stdout: 2 2024-04-26T04:24:57.716 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:24:57.716 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_age": 1, 2024-04-26T04:24:57.716 INFO:teuthology.orchestra.run.smithi063.stdout: "features": { 2024-04-26T04:24:57.716 INFO:teuthology.orchestra.run.smithi063.stdout: "required_con": "2450521708007866372", 2024-04-26T04:24:57.716 INFO:teuthology.orchestra.run.smithi063.stdout: "required_mon": [ 2024-04-26T04:24:57.716 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:24:57.716 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:24:57.716 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:24:57.716 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:24:57.716 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:24:57.716 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:24:57.716 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:24:57.716 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:24:57.717 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:24:57.717 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:24:57.717 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:24:57.717 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:24:57.717 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_con": "4540701547738038271", 2024-04-26T04:24:57.717 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_mon": [ 2024-04-26T04:24:57.717 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:24:57.717 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:24:57.717 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:24:57.717 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:24:57.717 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:24:57.717 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:24:57.717 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:24:57.717 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:24:57.718 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:24:57.718 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:24:57.718 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:24:57.718 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:24:57.718 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:57.718 INFO:teuthology.orchestra.run.smithi063.stdout: "outside_quorum": [], 2024-04-26T04:24:57.718 INFO:teuthology.orchestra.run.smithi063.stdout: "extra_probe_peers": [], 2024-04-26T04:24:57.718 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_provider": [], 2024-04-26T04:24:57.718 INFO:teuthology.orchestra.run.smithi063.stdout: "monmap": { 2024-04-26T04:24:57.718 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 1, 2024-04-26T04:24:57.718 INFO:teuthology.orchestra.run.smithi063.stdout: "fsid": "cd1c6a0c-4e4b-4014-89c0-2e535479ef39", 2024-04-26T04:24:57.718 INFO:teuthology.orchestra.run.smithi063.stdout: "modified": "2024-04-26T04:18:03.702339Z", 2024-04-26T04:24:57.718 INFO:teuthology.orchestra.run.smithi063.stdout: "created": "2024-04-26T04:18:03.702339Z", 2024-04-26T04:24:57.718 INFO:teuthology.orchestra.run.smithi063.stdout: "min_mon_release": 19, 2024-04-26T04:24:57.719 INFO:teuthology.orchestra.run.smithi063.stdout: "min_mon_release_name": "squid", 2024-04-26T04:24:57.719 INFO:teuthology.orchestra.run.smithi063.stdout: "election_strategy": 1, 2024-04-26T04:24:57.719 INFO:teuthology.orchestra.run.smithi063.stdout: "disallowed_leaders: ": "", 2024-04-26T04:24:57.719 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": false, 2024-04-26T04:24:57.719 INFO:teuthology.orchestra.run.smithi063.stdout: "tiebreaker_mon": "", 2024-04-26T04:24:57.719 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_ranks: ": "", 2024-04-26T04:24:57.719 INFO:teuthology.orchestra.run.smithi063.stdout: "features": { 2024-04-26T04:24:57.719 INFO:teuthology.orchestra.run.smithi063.stdout: "persistent": [ 2024-04-26T04:24:57.719 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:24:57.719 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:24:57.719 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:24:57.719 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:24:57.719 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:24:57.719 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:24:57.720 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:24:57.720 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:24:57.720 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:24:57.720 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:24:57.720 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:24:57.720 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:24:57.720 INFO:teuthology.orchestra.run.smithi063.stdout: "optional": [] 2024-04-26T04:24:57.720 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:57.720 INFO:teuthology.orchestra.run.smithi063.stdout: "mons": [ 2024-04-26T04:24:57.720 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:57.720 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 0, 2024-04-26T04:24:57.720 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a", 2024-04-26T04:24:57.720 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:24:57.720 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:24:57.720 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:57.721 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:24:57.721 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:3300", 2024-04-26T04:24:57.721 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:24:57.721 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:57.721 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:57.721 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:24:57.721 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6789", 2024-04-26T04:24:57.721 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:24:57.721 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:24:57.721 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:24:57.721 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:57.721 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6789/0", 2024-04-26T04:24:57.721 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6789/0", 2024-04-26T04:24:57.721 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:24:57.722 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:24:57.722 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:24:57.722 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:57.722 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:57.722 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 1, 2024-04-26T04:24:57.722 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b", 2024-04-26T04:24:57.722 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:24:57.722 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:24:57.722 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:57.722 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:24:57.722 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:3300", 2024-04-26T04:24:57.722 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:24:57.722 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:57.722 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:57.723 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:24:57.723 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:6789", 2024-04-26T04:24:57.723 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:24:57.723 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:24:57.723 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:24:57.723 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:57.723 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:6789/0", 2024-04-26T04:24:57.723 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.186:6789/0", 2024-04-26T04:24:57.723 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:24:57.723 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:24:57.723 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:24:57.723 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:57.723 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:57.723 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 2, 2024-04-26T04:24:57.723 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "c", 2024-04-26T04:24:57.724 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:24:57.724 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:24:57.724 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:57.724 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:24:57.724 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:3301", 2024-04-26T04:24:57.724 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:24:57.724 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:57.724 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:57.724 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:24:57.724 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6790", 2024-04-26T04:24:57.724 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:24:57.724 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:24:57.724 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:24:57.724 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:57.725 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6790/0", 2024-04-26T04:24:57.725 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6790/0", 2024-04-26T04:24:57.725 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:24:57.725 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:24:57.725 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:24:57.725 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:24:57.725 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:24:57.725 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:57.725 INFO:teuthology.orchestra.run.smithi063.stdout: "feature_map": { 2024-04-26T04:24:57.725 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": [ 2024-04-26T04:24:57.725 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:57.725 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:24:57.725 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:24:57.725 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 1 2024-04-26T04:24:57.725 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:24:57.726 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:24:57.726 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": [ 2024-04-26T04:24:57.726 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:57.726 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:24:57.726 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:24:57.726 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 1 2024-04-26T04:24:57.726 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:24:57.726 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:24:57.726 INFO:teuthology.orchestra.run.smithi063.stdout: "client": [ 2024-04-26T04:24:57.726 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:57.726 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:24:57.726 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:24:57.726 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 5 2024-04-26T04:24:57.726 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:24:57.727 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:24:57.727 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": [ 2024-04-26T04:24:57.727 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:24:57.727 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:24:57.727 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:24:57.727 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 1 2024-04-26T04:24:57.727 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:24:57.727 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:24:57.727 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:24:57.727 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": false 2024-04-26T04:24:57.727 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-26T04:24:57.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.707+0000 7f56b6f80640 1 -- 172.21.15.63:0/220060305 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f5694035e70 msgr2=0x7f5694038330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:57.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.707+0000 7f56b6f80640 1 --2- 172.21.15.63:0/220060305 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f5694035e70 0x7f5694038330 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f56b00723f0 tx=0x7f56a800b040 comp rx=0 tx=0).stop 2024-04-26T04:24:57.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.707+0000 7f56b6f80640 10 monclient: shutdown 2024-04-26T04:24:57.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.707+0000 7f56b6f80640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:24:57.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.707+0000 7f56b6f80640 1 -- 172.21.15.63:0/220060305 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f56b00a30b0 msgr2=0x7f56b01cf580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:24:57.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.707+0000 7f56b6f80640 1 --2- 172.21.15.63:0/220060305 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f56b00a30b0 0x7f56b01cf580 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f569c01cd10 tx=0x7f569c019210 comp rx=0 tx=0).stop 2024-04-26T04:24:57.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.707+0000 7f56b551d640 1 -- 172.21.15.63:0/220060305 reap_dead start 2024-04-26T04:24:57.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.707+0000 7f56b551d640 1 -- 172.21.15.63:0/220060305 reap_dead start 2024-04-26T04:24:57.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.707+0000 7f56b6f80640 1 -- 172.21.15.63:0/220060305 shutdown_connections 2024-04-26T04:24:57.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.707+0000 7f56b6f80640 1 -- 172.21.15.63:0/220060305 >> 172.21.15.63:0/220060305 conn(0x7f56b00a5f40 msgr2=0x7f56b00a1a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:24:57.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.707+0000 7f56b6f80640 1 -- 172.21.15.63:0/220060305 shutdown_connections 2024-04-26T04:24:57.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:24:57.707+0000 7f56b6f80640 1 -- 172.21.15.63:0/220060305 wait complete. 2024-04-26T04:24:57.729 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-04-26T04:24:57.730 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-04-26T04:24:58.857 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:24:58.858 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30669]: starting. 2024-04-26T04:24:58.858 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30669]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-30634 2024-04-26T04:24:58.858 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30669]: created object 0... 2024-04-26T04:24:58.858 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30669]: created object 25... 2024-04-26T04:24:58.858 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30669]: created object 49... 2024-04-26T04:24:58.858 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30669]: finishing. 2024-04-26T04:24:58.858 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30669]: shutting down. 2024-04-26T04:24:59.765 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:24:59.765 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[30671]: starting. 2024-04-26T04:24:59.765 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[30671]: listing objects. 2024-04-26T04:24:59.765 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[30671]: listed object 0... 2024-04-26T04:24:59.765 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[30671]: listed object 25... 2024-04-26T04:24:59.765 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[30671]: saw 50 objects 2024-04-26T04:24:59.765 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[30671]: shutting down. 2024-04-26T04:25:00.650 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:25:00.651 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[30670]: starting. 2024-04-26T04:25:00.651 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[30670]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-30634 2024-04-26T04:25:00.651 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[30670]: shutting down. 2024-04-26T04:25:00.653 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:25:00.653 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:25:00.653 INFO:teuthology.orchestra.run.smithi186.stdout:******* SUCCESS ********** 2024-04-26T04:25:00.656 DEBUG:teuthology.orchestra.run.smithi186:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-26T04:25:03.421 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[30724]: starting. 2024-04-26T04:25:03.421 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[30724]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-30723 2024-04-26T04:25:03.422 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[30724]: created object 0... 2024-04-26T04:25:03.422 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[30724]: created object 25... 2024-04-26T04:25:03.422 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[30724]: created object 49... 2024-04-26T04:25:03.422 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[30724]: finishing. 2024-04-26T04:25:03.422 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[30724]: shutting down. 2024-04-26T04:25:04.260 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[30725]: starting. 2024-04-26T04:25:04.260 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[30725]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-30723 2024-04-26T04:25:04.260 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[30725]: shutting down. 2024-04-26T04:25:06.501 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:25:06.501 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30763]: starting. 2024-04-26T04:25:06.501 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30763]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-30723 2024-04-26T04:25:06.501 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30763]: created object 0... 2024-04-26T04:25:06.501 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30763]: created object 25... 2024-04-26T04:25:06.501 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30763]: created object 49... 2024-04-26T04:25:06.501 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30763]: finishing. 2024-04-26T04:25:06.501 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30763]: shutting down. 2024-04-26T04:25:06.559 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:25:06.559 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[30765]: starting. 2024-04-26T04:25:06.559 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[30765]: listing objects. 2024-04-26T04:25:06.559 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[30765]: listed object 0... 2024-04-26T04:25:06.559 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[30765]: listed object 25... 2024-04-26T04:25:06.560 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[30765]: saw 50 objects 2024-04-26T04:25:06.560 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[30765]: shutting down. 2024-04-26T04:25:07.284 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:25:07.284 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[30764]: starting. 2024-04-26T04:25:07.284 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[30764]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-30723 2024-04-26T04:25:07.284 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[30764]: shutting down. 2024-04-26T04:25:07.288 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:25:07.288 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:25:07.288 INFO:teuthology.orchestra.run.smithi186.stdout:******* SUCCESS ********** 2024-04-26T04:25:07.291 DEBUG:teuthology.orchestra.run.smithi186:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-26T04:25:09.527 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[30818]: starting. 2024-04-26T04:25:09.527 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[30818]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-30817 2024-04-26T04:25:09.527 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[30818]: created object 0... 2024-04-26T04:25:09.527 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[30818]: created object 25... 2024-04-26T04:25:09.528 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[30818]: created object 49... 2024-04-26T04:25:09.528 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[30818]: finishing. 2024-04-26T04:25:09.528 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[30818]: shutting down. 2024-04-26T04:25:10.319 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[30819]: starting. 2024-04-26T04:25:10.320 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[30819]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-30817 2024-04-26T04:25:10.320 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[30819]: shutting down. 2024-04-26T04:25:12.466 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:25:12.466 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30854]: starting. 2024-04-26T04:25:12.466 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30854]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-30817 2024-04-26T04:25:12.466 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30854]: created object 0... 2024-04-26T04:25:12.466 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30854]: created object 25... 2024-04-26T04:25:12.466 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30854]: created object 49... 2024-04-26T04:25:12.466 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30854]: finishing. 2024-04-26T04:25:12.466 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30854]: shutting down. 2024-04-26T04:25:13.377 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:25:13.377 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[30856]: starting. 2024-04-26T04:25:13.377 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[30856]: listing objects. 2024-04-26T04:25:13.377 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[30856]: listed object 0... 2024-04-26T04:25:13.377 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[30856]: listed object 25... 2024-04-26T04:25:13.377 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[30856]: saw 50 objects 2024-04-26T04:25:13.378 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[30856]: shutting down. 2024-04-26T04:25:14.271 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:25:14.271 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[30855]: starting. 2024-04-26T04:25:14.271 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[30855]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-30817 2024-04-26T04:25:14.272 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[30855]: shutting down. 2024-04-26T04:25:14.275 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:25:14.275 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:25:14.275 INFO:teuthology.orchestra.run.smithi186.stdout:******* SUCCESS ********** 2024-04-26T04:25:14.278 DEBUG:teuthology.orchestra.run.smithi186:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-26T04:25:16.500 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[30909]: starting. 2024-04-26T04:25:16.500 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[30909]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-30908 2024-04-26T04:25:16.500 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[30909]: created object 0... 2024-04-26T04:25:16.500 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[30909]: created object 25... 2024-04-26T04:25:16.501 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[30909]: created object 49... 2024-04-26T04:25:16.501 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[30909]: finishing. 2024-04-26T04:25:16.501 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[30909]: shutting down. 2024-04-26T04:25:17.280 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[30910]: starting. 2024-04-26T04:25:17.281 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[30910]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-30908 2024-04-26T04:25:17.281 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[30910]: shutting down. 2024-04-26T04:25:19.786 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:25:19.786 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30945]: starting. 2024-04-26T04:25:19.786 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30945]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-30908 2024-04-26T04:25:19.786 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30945]: created object 0... 2024-04-26T04:25:19.786 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30945]: created object 25... 2024-04-26T04:25:19.786 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30945]: created object 49... 2024-04-26T04:25:19.786 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30945]: finishing. 2024-04-26T04:25:19.786 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[30945]: shutting down. 2024-04-26T04:25:19.848 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:25:19.848 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[30947]: starting. 2024-04-26T04:25:19.848 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[30947]: listing objects. 2024-04-26T04:25:19.849 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[30947]: listed object 0... 2024-04-26T04:25:19.849 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[30947]: listed object 25... 2024-04-26T04:25:19.849 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[30947]: saw 50 objects 2024-04-26T04:25:19.849 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[30947]: shutting down. 2024-04-26T04:25:20.286 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:25:20.287 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[30946]: starting. 2024-04-26T04:25:20.287 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[30946]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-30908 2024-04-26T04:25:20.287 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[30946]: shutting down. 2024-04-26T04:25:20.290 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:25:20.290 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:25:20.290 INFO:teuthology.orchestra.run.smithi186.stdout:******* SUCCESS ********** 2024-04-26T04:25:20.294 DEBUG:teuthology.orchestra.run.smithi186:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-26T04:25:22.468 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31000]: starting. 2024-04-26T04:25:22.468 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31000]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-30999 2024-04-26T04:25:22.468 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31000]: created object 0... 2024-04-26T04:25:22.468 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31000]: created object 25... 2024-04-26T04:25:22.469 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31000]: created object 49... 2024-04-26T04:25:22.469 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31000]: finishing. 2024-04-26T04:25:22.469 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31000]: shutting down. 2024-04-26T04:25:23.258 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[31001]: starting. 2024-04-26T04:25:23.259 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[31001]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-30999 2024-04-26T04:25:23.259 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[31001]: shutting down. 2024-04-26T04:25:25.591 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:25:25.592 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31036]: starting. 2024-04-26T04:25:25.592 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31036]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-30999 2024-04-26T04:25:25.592 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31036]: created object 0... 2024-04-26T04:25:25.592 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31036]: created object 25... 2024-04-26T04:25:25.592 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31036]: created object 49... 2024-04-26T04:25:25.592 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31036]: finishing. 2024-04-26T04:25:25.592 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31036]: shutting down. 2024-04-26T04:25:26.498 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:25:26.498 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[31038]: starting. 2024-04-26T04:25:26.498 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[31038]: listing objects. 2024-04-26T04:25:26.498 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[31038]: listed object 0... 2024-04-26T04:25:26.498 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[31038]: listed object 25... 2024-04-26T04:25:26.498 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[31038]: saw 50 objects 2024-04-26T04:25:26.498 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[31038]: shutting down. 2024-04-26T04:25:27.304 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:25:27.304 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[31037]: starting. 2024-04-26T04:25:27.304 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[31037]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-30999 2024-04-26T04:25:27.305 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[31037]: shutting down. 2024-04-26T04:25:27.307 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:25:27.307 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:25:27.307 INFO:teuthology.orchestra.run.smithi186.stdout:******* SUCCESS ********** 2024-04-26T04:25:27.310 DEBUG:teuthology.orchestra.run.smithi186:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-26T04:25:29.528 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31091]: starting. 2024-04-26T04:25:29.528 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31091]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-31090 2024-04-26T04:25:29.528 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31091]: created object 0... 2024-04-26T04:25:29.528 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31091]: created object 25... 2024-04-26T04:25:29.528 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31091]: created object 49... 2024-04-26T04:25:29.528 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31091]: finishing. 2024-04-26T04:25:29.528 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31091]: shutting down. 2024-04-26T04:25:30.330 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[31092]: starting. 2024-04-26T04:25:30.330 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[31092]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-31090 2024-04-26T04:25:30.330 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[31092]: shutting down. 2024-04-26T04:25:32.477 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:25:32.477 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31128]: starting. 2024-04-26T04:25:32.477 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31128]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-31090 2024-04-26T04:25:32.477 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31128]: created object 0... 2024-04-26T04:25:32.477 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31128]: created object 25... 2024-04-26T04:25:32.477 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31128]: created object 49... 2024-04-26T04:25:32.478 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31128]: finishing. 2024-04-26T04:25:32.478 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31128]: shutting down. 2024-04-26T04:25:32.534 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:25:32.534 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[31130]: starting. 2024-04-26T04:25:32.534 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[31130]: listing objects. 2024-04-26T04:25:32.534 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[31130]: listed object 0... 2024-04-26T04:25:32.534 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[31130]: listed object 25... 2024-04-26T04:25:32.535 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[31130]: saw 50 objects 2024-04-26T04:25:32.535 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[31130]: shutting down. 2024-04-26T04:25:33.266 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:25:33.266 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[31129]: starting. 2024-04-26T04:25:33.266 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[31129]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-31090 2024-04-26T04:25:33.266 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[31129]: shutting down. 2024-04-26T04:25:33.270 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:25:33.270 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:25:33.270 INFO:teuthology.orchestra.run.smithi186.stdout:******* SUCCESS ********** 2024-04-26T04:25:33.273 DEBUG:teuthology.orchestra.run.smithi186:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-26T04:25:35.485 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31183]: starting. 2024-04-26T04:25:35.485 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31183]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-31182 2024-04-26T04:25:35.485 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31183]: created object 0... 2024-04-26T04:25:35.485 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31183]: created object 25... 2024-04-26T04:25:35.486 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31183]: created object 49... 2024-04-26T04:25:35.486 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31183]: finishing. 2024-04-26T04:25:35.486 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31183]: shutting down. 2024-04-26T04:25:36.289 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[31184]: starting. 2024-04-26T04:25:36.289 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[31184]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-31182 2024-04-26T04:25:36.290 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[31184]: shutting down. 2024-04-26T04:25:39.058 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:25:39.058 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31219]: starting. 2024-04-26T04:25:39.058 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31219]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-31182 2024-04-26T04:25:39.058 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31219]: created object 0... 2024-04-26T04:25:39.058 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31219]: created object 25... 2024-04-26T04:25:39.058 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31219]: created object 49... 2024-04-26T04:25:39.058 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31219]: finishing. 2024-04-26T04:25:39.058 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31219]: shutting down. 2024-04-26T04:25:39.121 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:25:39.121 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[31221]: starting. 2024-04-26T04:25:39.121 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[31221]: listing objects. 2024-04-26T04:25:39.121 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[31221]: listed object 0... 2024-04-26T04:25:39.121 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[31221]: listed object 25... 2024-04-26T04:25:39.121 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[31221]: saw 50 objects 2024-04-26T04:25:39.121 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[31221]: shutting down. 2024-04-26T04:25:39.867 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:25:39.867 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[31220]: starting. 2024-04-26T04:25:39.867 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[31220]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-31182 2024-04-26T04:25:39.867 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[31220]: shutting down. 2024-04-26T04:25:39.871 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:25:39.871 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:25:39.871 INFO:teuthology.orchestra.run.smithi186.stdout:******* SUCCESS ********** 2024-04-26T04:25:39.875 DEBUG:teuthology.orchestra.run.smithi186:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-26T04:25:42.121 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31274]: starting. 2024-04-26T04:25:42.121 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31274]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-31273 2024-04-26T04:25:42.121 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31274]: created object 0... 2024-04-26T04:25:42.121 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31274]: created object 25... 2024-04-26T04:25:42.121 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31274]: created object 49... 2024-04-26T04:25:42.121 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31274]: finishing. 2024-04-26T04:25:42.121 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31274]: shutting down. 2024-04-26T04:25:42.267 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[31275]: starting. 2024-04-26T04:25:42.267 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[31275]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-31273 2024-04-26T04:25:42.267 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[31275]: shutting down. 2024-04-26T04:25:44.512 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:25:44.513 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31310]: starting. 2024-04-26T04:25:44.513 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31310]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-31273 2024-04-26T04:25:44.513 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31310]: created object 0... 2024-04-26T04:25:44.513 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31310]: created object 25... 2024-04-26T04:25:44.513 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31310]: created object 49... 2024-04-26T04:25:44.513 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31310]: finishing. 2024-04-26T04:25:44.513 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31310]: shutting down. 2024-04-26T04:25:44.583 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:25:44.583 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[31312]: starting. 2024-04-26T04:25:44.583 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[31312]: listing objects. 2024-04-26T04:25:44.583 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[31312]: listed object 0... 2024-04-26T04:25:44.583 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[31312]: listed object 25... 2024-04-26T04:25:44.583 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[31312]: saw 50 objects 2024-04-26T04:25:44.583 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[31312]: shutting down. 2024-04-26T04:25:45.280 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:25:45.280 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[31311]: starting. 2024-04-26T04:25:45.280 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[31311]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-31273 2024-04-26T04:25:45.280 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[31311]: shutting down. 2024-04-26T04:25:45.283 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:25:45.283 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:25:45.283 INFO:teuthology.orchestra.run.smithi186.stdout:******* SUCCESS ********** 2024-04-26T04:25:45.286 DEBUG:teuthology.orchestra.run.smithi186:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-26T04:25:47.629 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31365]: starting. 2024-04-26T04:25:47.629 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31365]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-31364 2024-04-26T04:25:47.629 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31365]: created object 0... 2024-04-26T04:25:47.629 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31365]: created object 25... 2024-04-26T04:25:47.629 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31365]: created object 49... 2024-04-26T04:25:47.629 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31365]: finishing. 2024-04-26T04:25:47.629 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31365]: shutting down. 2024-04-26T04:25:48.427 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[31366]: starting. 2024-04-26T04:25:48.427 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[31366]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-31364 2024-04-26T04:25:48.427 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[31366]: shutting down. 2024-04-26T04:25:50.667 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:25:50.667 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31401]: starting. 2024-04-26T04:25:50.667 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31401]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-31364 2024-04-26T04:25:50.667 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31401]: created object 0... 2024-04-26T04:25:50.667 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31401]: created object 25... 2024-04-26T04:25:50.667 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31401]: created object 49... 2024-04-26T04:25:50.668 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31401]: finishing. 2024-04-26T04:25:50.668 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31401]: shutting down. 2024-04-26T04:25:50.725 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:25:50.725 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[31403]: starting. 2024-04-26T04:25:50.725 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[31403]: listing objects. 2024-04-26T04:25:50.725 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[31403]: listed object 0... 2024-04-26T04:25:50.725 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[31403]: listed object 25... 2024-04-26T04:25:50.725 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[31403]: saw 50 objects 2024-04-26T04:25:50.725 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[31403]: shutting down. 2024-04-26T04:25:51.458 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:25:51.459 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[31402]: starting. 2024-04-26T04:25:51.459 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[31402]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-31364 2024-04-26T04:25:51.459 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[31402]: shutting down. 2024-04-26T04:25:51.461 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:25:51.461 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:25:51.461 INFO:teuthology.orchestra.run.smithi186.stdout:******* SUCCESS ********** 2024-04-26T04:25:51.464 DEBUG:teuthology.orchestra.run.smithi186:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-26T04:25:53.477 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31456]: starting. 2024-04-26T04:25:53.477 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31456]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-31455 2024-04-26T04:25:53.477 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31456]: created object 0... 2024-04-26T04:25:53.477 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31456]: created object 25... 2024-04-26T04:25:53.477 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31456]: created object 49... 2024-04-26T04:25:53.477 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31456]: finishing. 2024-04-26T04:25:53.477 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31456]: shutting down. 2024-04-26T04:25:54.282 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[31457]: starting. 2024-04-26T04:25:54.282 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[31457]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-31455 2024-04-26T04:25:54.282 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[31457]: shutting down. 2024-04-26T04:25:56.518 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:25:56.518 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31492]: starting. 2024-04-26T04:25:56.518 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31492]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-31455 2024-04-26T04:25:56.518 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31492]: created object 0... 2024-04-26T04:25:56.519 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31492]: created object 25... 2024-04-26T04:25:56.519 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31492]: created object 49... 2024-04-26T04:25:56.519 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31492]: finishing. 2024-04-26T04:25:56.519 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31492]: shutting down. 2024-04-26T04:25:56.783 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:25:56.783 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[31494]: starting. 2024-04-26T04:25:56.783 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[31494]: listing objects. 2024-04-26T04:25:56.783 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[31494]: listed object 0... 2024-04-26T04:25:56.783 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[31494]: listed object 25... 2024-04-26T04:25:56.784 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[31494]: saw 50 objects 2024-04-26T04:25:56.784 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[31494]: shutting down. 2024-04-26T04:25:57.298 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:25:57.298 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[31493]: starting. 2024-04-26T04:25:57.298 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[31493]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-31455 2024-04-26T04:25:57.298 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[31493]: shutting down. 2024-04-26T04:25:57.301 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:25:57.301 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:25:57.301 INFO:teuthology.orchestra.run.smithi186.stdout:******* SUCCESS ********** 2024-04-26T04:25:57.304 DEBUG:teuthology.orchestra.run.smithi186:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-26T04:25:59.526 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31547]: starting. 2024-04-26T04:25:59.526 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31547]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-31546 2024-04-26T04:25:59.526 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31547]: created object 0... 2024-04-26T04:25:59.526 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31547]: created object 25... 2024-04-26T04:25:59.526 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31547]: created object 49... 2024-04-26T04:25:59.526 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31547]: finishing. 2024-04-26T04:25:59.526 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31547]: shutting down. 2024-04-26T04:26:00.418 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[31548]: starting. 2024-04-26T04:26:00.418 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[31548]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-31546 2024-04-26T04:26:00.418 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[31548]: shutting down. 2024-04-26T04:26:02.485 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:26:02.485 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31583]: starting. 2024-04-26T04:26:02.485 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31583]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-31546 2024-04-26T04:26:02.485 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31583]: created object 0... 2024-04-26T04:26:02.485 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31583]: created object 25... 2024-04-26T04:26:02.485 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31583]: created object 49... 2024-04-26T04:26:02.486 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31583]: finishing. 2024-04-26T04:26:02.486 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31583]: shutting down. 2024-04-26T04:26:02.541 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:26:02.542 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[31585]: starting. 2024-04-26T04:26:02.542 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[31585]: listing objects. 2024-04-26T04:26:02.542 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[31585]: listed object 0... 2024-04-26T04:26:02.542 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[31585]: listed object 25... 2024-04-26T04:26:02.542 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[31585]: saw 50 objects 2024-04-26T04:26:02.542 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[31585]: shutting down. 2024-04-26T04:26:03.295 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:26:03.296 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[31584]: starting. 2024-04-26T04:26:03.296 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[31584]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-31546 2024-04-26T04:26:03.296 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[31584]: shutting down. 2024-04-26T04:26:03.298 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:26:03.298 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:26:03.299 INFO:teuthology.orchestra.run.smithi186.stdout:******* SUCCESS ********** 2024-04-26T04:26:03.302 DEBUG:teuthology.orchestra.run.smithi186:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-26T04:26:05.507 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31638]: starting. 2024-04-26T04:26:05.507 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31638]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-31637 2024-04-26T04:26:05.507 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31638]: created object 0... 2024-04-26T04:26:05.507 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31638]: created object 25... 2024-04-26T04:26:05.507 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31638]: created object 49... 2024-04-26T04:26:05.507 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31638]: finishing. 2024-04-26T04:26:05.507 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31638]: shutting down. 2024-04-26T04:26:06.300 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[31639]: starting. 2024-04-26T04:26:06.300 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[31639]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-31637 2024-04-26T04:26:06.301 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[31639]: shutting down. 2024-04-26T04:26:08.509 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:26:08.509 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31674]: starting. 2024-04-26T04:26:08.509 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31674]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-31637 2024-04-26T04:26:08.509 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31674]: created object 0... 2024-04-26T04:26:08.509 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31674]: created object 25... 2024-04-26T04:26:08.510 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31674]: created object 49... 2024-04-26T04:26:08.510 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31674]: finishing. 2024-04-26T04:26:08.510 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31674]: shutting down. 2024-04-26T04:26:08.570 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:26:08.570 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[31676]: starting. 2024-04-26T04:26:08.570 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[31676]: listing objects. 2024-04-26T04:26:08.570 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[31676]: listed object 0... 2024-04-26T04:26:08.570 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[31676]: listed object 25... 2024-04-26T04:26:08.570 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[31676]: saw 50 objects 2024-04-26T04:26:08.570 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[31676]: shutting down. 2024-04-26T04:26:09.322 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:26:09.323 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[31675]: starting. 2024-04-26T04:26:09.323 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[31675]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-31637 2024-04-26T04:26:09.323 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[31675]: shutting down. 2024-04-26T04:26:09.327 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:26:09.327 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:26:09.327 INFO:teuthology.orchestra.run.smithi186.stdout:******* SUCCESS ********** 2024-04-26T04:26:09.329 DEBUG:teuthology.orchestra.run.smithi186:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-26T04:26:11.758 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31729]: starting. 2024-04-26T04:26:11.758 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31729]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-31728 2024-04-26T04:26:11.758 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31729]: created object 0... 2024-04-26T04:26:11.758 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31729]: created object 25... 2024-04-26T04:26:11.758 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31729]: created object 49... 2024-04-26T04:26:11.758 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31729]: finishing. 2024-04-26T04:26:11.758 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31729]: shutting down. 2024-04-26T04:26:13.358 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[31730]: starting. 2024-04-26T04:26:13.358 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[31730]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-31728 2024-04-26T04:26:13.358 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[31730]: shutting down. 2024-04-26T04:26:15.574 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:26:15.574 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31765]: starting. 2024-04-26T04:26:15.575 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31765]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-31728 2024-04-26T04:26:15.575 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31765]: created object 0... 2024-04-26T04:26:15.575 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31765]: created object 25... 2024-04-26T04:26:15.575 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31765]: created object 49... 2024-04-26T04:26:15.575 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31765]: finishing. 2024-04-26T04:26:15.575 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31765]: shutting down. 2024-04-26T04:26:15.630 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:26:15.630 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[31767]: starting. 2024-04-26T04:26:15.630 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[31767]: listing objects. 2024-04-26T04:26:15.630 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[31767]: listed object 0... 2024-04-26T04:26:15.630 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[31767]: listed object 25... 2024-04-26T04:26:15.631 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[31767]: saw 50 objects 2024-04-26T04:26:15.631 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[31767]: shutting down. 2024-04-26T04:26:16.373 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:26:16.373 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[31766]: starting. 2024-04-26T04:26:16.373 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[31766]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-31728 2024-04-26T04:26:16.373 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[31766]: shutting down. 2024-04-26T04:26:16.375 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:26:16.375 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:26:16.375 INFO:teuthology.orchestra.run.smithi186.stdout:******* SUCCESS ********** 2024-04-26T04:26:16.378 DEBUG:teuthology.orchestra.run.smithi186:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-26T04:26:18.590 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31820]: starting. 2024-04-26T04:26:18.590 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31820]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-31819 2024-04-26T04:26:18.590 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31820]: created object 0... 2024-04-26T04:26:18.590 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31820]: created object 25... 2024-04-26T04:26:18.590 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31820]: created object 49... 2024-04-26T04:26:18.591 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31820]: finishing. 2024-04-26T04:26:18.591 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31820]: shutting down. 2024-04-26T04:26:19.393 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[31821]: starting. 2024-04-26T04:26:19.393 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[31821]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-31819 2024-04-26T04:26:19.393 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[31821]: shutting down. 2024-04-26T04:26:21.628 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:26:21.628 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31856]: starting. 2024-04-26T04:26:21.628 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31856]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-31819 2024-04-26T04:26:21.629 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31856]: created object 0... 2024-04-26T04:26:21.629 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31856]: created object 25... 2024-04-26T04:26:21.629 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31856]: created object 49... 2024-04-26T04:26:21.629 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31856]: finishing. 2024-04-26T04:26:21.629 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31856]: shutting down. 2024-04-26T04:26:22.542 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:26:22.543 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[31858]: starting. 2024-04-26T04:26:22.543 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[31858]: listing objects. 2024-04-26T04:26:22.543 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[31858]: listed object 0... 2024-04-26T04:26:22.543 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[31858]: listed object 25... 2024-04-26T04:26:22.543 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[31858]: saw 50 objects 2024-04-26T04:26:22.543 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[31858]: shutting down. 2024-04-26T04:26:23.432 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:26:23.433 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[31857]: starting. 2024-04-26T04:26:23.433 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[31857]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-31819 2024-04-26T04:26:23.433 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[31857]: shutting down. 2024-04-26T04:26:23.436 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:26:23.436 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:26:23.436 INFO:teuthology.orchestra.run.smithi186.stdout:******* SUCCESS ********** 2024-04-26T04:26:23.439 DEBUG:teuthology.orchestra.run.smithi186:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-26T04:26:25.644 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31911]: starting. 2024-04-26T04:26:25.644 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31911]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-31910 2024-04-26T04:26:25.644 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31911]: created object 0... 2024-04-26T04:26:25.644 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31911]: created object 25... 2024-04-26T04:26:25.644 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31911]: created object 49... 2024-04-26T04:26:25.644 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31911]: finishing. 2024-04-26T04:26:25.644 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[31911]: shutting down. 2024-04-26T04:26:26.442 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[31912]: starting. 2024-04-26T04:26:26.442 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[31912]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-31910 2024-04-26T04:26:26.442 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[31912]: shutting down. 2024-04-26T04:26:27.731 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-04-26T04:26:27.731 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-04-26T04:26:27.731 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-26T04:26:27.731 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-26T04:26:27.734 INFO:tasks.ceph.mon.a:Started 2024-04-26T04:26:27.734 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-26T04:26:27.734 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T04:26:27.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.850+0000 7ff5e9327640 10 monclient: get_monmap_and_config 2024-04-26T04:26:27.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.850+0000 7ff5e9327640 10 monclient: build_initial_monmap 2024-04-26T04:26:27.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.850+0000 7ff5e9327640 10 monclient: monmap: 2024-04-26T04:26:27.854 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:26:27.854 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:27.855 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:26:27.854633+0000 2024-04-26T04:26:27.855 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:26:27.854633+0000 2024-04-26T04:26:27.855 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:26:27.855 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:27.855 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:26:27.855 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:26:27.855 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:26:27.855 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:26:27.855 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:26:27.855 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:27.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.850+0000 7ff5e9327640 1 Processor -- start 2024-04-26T04:26:27.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.850+0000 7ff5e9327640 1 -- start start 2024-04-26T04:26:27.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.850+0000 7ff5e9327640 10 monclient: init 2024-04-26T04:26:27.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.850+0000 7ff5e9327640 10 monclient: _reopen_session rank -1 2024-04-26T04:26:27.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.850+0000 7ff5e9327640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-04-26T04:26:27.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.850+0000 7ff5e9327640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff5e409fe60 addr v1:172.21.15.186:6789/0 2024-04-26T04:26:27.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.850+0000 7ff5e9327640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff5e40a0240 0x7ff5e40a2a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:27.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.850+0000 7ff5e9327640 10 monclient(hunting): picked mon.noname-b con 0x7ff5e40a0240 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:26:27.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.850+0000 7ff5e9327640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7ff5e40a2f70 0x7ff5e413e110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:27.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.850+0000 7ff5e9327640 10 monclient(hunting): picked mon.noname-a con 0x7ff5e40a2f70 addr v2:172.21.15.63:3300/0 2024-04-26T04:26:27.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.850+0000 7ff5e9327640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5e40a0c60 con 0x7ff5e409fe60 2024-04-26T04:26:27.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.850+0000 7ff5e9327640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:27.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.850+0000 7ff5e9327640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7ff5e40a3730 con 0x7ff5e40a2f70 2024-04-26T04:26:27.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.850+0000 7ff5e9327640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:27.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.850+0000 7ff5e9327640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7ff5e40a0f40 con 0x7ff5e40a0240 2024-04-26T04:26:27.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.850+0000 7ff5e9327640 10 monclient(hunting): _renew_subs 2024-04-26T04:26:27.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.850+0000 7ff5e9327640 10 monclient(hunting): authenticate will time out at 1419.561941s 2024-04-26T04:26:27.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.850+0000 7ff5e27fc640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff5e40a0240 0x7ff5e40a2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:27.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.850+0000 7ff5e37fe640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7ff5e40a2f70 0x7ff5e413e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:27.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.850+0000 7ff5e27fc640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff5e40a0240 0x7ff5e40a2a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:51122/0 (socket says 172.21.15.63:51122) 2024-04-26T04:26:27.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.850+0000 7ff5e27fc640 1 -- 172.21.15.63:0/1119462771 learned_addr learned my addr 172.21.15.63:0/1119462771 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:26:27.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.850+0000 7ff5e37fe640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7ff5e40a2f70 0x7ff5e413e110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:51926/0 (socket says 172.21.15.63:51926) 2024-04-26T04:26:27.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.850+0000 7ff5e37fe640 10 monclient(hunting): get_auth_request con 0x7ff5e40a2f70 auth_method 0 2024-04-26T04:26:27.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.850+0000 7ff5e37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:27.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.850+0000 7ff5e37fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:27.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.850+0000 7ff5e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:27.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.850+0000 7ff5e27fc640 10 monclient(hunting): get_auth_request con 0x7ff5e40a0240 auth_method 0 2024-04-26T04:26:27.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.850+0000 7ff5e27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:27.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.850+0000 7ff5e27fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:27.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.850+0000 7ff5e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:27.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.850+0000 7ff5e27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:26:27.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.850+0000 7ff5e27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:26:27.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.850+0000 7ff5e37fe640 1 -- 172.21.15.63:0/1119462771 >> v2:172.21.15.63:3300/0 conn(0x7ff5e40a2f70 msgr2=0x7ff5e413e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T04:26:27.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.850+0000 7ff5e37fe640 1 -- 172.21.15.63:0/1119462771 >> v2:172.21.15.63:3300/0 conn(0x7ff5e40a2f70 msgr2=0x7ff5e413e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:26:27.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.850+0000 7ff5e37fe640 1 --2- 172.21.15.63:0/1119462771 >> v2:172.21.15.63:3300/0 conn(0x7ff5e40a2f70 0x7ff5e413e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:26:27.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.850+0000 7ff5e27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:26:27.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.850+0000 7ff5e37fe640 1 --2- 172.21.15.63:0/1119462771 >> v2:172.21.15.63:3300/0 conn(0x7ff5e40a2f70 0x7ff5e413e110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:26:27.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.850+0000 7ff5e27fc640 10 monclient(hunting): handle_auth_done global_id 24632 payload 274 2024-04-26T04:26:27.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.850+0000 7ff5e27fc640 1 -- 172.21.15.63:0/1119462771 >> v2:172.21.15.63:3300/0 conn(0x7ff5e40a2f70 msgr2=0x7ff5e413e110 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T04:26:27.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.850+0000 7ff5e27fc640 1 --2- 172.21.15.63:0/1119462771 >> v2:172.21.15.63:3300/0 conn(0x7ff5e40a2f70 0x7ff5e413e110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:27.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.850+0000 7ff5e27fc640 1 -- 172.21.15.63:0/1119462771 >> v1:172.21.15.186:6789/0 conn(0x7ff5e409fe60 legacy=0x7ff5e41372f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:26:27.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.850+0000 7ff5e27fc640 10 monclient: _finish_hunting 0 2024-04-26T04:26:27.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.850+0000 7ff5e27fc640 1 monclient: found mon.noname-b 2024-04-26T04:26:27.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.850+0000 7ff5e27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:26:27.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.850+0000 7ff5e27fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.63:3301/0 2024-04-26T04:26:27.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.850+0000 7ff5e27fc640 1 -- 172.21.15.63:0/1119462771 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5e40a1220 con 0x7ff5e40a0240 2024-04-26T04:26:27.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.854+0000 7ff5e27fc640 1 -- 172.21.15.63:0/1119462771 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff5e40a0240 msgr2=0x7ff5e40a2a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:26:27.865 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.854+0000 7ff5e27fc640 1 --2- 172.21.15.63:0/1119462771 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff5e40a0240 0x7ff5e40a2a30 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff5d8009870 tx=0x7ff5d801c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f78856d4d1dbe89d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:26:27.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.854+0000 7ff5e1ffb640 1 -- 172.21.15.63:0/1119462771 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5d80b1070 con 0x7ff5e40a0240 2024-04-26T04:26:27.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.854+0000 7ff5e1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:27.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.854+0000 7ff5e1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T04:26:27.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.854+0000 7ff5e1ffb640 10 monclient: dump: 2024-04-26T04:26:27.866 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:27.866 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:27.866 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:27.866 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:27.866 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:27.866 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:27.866 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:27.866 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:27.866 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:27.867 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:27.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.854+0000 7ff5e1ffb640 10 monclient: _finish_auth 0 2024-04-26T04:26:27.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.854+0000 7ff5e1ffb640 10 monclient: _check_auth_tickets 2024-04-26T04:26:27.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.854+0000 7ff5e1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:26:27.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.854+0000 7ff5e1ffb640 1 -- 172.21.15.63:0/1119462771 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5d80ac620 con 0x7ff5e40a0240 2024-04-26T04:26:27.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.854+0000 7ff5e9327640 5 monclient: authenticate success, global_id 24632 2024-04-26T04:26:27.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.854+0000 7ff5e9327640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T04:26:27.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.854+0000 7ff5e1ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T04:26:27.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.854+0000 7ff5e9327640 10 monclient: get_monmap_and_config success 2024-04-26T04:26:27.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.854+0000 7ff5e1ffb640 1 -- 172.21.15.63:0/1119462771 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5d80ac9f0 con 0x7ff5e40a0240 2024-04-26T04:26:27.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.854+0000 7ff5e9327640 4 set_mon_vals no callback set 2024-04-26T04:26:27.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.854+0000 7ff5e1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:27.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.854+0000 7ff5e1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:26:27.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.854+0000 7ff5e1ffb640 10 monclient: dump: 2024-04-26T04:26:27.867 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:27.868 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:27.868 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:27.868 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:27.868 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:27.868 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:27.868 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:27.868 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:27.868 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:27.868 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:27.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.854+0000 7ff5e9327640 10 monclient: shutdown 2024-04-26T04:26:27.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.854+0000 7ff5e9327640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:26:27.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.854+0000 7ff5e9327640 1 -- 172.21.15.63:0/1119462771 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff5e40a0240 msgr2=0x7ff5e40a2a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:27.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.854+0000 7ff5e9327640 1 --2- 172.21.15.63:0/1119462771 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff5e40a0240 0x7ff5e40a2a30 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff5d8009870 tx=0x7ff5d801c3e0 comp rx=0 tx=0).stop 2024-04-26T04:26:27.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.854+0000 7ff5e9327640 1 -- 172.21.15.63:0/1119462771 shutdown_connections 2024-04-26T04:26:27.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.854+0000 7ff5e9327640 1 --2- 172.21.15.63:0/1119462771 >> v2:172.21.15.63:3300/0 conn(0x7ff5e40a2f70 0x7ff5e413e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:27.869 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.854+0000 7ff5e9327640 1 --2- 172.21.15.63:0/1119462771 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff5e40a0240 0x7ff5e40a2a30 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:27.869 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.854+0000 7ff5e9327640 1 -- 172.21.15.63:0/1119462771 >> 172.21.15.63:0/1119462771 conn(0x7ff5e40a5fb0 msgr2=0x7ff5e40a63c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:26:27.869 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.854+0000 7ff5e9327640 1 -- 172.21.15.63:0/1119462771 shutdown_connections 2024-04-26T04:26:27.869 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.854+0000 7ff5e9327640 1 -- 172.21.15.63:0/1119462771 wait complete. 2024-04-26T04:26:27.869 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.858+0000 7ff5e9327640 10 monclient: build_initial_monmap 2024-04-26T04:26:27.869 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.858+0000 7ff5e9327640 10 monclient: monmap: 2024-04-26T04:26:27.869 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:26:27.869 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:26:27.869 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:26:27.869 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:26:27.869 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:26:27.869 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:27.869 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:26:27.869 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:26:27.870 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:26:27.870 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:27.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.858+0000 7ff5e9327640 1 Processor -- start 2024-04-26T04:26:27.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.858+0000 7ff5e9327640 1 -- start start 2024-04-26T04:26:27.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.858+0000 7ff5e9327640 10 monclient: init 2024-04-26T04:26:27.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.858+0000 7ff5e9327640 10 monclient: _reopen_session rank -1 2024-04-26T04:26:27.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.858+0000 7ff5e9327640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T04:26:27.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.858+0000 7ff5e9327640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff5e409fe60 0x7ff5e4132f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:27.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.858+0000 7ff5e9327640 10 monclient(hunting): picked mon.noname-a con 0x7ff5e409fe60 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:26:27.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.858+0000 7ff5e9327640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5e40a0240 0x7ff5e4133460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:27.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.858+0000 7ff5e9327640 10 monclient(hunting): picked mon.noname-b con 0x7ff5e40a0240 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:26:27.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.858+0000 7ff5e9327640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff5e40a2f70 0x7ff5e4137df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:27.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.858+0000 7ff5e9327640 10 monclient(hunting): picked mon.noname-c con 0x7ff5e40a2f70 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:26:27.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.858+0000 7ff5e9327640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:27.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.858+0000 7ff5e9327640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff5e4141200 con 0x7ff5e409fe60 2024-04-26T04:26:27.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.858+0000 7ff5e9327640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:27.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.858+0000 7ff5e9327640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff5e4141080 con 0x7ff5e40a0240 2024-04-26T04:26:27.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.858+0000 7ff5e9327640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:27.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.858+0000 7ff5e9327640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7ff5e4141380 con 0x7ff5e40a2f70 2024-04-26T04:26:27.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.858+0000 7ff5e9327640 10 monclient(hunting): _renew_subs 2024-04-26T04:26:27.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.858+0000 7ff5e9327640 10 monclient(hunting): authenticate will time out at 1419.568937s 2024-04-26T04:26:27.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.858+0000 7ff5e2ffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff5e409fe60 0x7ff5e4132f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:27.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.858+0000 7ff5e37fe640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff5e40a2f70 0x7ff5e4137df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:27.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.858+0000 7ff5e2ffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff5e409fe60 0x7ff5e4132f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:51938/0 (socket says 172.21.15.63:51938) 2024-04-26T04:26:27.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.858+0000 7ff5e2ffd640 1 -- 172.21.15.63:0/2551908911 learned_addr learned my addr 172.21.15.63:0/2551908911 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:26:27.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.858+0000 7ff5e2ffd640 10 monclient(hunting): get_auth_request con 0x7ff5e409fe60 auth_method 0 2024-04-26T04:26:27.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.858+0000 7ff5e2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:27.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.858+0000 7ff5e2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:27.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.858+0000 7ff5e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:27.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.858+0000 7ff5e37fe640 10 monclient(hunting): get_auth_request con 0x7ff5e40a2f70 auth_method 0 2024-04-26T04:26:27.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.858+0000 7ff5e37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:27.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.858+0000 7ff5e37fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:27.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.858+0000 7ff5e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:27.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.858+0000 7ff5e2ffd640 1 -- 172.21.15.63:0/2551908911 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff5e409fe60 msgr2=0x7ff5e4132f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T04:26:27.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.858+0000 7ff5e37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:26:27.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.858+0000 7ff5e37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:26:27.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.858+0000 7ff5e2ffd640 1 -- 172.21.15.63:0/2551908911 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff5e409fe60 msgr2=0x7ff5e4132f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T04:26:27.873 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.858+0000 7ff5e2ffd640 1 --2- 172.21.15.63:0/2551908911 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff5e409fe60 0x7ff5e4132f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:26:27.873 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.858+0000 7ff5e37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:26:27.873 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.858+0000 7ff5e2ffd640 1 --2- 172.21.15.63:0/2551908911 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff5e409fe60 0x7ff5e4132f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:26:27.874 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.858+0000 7ff5e27fc640 1 --2- 172.21.15.63:0/2551908911 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5e40a0240 0x7ff5e4133460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:27.874 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.858+0000 7ff5e37fe640 10 monclient(hunting): handle_auth_done global_id 24635 payload 824 2024-04-26T04:26:27.874 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.858+0000 7ff5e37fe640 1 -- 172.21.15.63:0/2551908911 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5e40a0240 msgr2=0x7ff5e4133460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:27.874 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.858+0000 7ff5e37fe640 1 --2- 172.21.15.63:0/2551908911 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5e40a0240 0x7ff5e4133460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:27.874 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.858+0000 7ff5e37fe640 1 -- 172.21.15.63:0/2551908911 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff5e409fe60 msgr2=0x7ff5e4132f20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T04:26:27.875 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.858+0000 7ff5e37fe640 1 --2- 172.21.15.63:0/2551908911 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff5e409fe60 0x7ff5e4132f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:27.875 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.858+0000 7ff5e37fe640 10 monclient: _finish_hunting 0 2024-04-26T04:26:27.875 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.858+0000 7ff5e37fe640 1 monclient: found mon.noname-c 2024-04-26T04:26:27.875 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.858+0000 7ff5e37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:26:27.875 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.858+0000 7ff5e37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.63:3301/0 2024-04-26T04:26:27.875 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.858+0000 7ff5e37fe640 1 -- 172.21.15.63:0/2551908911 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5e413b990 con 0x7ff5e40a2f70 2024-04-26T04:26:27.875 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.858+0000 7ff5e27fc640 10 monclient: get_auth_request con 0x7ff5e40a0240 auth_method 0 2024-04-26T04:26:27.875 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.858+0000 7ff5e27fc640 1 --2- 172.21.15.63:0/2551908911 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5e40a0240 0x7ff5e4133460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T04:26:27.875 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.858+0000 7ff5e37fe640 1 -- 172.21.15.63:0/2551908911 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff5e40a2f70 msgr2=0x7ff5e4137df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:26:27.875 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.858+0000 7ff5e37fe640 1 --2- 172.21.15.63:0/2551908911 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff5e40a2f70 0x7ff5e4137df0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc007cf0 tx=0x7ff5dc00e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:26:27.875 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.858+0000 7ff5cb7fe640 1 -- 172.21.15.63:0/2551908911 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5dc0039a0 con 0x7ff5e40a2f70 2024-04-26T04:26:27.875 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.858+0000 7ff5cb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:27.875 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.858+0000 7ff5cb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:26:27.875 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.858+0000 7ff5cb7fe640 10 monclient: dump: 2024-04-26T04:26:27.876 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:27.876 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:27.876 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:27.876 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:27.876 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:27.876 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:27.876 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:27.876 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:27.876 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:27.876 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:27.876 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.858+0000 7ff5cb7fe640 10 monclient: _finish_auth 0 2024-04-26T04:26:27.877 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.858+0000 7ff5cb7fe640 10 monclient: _check_auth_tickets 2024-04-26T04:26:27.877 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.858+0000 7ff5cb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:26:27.877 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.858+0000 7ff5cb7fe640 1 -- 172.21.15.63:0/2551908911 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5dc003b60 con 0x7ff5e40a2f70 2024-04-26T04:26:27.877 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.858+0000 7ff5cb7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T04:26:27.877 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.858+0000 7ff5cb7fe640 1 -- 172.21.15.63:0/2551908911 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5dc004020 con 0x7ff5e40a2f70 2024-04-26T04:26:27.877 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.858+0000 7ff5cb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:27.877 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.858+0000 7ff5cb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:26:27.878 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.858+0000 7ff5cb7fe640 10 monclient: dump: 2024-04-26T04:26:27.878 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:27.878 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:27.878 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:27.878 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:27.878 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:27.878 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:27.878 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:27.878 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:27.879 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:27.879 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:27.879 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.858+0000 7ff5e17fa640 4 set_mon_vals no callback set 2024-04-26T04:26:27.879 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.862+0000 7ff5e9327640 5 monclient: authenticate success, global_id 24635 2024-04-26T04:26:27.879 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.862+0000 7ff5e9327640 10 monclient: _renew_subs 2024-04-26T04:26:27.879 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.862+0000 7ff5e9327640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:26:27.879 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.862+0000 7ff5e9327640 1 -- 172.21.15.63:0/2551908911 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5e40a38f0 con 0x7ff5e40a2f70 2024-04-26T04:26:27.879 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.862+0000 7ff5e9327640 10 monclient: _renew_subs 2024-04-26T04:26:27.879 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.862+0000 7ff5e9327640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:26:27.879 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.862+0000 7ff5e9327640 1 -- 172.21.15.63:0/2551908911 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5e41339a0 con 0x7ff5e40a2f70 2024-04-26T04:26:27.881 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.862+0000 7ff5e9327640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:26:27.881 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.862+0000 7ff5e9327640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:26:27.881 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.862+0000 7ff5e9327640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:26:27.881 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.862+0000 7ff5e9327640 1 -- 172.21.15.63:0/2551908911 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5e4062e10 con 0x7ff5e40a2f70 2024-04-26T04:26:27.881 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.862+0000 7ff5cb7fe640 1 -- 172.21.15.63:0/2551908911 <== mon.2 v2:172.21.15.63:3301/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7ff5dc01b020 con 0x7ff5e40a2f70 2024-04-26T04:26:27.881 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.862+0000 7ff5cb7fe640 1 --2- 172.21.15.63:0/2551908911 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7ff5d0035ec0 0x7ff5d0038380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:27.881 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.862+0000 7ff5cb7fe640 1 -- 172.21.15.63:0/2551908911 <== mon.2 v2:172.21.15.63:3301/0 5 ==== osd_map(314..314 src has 1..314) ==== 4738+0+0 (secure 0 0 0) 0x7ff5dc049040 con 0x7ff5e40a2f70 2024-04-26T04:26:27.881 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.862+0000 7ff5e2ffd640 1 --2- 172.21.15.63:0/2551908911 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7ff5d0035ec0 0x7ff5d0038380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:27.881 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.862+0000 7ff5e2ffd640 10 monclient: get_auth_request con 0x7ff5d0035ec0 auth_method 0 2024-04-26T04:26:27.881 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.866+0000 7ff5e2ffd640 1 --2- 172.21.15.63:0/2551908911 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7ff5d0035ec0 0x7ff5d0038380 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7ff5d4007d20 tx=0x7ff5d4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:26:27.881 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.866+0000 7ff5cb7fe640 1 -- 172.21.15.63:0/2551908911 <== mon.2 v2:172.21.15.63:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7ff5dc0147a0 con 0x7ff5e40a2f70 2024-04-26T04:26:27.881 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.866+0000 7ff5cb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:26:27.881 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:27.866+0000 7ff5cb7fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:26:28.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:28.086+0000 7ff5e9327640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T04:26:28.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:28.086+0000 7ff5e9327640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T04:26:28.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:28.086+0000 7ff5e9327640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:26:28.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:28.086+0000 7ff5e9327640 1 -- 172.21.15.63:0/2551908911 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff5e4068430 con 0x7ff5e40a2f70 2024-04-26T04:26:28.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:28.086+0000 7ff5cb7fe640 1 -- 172.21.15.63:0/2551908911 <== mon.2 v2:172.21.15.63:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7ff5dc012b50 con 0x7ff5e40a2f70 2024-04-26T04:26:28.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:28.086+0000 7ff5cb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T04:26:28.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:28.086+0000 7ff5cb7fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:26:28.093 INFO:teuthology.orchestra.run.smithi063.stdout:{"election_epoch":28,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":92,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cd1c6a0c-4e4b-4014-89c0-2e535479ef39","modified":"2024-04-26T04:18:03.702339Z","created":"2024-04-26T04:18:03.702339Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3300","nonce":0},{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:3300","nonce":0},{"type":"v1","addr":"172.21.15.186:6789","nonce":0}]},"addr":"172.21.15.186:6789/0","public_addr":"172.21.15.186:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3301","nonce":0},{"type":"v1","addr":"172.21.15.63:6790","nonce":0}]},"addr":"172.21.15.63:6790/0","public_addr":"172.21.15.63:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T04:26:28.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:28.090+0000 7ff5e9327640 1 -- 172.21.15.63:0/2551908911 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7ff5d0035ec0 msgr2=0x7ff5d0038380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:28.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:28.090+0000 7ff5e9327640 1 --2- 172.21.15.63:0/2551908911 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7ff5d0035ec0 0x7ff5d0038380 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7ff5d4007d20 tx=0x7ff5d4009040 comp rx=0 tx=0).stop 2024-04-26T04:26:28.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:28.090+0000 7ff5e9327640 10 monclient: shutdown 2024-04-26T04:26:28.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:28.090+0000 7ff5e9327640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:26:28.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:28.090+0000 7ff5e9327640 1 -- 172.21.15.63:0/2551908911 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff5e40a2f70 msgr2=0x7ff5e4137df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:28.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:28.090+0000 7ff5e9327640 1 --2- 172.21.15.63:0/2551908911 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff5e40a2f70 0x7ff5e4137df0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc007cf0 tx=0x7ff5dc00e4b0 comp rx=0 tx=0).stop 2024-04-26T04:26:28.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:28.090+0000 7ff5e9327640 1 -- 172.21.15.63:0/2551908911 shutdown_connections 2024-04-26T04:26:28.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:28.090+0000 7ff5e9327640 1 --2- 172.21.15.63:0/2551908911 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7ff5d0035ec0 0x7ff5d0038380 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:28.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:28.090+0000 7ff5e9327640 1 --2- 172.21.15.63:0/2551908911 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff5e40a2f70 0x7ff5e4137df0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:28.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:28.090+0000 7ff5e9327640 1 --2- 172.21.15.63:0/2551908911 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5e40a0240 0x7ff5e4133460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:28.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:28.090+0000 7ff5e9327640 1 --2- 172.21.15.63:0/2551908911 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff5e409fe60 0x7ff5e4132f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:28.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:28.090+0000 7ff5e9327640 1 -- 172.21.15.63:0/2551908911 >> 172.21.15.63:0/2551908911 conn(0x7ff5e40a5fb0 msgr2=0x7ff5e40a8880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:26:28.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:28.090+0000 7ff5e9327640 1 -- 172.21.15.63:0/2551908911 shutdown_connections 2024-04-26T04:26:28.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:28.090+0000 7ff5e9327640 1 -- 172.21.15.63:0/2551908911 wait complete. 2024-04-26T04:26:28.699 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:26:28.699 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31947]: starting. 2024-04-26T04:26:28.699 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31947]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-31910 2024-04-26T04:26:28.700 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31947]: created object 0... 2024-04-26T04:26:28.700 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31947]: created object 25... 2024-04-26T04:26:28.700 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31947]: created object 49... 2024-04-26T04:26:28.700 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31947]: finishing. 2024-04-26T04:26:28.700 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[31947]: shutting down. 2024-04-26T04:26:28.756 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:26:28.756 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[31949]: starting. 2024-04-26T04:26:28.756 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[31949]: listing objects. 2024-04-26T04:26:28.756 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[31949]: listed object 0... 2024-04-26T04:26:28.756 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[31949]: listed object 25... 2024-04-26T04:26:28.756 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[31949]: saw 50 objects 2024-04-26T04:26:28.756 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[31949]: shutting down. 2024-04-26T04:26:29.493 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:26:29.493 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[31948]: starting. 2024-04-26T04:26:29.493 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[31948]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-31910 2024-04-26T04:26:29.493 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[31948]: shutting down. 2024-04-26T04:26:29.496 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:26:29.496 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:26:29.496 INFO:teuthology.orchestra.run.smithi186.stdout:******* SUCCESS ********** 2024-04-26T04:26:29.499 DEBUG:teuthology.orchestra.run.smithi186:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_rados_delete_pools_parallel 2024-04-26T04:26:31.111 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T04:26:31.244 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.238+0000 7fcde0962640 10 monclient: get_monmap_and_config 2024-04-26T04:26:31.244 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.238+0000 7fcde0962640 10 monclient: build_initial_monmap 2024-04-26T04:26:31.244 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.238+0000 7fcde0962640 10 monclient: monmap: 2024-04-26T04:26:31.245 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:26:31.245 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:31.245 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:26:31.244587+0000 2024-04-26T04:26:31.245 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:26:31.244587+0000 2024-04-26T04:26:31.245 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:26:31.245 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:31.245 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:26:31.245 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:26:31.245 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:26:31.245 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:26:31.245 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:26:31.246 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:31.247 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.238+0000 7fcde0962640 1 Processor -- start 2024-04-26T04:26:31.247 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.238+0000 7fcde0962640 1 -- start start 2024-04-26T04:26:31.247 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.238+0000 7fcde0962640 10 monclient: init 2024-04-26T04:26:31.247 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.242+0000 7fcde0962640 10 monclient: _reopen_session rank -1 2024-04-26T04:26:31.247 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.242+0000 7fcde0962640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-04-26T04:26:31.248 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.242+0000 7fcde0962640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcddc0a2cd0 addr v1:172.21.15.63:6789/0 2024-04-26T04:26:31.248 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.242+0000 7fcde0962640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7fcddc0a30b0 0x7fcddc13c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:31.248 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.242+0000 7fcde0962640 10 monclient(hunting): picked mon.noname-a con 0x7fcddc0a30b0 addr v2:172.21.15.63:3300/0 2024-04-26T04:26:31.248 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.242+0000 7fcde0962640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcddc13c540 addr v1:172.21.15.186:6789/0 2024-04-26T04:26:31.248 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.242+0000 7fcde0962640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcddc0a4ee0 con 0x7fcddc0a2cd0 2024-04-26T04:26:31.248 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.242+0000 7fcde0962640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcddc0a0fb0 con 0x7fcddc13c540 2024-04-26T04:26:31.248 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.242+0000 7fcde0962640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:31.248 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.242+0000 7fcde0962640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7fcddc0a5380 con 0x7fcddc0a30b0 2024-04-26T04:26:31.248 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.242+0000 7fcde0962640 10 monclient(hunting): _renew_subs 2024-04-26T04:26:31.248 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.242+0000 7fcde0962640 10 monclient(hunting): authenticate will time out at 1422.952437s 2024-04-26T04:26:31.248 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.242+0000 7fcdd9d9b640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7fcddc0a30b0 0x7fcddc13c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:31.248 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.242+0000 7fcdda59c640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7fcddc0a2cd0 0x7fcddc0a47d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:54006/0 (socket says 172.21.15.63:54006) 2024-04-26T04:26:31.248 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.242+0000 7fcdda59c640 1 -- 172.21.15.63:0/352213451 learned_addr learned my addr 172.21.15.63:0/352213451 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-04-26T04:26:31.248 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.242+0000 7fcdd9d9b640 10 monclient(hunting): get_auth_request con 0x7fcddc0a30b0 auth_method 0 2024-04-26T04:26:31.248 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.242+0000 7fcdd9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:31.249 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.242+0000 7fcdd9d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:31.249 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.242+0000 7fcdd9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:31.249 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.242+0000 7fcdd9d9b640 1 -- 172.21.15.63:0/352213451 >> v2:172.21.15.63:3300/0 conn(0x7fcddc0a30b0 msgr2=0x7fcddc13c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T04:26:31.249 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.242+0000 7fcdd9d9b640 1 -- 172.21.15.63:0/352213451 >> v2:172.21.15.63:3300/0 conn(0x7fcddc0a30b0 msgr2=0x7fcddc13c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:26:31.249 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.242+0000 7fcdd9d9b640 1 --2- 172.21.15.63:0/352213451 >> v2:172.21.15.63:3300/0 conn(0x7fcddc0a30b0 0x7fcddc13c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:26:31.249 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.242+0000 7fcdd9d9b640 1 --2- 172.21.15.63:0/352213451 >> v2:172.21.15.63:3300/0 conn(0x7fcddc0a30b0 0x7fcddc13c000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:26:31.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.242+0000 7fcdda59c640 1 -- 172.21.15.63:0/352213451 >> v1:172.21.15.63:6789/0 conn(0x7fcddc0a2cd0 legacy=0x7fcddc0a47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:26:31.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.242+0000 7fcddad9d640 1 -- 172.21.15.63:0/352213451 >> v1:172.21.15.186:6789/0 conn(0x7fcddc13c540 legacy=0x7fcddc13e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:26:31.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.242+0000 7fcdd959a640 1 -- 172.21.15.63:0/352213451 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2544597503 0 0) 0x7fcddc0a0fb0 con 0x7fcddc13c540 2024-04-26T04:26:31.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.242+0000 7fcdd959a640 10 monclient(hunting): _negotiate 2024-04-26T04:26:31.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.242+0000 7fcdd959a640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:31.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.242+0000 7fcdd959a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:31.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.242+0000 7fcdd959a640 10 monclient(hunting): my global_id is 34519 2024-04-26T04:26:31.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.242+0000 7fcdd959a640 1 -- 172.21.15.63:0/352213451 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcdbc003530 con 0x7fcddc13c540 2024-04-26T04:26:31.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcdd959a640 1 -- 172.21.15.63:0/352213451 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3067388931 0 0) 0x7fcdbc003530 con 0x7fcddc13c540 2024-04-26T04:26:31.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcdd959a640 1 -- 172.21.15.63:0/352213451 >> v2:172.21.15.63:3300/0 conn(0x7fcddc0a30b0 msgr2=0x7fcddc13c000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T04:26:31.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcdd959a640 1 --2- 172.21.15.63:0/352213451 >> v2:172.21.15.63:3300/0 conn(0x7fcddc0a30b0 0x7fcddc13c000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:31.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcdd959a640 1 -- 172.21.15.63:0/352213451 >> v1:172.21.15.63:6789/0 conn(0x7fcddc0a2cd0 legacy=0x7fcddc0a47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:31.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcdd959a640 10 monclient: _finish_hunting 0 2024-04-26T04:26:31.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcdd959a640 1 monclient: found mon.noname-c-legacy 2024-04-26T04:26:31.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcdd959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:26:31.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcdd959a640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.186:6789/0 2024-04-26T04:26:31.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcdd959a640 1 -- 172.21.15.63:0/352213451 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcddc0a1290 con 0x7fcddc13c540 2024-04-26T04:26:31.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcdd959a640 10 monclient: _finish_auth 0 2024-04-26T04:26:31.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcdd959a640 10 monclient: _check_auth_tickets 2024-04-26T04:26:31.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcdd959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:26:31.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcdd959a640 1 -- 172.21.15.63:0/352213451 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2612289768 0 0) 0x7fcdc4001980 con 0x7fcddc13c540 2024-04-26T04:26:31.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcdd959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:31.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcdd959a640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-04-26T04:26:31.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcdd959a640 10 monclient: dump: 2024-04-26T04:26:31.254 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:31.254 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:31.254 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:31.254 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:31.254 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:31.254 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:31.254 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:31.254 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:31.254 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:31.255 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:31.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcdd959a640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] but i'm connected to v1:172.21.15.186:6789/0, reconnecting 2024-04-26T04:26:31.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcdd959a640 10 monclient: _reopen_session rank -1 2024-04-26T04:26:31.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcdd959a640 1 -- 172.21.15.63:0/352213451 >> v1:172.21.15.186:6789/0 conn(0x7fcddc13c540 legacy=0x7fcddc13e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:31.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcdd959a640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T04:26:31.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcdd959a640 1 --2- 172.21.15.63:0/352213451 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcdbc003af0 0x7fcdbc003f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:31.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcdd959a640 10 monclient(hunting): picked mon.a con 0x7fcdbc003af0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:26:31.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcdd959a640 1 --2- 172.21.15.63:0/352213451 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcdbc004810 0x7fcdbc004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:31.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcdd959a640 10 monclient(hunting): picked mon.b con 0x7fcdbc004810 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:26:31.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcdd959a640 1 --2- 172.21.15.63:0/352213451 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcdbc0053d0 0x7fcdbc00ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:31.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcdd959a640 10 monclient(hunting): picked mon.c con 0x7fcdbc0053d0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:26:31.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcdd959a640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:31.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcdd959a640 1 -- 172.21.15.63:0/352213451 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fcddc0a5380 con 0x7fcdbc003af0 2024-04-26T04:26:31.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcdd959a640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:31.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcdd959a640 1 -- 172.21.15.63:0/352213451 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcdbc00c2f0 con 0x7fcdbc004810 2024-04-26T04:26:31.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcdd959a640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:31.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcdd959a640 1 -- 172.21.15.63:0/352213451 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7fcdbc00c4f0 con 0x7fcdbc0053d0 2024-04-26T04:26:31.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcdd959a640 10 monclient(hunting): _renew_subs 2024-04-26T04:26:31.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcdd959a640 10 monclient(hunting): _finish_auth 0 2024-04-26T04:26:31.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcdda59c640 1 --2- 172.21.15.63:0/352213451 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcdbc003af0 0x7fcdbc003f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:31.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcdda59c640 10 monclient(hunting): get_auth_request con 0x7fcdbc003af0 auth_method 0 2024-04-26T04:26:31.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcdda59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:31.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcdda59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:31.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcdda59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:26:31.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcddad9d640 1 --2- 172.21.15.63:0/352213451 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcdbc0053d0 0x7fcdbc00ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:31.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcdda59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:26:31.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcdda59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:26:31.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcdda59c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T04:26:31.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcddad9d640 10 monclient(hunting): get_auth_request con 0x7fcdbc0053d0 auth_method 0 2024-04-26T04:26:31.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcddad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:31.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcddad9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:31.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcddad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:26:31.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcddad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:26:31.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcddad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:26:31.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcdd9d9b640 1 --2- 172.21.15.63:0/352213451 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcdbc004810 0x7fcdbc004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:31.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcddad9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T04:26:31.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcdd9d9b640 10 monclient(hunting): get_auth_request con 0x7fcdbc004810 auth_method 0 2024-04-26T04:26:31.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcdd9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:31.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcdd9d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:31.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcdd9d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:26:31.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcdda59c640 10 monclient(hunting): handle_auth_done global_id 34519 payload 293 2024-04-26T04:26:31.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcdda59c640 1 -- 172.21.15.63:0/352213451 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcdbc0053d0 msgr2=0x7fcdbc00ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:26:31.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcdda59c640 1 --2- 172.21.15.63:0/352213451 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcdbc0053d0 0x7fcdbc00ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:31.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcdda59c640 1 -- 172.21.15.63:0/352213451 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcdbc004810 msgr2=0x7fcdbc004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:26:31.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcdda59c640 1 --2- 172.21.15.63:0/352213451 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcdbc004810 0x7fcdbc004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:31.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcdda59c640 10 monclient: _finish_hunting 0 2024-04-26T04:26:31.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcdda59c640 1 monclient: found mon.a 2024-04-26T04:26:31.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcdda59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:26:31.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcdda59c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:26:31.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcdda59c640 1 -- 172.21.15.63:0/352213451 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdbc003530 con 0x7fcdbc003af0 2024-04-26T04:26:31.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcdda59c640 10 monclient: _finish_auth 0 2024-04-26T04:26:31.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcdda59c640 10 monclient: _check_auth_tickets 2024-04-26T04:26:31.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcdda59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:26:31.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcddad9d640 1 -- 172.21.15.63:0/352213451 reap_dead start 2024-04-26T04:26:31.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcde0962640 5 monclient: authenticate success, global_id 34519 2024-04-26T04:26:31.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.246+0000 7fcde0962640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T04:26:31.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.250+0000 7fcdda59c640 1 -- 172.21.15.63:0/352213451 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcdbc003af0 msgr2=0x7fcdbc003f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:26:31.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.250+0000 7fcdda59c640 1 --2- 172.21.15.63:0/352213451 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcdbc003af0 0x7fcdbc003f10 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc001a30 tx=0x7fcdcc01c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b78b9063793d70b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:26:31.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.250+0000 7fcdd959a640 1 -- 172.21.15.63:0/352213451 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdcc0a8070 con 0x7fcdbc003af0 2024-04-26T04:26:31.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.250+0000 7fcdd959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:31.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.250+0000 7fcdd959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:26:31.265 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.250+0000 7fcdd959a640 10 monclient: dump: 2024-04-26T04:26:31.265 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:31.265 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:31.265 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:31.265 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:31.265 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:31.265 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:31.265 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:31.266 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:31.266 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:31.266 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:31.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.250+0000 7fcde0962640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T04:26:31.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.250+0000 7fcdd959a640 1 -- 172.21.15.63:0/352213451 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcdcc004ab0 con 0x7fcdbc003af0 2024-04-26T04:26:31.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.250+0000 7fcdd959a640 10 monclient: handle_config config(0 keys) 2024-04-26T04:26:31.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.250+0000 7fcde0962640 10 monclient: get_monmap_and_config success 2024-04-26T04:26:31.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.250+0000 7fcdd959a640 1 -- 172.21.15.63:0/352213451 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdcc001730 con 0x7fcdbc003af0 2024-04-26T04:26:31.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.250+0000 7fcde0962640 4 set_mon_vals no callback set 2024-04-26T04:26:31.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.250+0000 7fcdd959a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:31.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.250+0000 7fcdd959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:26:31.266 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.250+0000 7fcdd959a640 10 monclient: dump: 2024-04-26T04:26:31.266 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:31.267 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:31.267 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:31.267 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:31.267 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:31.267 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:31.267 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:31.267 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:31.267 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:31.267 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:31.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.250+0000 7fcde0962640 10 monclient: shutdown 2024-04-26T04:26:31.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.250+0000 7fcde0962640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:26:31.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.250+0000 7fcde0962640 1 -- 172.21.15.63:0/352213451 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcdbc003af0 msgr2=0x7fcdbc003f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:31.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.250+0000 7fcde0962640 1 --2- 172.21.15.63:0/352213451 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcdbc003af0 0x7fcdbc003f10 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc001a30 tx=0x7fcdcc01c650 comp rx=0 tx=0).stop 2024-04-26T04:26:31.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.250+0000 7fcde0962640 1 -- 172.21.15.63:0/352213451 shutdown_connections 2024-04-26T04:26:31.268 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.250+0000 7fcde0962640 1 --2- 172.21.15.63:0/352213451 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcdbc003af0 0x7fcdbc003f10 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:31.268 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.250+0000 7fcde0962640 1 -- 172.21.15.63:0/352213451 >> 172.21.15.63:0/352213451 conn(0x7fcddc0a5f40 msgr2=0x7fcddc0a6320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:26:31.268 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.250+0000 7fcde0962640 1 -- 172.21.15.63:0/352213451 shutdown_connections 2024-04-26T04:26:31.268 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.250+0000 7fcde0962640 1 -- 172.21.15.63:0/352213451 wait complete. 2024-04-26T04:26:31.268 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.250+0000 7fcde0962640 10 monclient: build_initial_monmap 2024-04-26T04:26:31.268 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.250+0000 7fcde0962640 10 monclient: monmap: 2024-04-26T04:26:31.268 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:26:31.268 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:26:31.268 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:26:31.268 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:26:31.268 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:26:31.268 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:31.268 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:26:31.268 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:26:31.269 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:26:31.269 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:31.269 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.250+0000 7fcde0962640 1 Processor -- start 2024-04-26T04:26:31.269 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.250+0000 7fcde0962640 1 -- start start 2024-04-26T04:26:31.269 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.250+0000 7fcde0962640 10 monclient: init 2024-04-26T04:26:31.269 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.250+0000 7fcde0962640 10 monclient: _reopen_session rank -1 2024-04-26T04:26:31.269 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.250+0000 7fcde0962640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T04:26:31.269 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.250+0000 7fcde0962640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcddc1360b0 0x7fcddc136490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:31.269 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.250+0000 7fcde0962640 10 monclient(hunting): picked mon.noname-b con 0x7fcddc1360b0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:26:31.269 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.250+0000 7fcde0962640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcddc1cf180 0x7fcddc1cf5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:31.269 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.250+0000 7fcde0962640 10 monclient(hunting): picked mon.noname-a con 0x7fcddc1cf180 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:26:31.269 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.250+0000 7fcde0962640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcddc1cfb20 0x7fcddc1d6510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:31.270 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.250+0000 7fcde0962640 10 monclient(hunting): picked mon.noname-c con 0x7fcddc1cfb20 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:26:31.270 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.250+0000 7fcde0962640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:31.270 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.250+0000 7fcde0962640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fcddc137670 con 0x7fcddc1cf180 2024-04-26T04:26:31.270 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.250+0000 7fcde0962640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:31.270 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.250+0000 7fcde0962640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcddc1374f0 con 0x7fcddc1360b0 2024-04-26T04:26:31.270 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.250+0000 7fcde0962640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:31.270 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.250+0000 7fcde0962640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7fcddc1377f0 con 0x7fcddc1cfb20 2024-04-26T04:26:31.270 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.250+0000 7fcde0962640 10 monclient(hunting): _renew_subs 2024-04-26T04:26:31.270 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.250+0000 7fcde0962640 10 monclient(hunting): authenticate will time out at 1422.963708s 2024-04-26T04:26:31.270 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.250+0000 7fcdd9d9b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcddc1cf180 0x7fcddc1cf5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:31.270 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.250+0000 7fcddad9d640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcddc1cfb20 0x7fcddc1d6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:31.270 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.250+0000 7fcdd9d9b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcddc1cf180 0x7fcddc1cf5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:33486/0 (socket says 172.21.15.63:33486) 2024-04-26T04:26:31.270 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.250+0000 7fcddad9d640 10 monclient(hunting): get_auth_request con 0x7fcddc1cfb20 auth_method 0 2024-04-26T04:26:31.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.250+0000 7fcdd9d9b640 1 -- 172.21.15.63:0/3428532539 learned_addr learned my addr 172.21.15.63:0/3428532539 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:26:31.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.250+0000 7fcddad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:31.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.250+0000 7fcddad9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:31.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.250+0000 7fcddad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:31.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.250+0000 7fcdd9d9b640 10 monclient(hunting): get_auth_request con 0x7fcddc1cf180 auth_method 0 2024-04-26T04:26:31.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.250+0000 7fcdd9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:31.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.250+0000 7fcdd9d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:31.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.250+0000 7fcdd9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:31.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.250+0000 7fcdd9d9b640 1 -- 172.21.15.63:0/3428532539 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcddc1cf180 msgr2=0x7fcddc1cf5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T04:26:31.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.250+0000 7fcdd9d9b640 1 -- 172.21.15.63:0/3428532539 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcddc1cf180 msgr2=0x7fcddc1cf5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T04:26:31.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.250+0000 7fcddad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:26:31.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.250+0000 7fcddad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:26:31.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.250+0000 7fcdd9d9b640 1 --2- 172.21.15.63:0/3428532539 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcddc1cf180 0x7fcddc1cf5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:26:31.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.254+0000 7fcddad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:26:31.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.254+0000 7fcdd9d9b640 1 --2- 172.21.15.63:0/3428532539 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcddc1cf180 0x7fcddc1cf5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:26:31.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.254+0000 7fcdda59c640 1 --2- 172.21.15.63:0/3428532539 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcddc1360b0 0x7fcddc136490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:31.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.254+0000 7fcddad9d640 10 monclient(hunting): handle_auth_done global_id 24644 payload 824 2024-04-26T04:26:31.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.254+0000 7fcddad9d640 1 -- 172.21.15.63:0/3428532539 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcddc1360b0 msgr2=0x7fcddc136490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:31.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.254+0000 7fcddad9d640 1 --2- 172.21.15.63:0/3428532539 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcddc1360b0 0x7fcddc136490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:31.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.254+0000 7fcddad9d640 1 -- 172.21.15.63:0/3428532539 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcddc1cf180 msgr2=0x7fcddc1cf5e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T04:26:31.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.254+0000 7fcddad9d640 1 --2- 172.21.15.63:0/3428532539 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcddc1cf180 0x7fcddc1cf5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:31.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.254+0000 7fcddad9d640 10 monclient: _finish_hunting 0 2024-04-26T04:26:31.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.254+0000 7fcddad9d640 1 monclient: found mon.noname-c 2024-04-26T04:26:31.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.254+0000 7fcddad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:26:31.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.254+0000 7fcddad9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.63:3301/0 2024-04-26T04:26:31.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.254+0000 7fcddad9d640 1 -- 172.21.15.63:0/3428532539 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcddc121780 con 0x7fcddc1cfb20 2024-04-26T04:26:31.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.254+0000 7fcdda59c640 10 monclient: get_auth_request con 0x7fcddc1360b0 auth_method 0 2024-04-26T04:26:31.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.254+0000 7fcdda59c640 1 --2- 172.21.15.63:0/3428532539 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcddc1360b0 0x7fcddc136490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T04:26:31.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.254+0000 7fcddad9d640 1 -- 172.21.15.63:0/3428532539 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcddc1cfb20 msgr2=0x7fcddc1d6510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:26:31.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.254+0000 7fcddad9d640 1 --2- 172.21.15.63:0/3428532539 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcddc1cfb20 0x7fcddc1d6510 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fcdc4001ec0 tx=0x7fcdc400d520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:26:31.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.254+0000 7fcdd2ffd640 1 -- 172.21.15.63:0/3428532539 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdc400db80 con 0x7fcddc1cfb20 2024-04-26T04:26:31.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.254+0000 7fcdd2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:31.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.254+0000 7fcdd2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:26:31.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.254+0000 7fcdd2ffd640 10 monclient: dump: 2024-04-26T04:26:31.273 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:31.273 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:31.273 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:31.273 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:31.273 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:31.273 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:31.273 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:31.274 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:31.274 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:31.274 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:31.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.254+0000 7fcdd2ffd640 10 monclient: _finish_auth 0 2024-04-26T04:26:31.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.254+0000 7fcdd2ffd640 10 monclient: _check_auth_tickets 2024-04-26T04:26:31.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.254+0000 7fcdd2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:26:31.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.254+0000 7fcdd2ffd640 1 -- 172.21.15.63:0/3428532539 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcdc4018070 con 0x7fcddc1cfb20 2024-04-26T04:26:31.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.254+0000 7fcde0962640 5 monclient: authenticate success, global_id 24644 2024-04-26T04:26:31.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.254+0000 7fcde0962640 10 monclient: _renew_subs 2024-04-26T04:26:31.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.254+0000 7fcde0962640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:26:31.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.254+0000 7fcde0962640 1 -- 172.21.15.63:0/3428532539 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcddc1cffa0 con 0x7fcddc1cfb20 2024-04-26T04:26:31.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.254+0000 7fcdd2ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T04:26:31.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.254+0000 7fcdd2ffd640 1 -- 172.21.15.63:0/3428532539 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdc40070d0 con 0x7fcddc1cfb20 2024-04-26T04:26:31.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.254+0000 7fcdd2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:31.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.254+0000 7fcdd2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:26:31.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.254+0000 7fcdd2ffd640 10 monclient: dump: 2024-04-26T04:26:31.275 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:31.275 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:31.275 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:31.275 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:31.275 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:31.275 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:31.275 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:31.275 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:31.275 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:31.275 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:31.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.254+0000 7fcdd8d99640 4 set_mon_vals no callback set 2024-04-26T04:26:31.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.254+0000 7fcde0962640 10 monclient: _renew_subs 2024-04-26T04:26:31.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.254+0000 7fcde0962640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:26:31.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.254+0000 7fcde0962640 1 -- 172.21.15.63:0/3428532539 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcddc1c2640 con 0x7fcddc1cfb20 2024-04-26T04:26:31.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.254+0000 7fcde0962640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:26:31.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.254+0000 7fcde0962640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:26:31.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.254+0000 7fcde0962640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:26:31.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.254+0000 7fcdd2ffd640 1 -- 172.21.15.63:0/3428532539 <== mon.2 v2:172.21.15.63:3301/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fcdc4007270 con 0x7fcddc1cfb20 2024-04-26T04:26:31.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.254+0000 7fcde0962640 1 -- 172.21.15.63:0/3428532539 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcddc062e10 con 0x7fcddc1cfb20 2024-04-26T04:26:31.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.254+0000 7fcdd2ffd640 1 --2- 172.21.15.63:0/3428532539 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fcdc0035f40 0x7fcdc0038400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:31.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.254+0000 7fcdd2ffd640 1 -- 172.21.15.63:0/3428532539 <== mon.2 v2:172.21.15.63:3301/0 5 ==== osd_map(317..317 src has 1..317) ==== 4738+0+0 (secure 0 0 0) 0x7fcdc4048c40 con 0x7fcddc1cfb20 2024-04-26T04:26:31.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.258+0000 7fcdda59c640 1 --2- 172.21.15.63:0/3428532539 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fcdc0035f40 0x7fcdc0038400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:31.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.258+0000 7fcdda59c640 10 monclient: get_auth_request con 0x7fcdc0035f40 auth_method 0 2024-04-26T04:26:31.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.258+0000 7fcdda59c640 1 --2- 172.21.15.63:0/3428532539 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fcdc0035f40 0x7fcdc0038400 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc001a00 tx=0x7fcdcc0014e0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:26:31.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.262+0000 7fcdd2ffd640 1 -- 172.21.15.63:0/3428532539 <== mon.2 v2:172.21.15.63:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fcdc4047660 con 0x7fcddc1cfb20 2024-04-26T04:26:31.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.262+0000 7fcdd2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:26:31.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.262+0000 7fcdd2ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:26:31.488 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.482+0000 7fcde0962640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T04:26:31.488 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.482+0000 7fcde0962640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T04:26:31.488 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.482+0000 7fcde0962640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:26:31.488 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.482+0000 7fcde0962640 1 -- 172.21.15.63:0/3428532539 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fcddc122500 con 0x7fcddc1cfb20 2024-04-26T04:26:31.489 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.482+0000 7fcdd2ffd640 1 -- 172.21.15.63:0/3428532539 <== mon.2 v2:172.21.15.63:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7fcdc401c580 con 0x7fcddc1cfb20 2024-04-26T04:26:31.489 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.482+0000 7fcdd2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T04:26:31.489 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.482+0000 7fcdd2ffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:26:31.489 INFO:teuthology.orchestra.run.smithi063.stdout:{"election_epoch":28,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":95,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cd1c6a0c-4e4b-4014-89c0-2e535479ef39","modified":"2024-04-26T04:18:03.702339Z","created":"2024-04-26T04:18:03.702339Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3300","nonce":0},{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:3300","nonce":0},{"type":"v1","addr":"172.21.15.186:6789","nonce":0}]},"addr":"172.21.15.186:6789/0","public_addr":"172.21.15.186:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3301","nonce":0},{"type":"v1","addr":"172.21.15.63:6790","nonce":0}]},"addr":"172.21.15.63:6790/0","public_addr":"172.21.15.63:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T04:26:31.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.486+0000 7fcde0962640 1 -- 172.21.15.63:0/3428532539 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fcdc0035f40 msgr2=0x7fcdc0038400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:31.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.486+0000 7fcde0962640 1 --2- 172.21.15.63:0/3428532539 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fcdc0035f40 0x7fcdc0038400 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc001a00 tx=0x7fcdcc0014e0 comp rx=0 tx=0).stop 2024-04-26T04:26:31.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.486+0000 7fcde0962640 10 monclient: shutdown 2024-04-26T04:26:31.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.486+0000 7fcde0962640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:26:31.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.486+0000 7fcde0962640 1 -- 172.21.15.63:0/3428532539 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcddc1cfb20 msgr2=0x7fcddc1d6510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:31.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.486+0000 7fcde0962640 1 --2- 172.21.15.63:0/3428532539 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcddc1cfb20 0x7fcddc1d6510 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fcdc4001ec0 tx=0x7fcdc400d520 comp rx=0 tx=0).stop 2024-04-26T04:26:31.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.486+0000 7fcde0962640 1 -- 172.21.15.63:0/3428532539 shutdown_connections 2024-04-26T04:26:31.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.486+0000 7fcde0962640 1 --2- 172.21.15.63:0/3428532539 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fcdc0035f40 0x7fcdc0038400 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:31.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.486+0000 7fcde0962640 1 --2- 172.21.15.63:0/3428532539 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fcddc1cfb20 0x7fcddc1d6510 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:31.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.486+0000 7fcde0962640 1 --2- 172.21.15.63:0/3428532539 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcddc1cf180 0x7fcddc1cf5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:31.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.486+0000 7fcde0962640 1 --2- 172.21.15.63:0/3428532539 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcddc1360b0 0x7fcddc136490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:31.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.486+0000 7fcde0962640 1 -- 172.21.15.63:0/3428532539 >> 172.21.15.63:0/3428532539 conn(0x7fcddc0a5f40 msgr2=0x7fcddc1c1ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:26:31.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.486+0000 7fcde0962640 1 -- 172.21.15.63:0/3428532539 shutdown_connections 2024-04-26T04:26:31.493 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:31.486+0000 7fcde0962640 1 -- 172.21.15.63:0/3428532539 wait complete. 2024-04-26T04:26:31.754 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[32002]: starting. 2024-04-26T04:26:31.754 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[32002]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-32001 2024-04-26T04:26:31.754 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[32002]: created object 0... 2024-04-26T04:26:31.754 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[32002]: created object 25... 2024-04-26T04:26:31.754 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[32002]: created object 49... 2024-04-26T04:26:31.754 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[32002]: finishing. 2024-04-26T04:26:31.754 INFO:teuthology.orchestra.run.smithi186.stdout:process_1_[32002]: shutting down. 2024-04-26T04:26:32.282 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[32003]: starting. 2024-04-26T04:26:32.282 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[32003]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-32001 2024-04-26T04:26:32.282 INFO:teuthology.orchestra.run.smithi186.stdout:process_2_[32003]: shutting down. 2024-04-26T04:26:34.512 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T04:26:34.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.642+0000 7f0b79a91640 10 monclient: get_monmap_and_config 2024-04-26T04:26:34.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.642+0000 7f0b79a91640 10 monclient: build_initial_monmap 2024-04-26T04:26:34.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.642+0000 7f0b79a91640 10 monclient: monmap: 2024-04-26T04:26:34.646 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:26:34.646 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:34.646 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:26:34.646447+0000 2024-04-26T04:26:34.647 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:26:34.646447+0000 2024-04-26T04:26:34.647 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:26:34.647 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:34.647 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:26:34.647 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:26:34.647 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:26:34.647 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:26:34.647 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:26:34.647 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:34.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.642+0000 7f0b79a91640 1 Processor -- start 2024-04-26T04:26:34.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.642+0000 7f0b79a91640 1 -- start start 2024-04-26T04:26:34.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.642+0000 7f0b79a91640 10 monclient: init 2024-04-26T04:26:34.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.642+0000 7f0b79a91640 10 monclient: _reopen_session rank -1 2024-04-26T04:26:34.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.642+0000 7f0b79a91640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-04-26T04:26:34.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.642+0000 7f0b79a91640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f0b7409fe60 0x7f0b741372f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:34.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.642+0000 7f0b79a91640 10 monclient(hunting): picked mon.noname-a con 0x7f0b7409fe60 addr v2:172.21.15.63:3300/0 2024-04-26T04:26:34.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.642+0000 7f0b79a91640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f0b740a0240 0x7f0b740a2a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:34.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.642+0000 7f0b79a91640 10 monclient(hunting): picked mon.noname-c con 0x7f0b740a0240 addr v2:172.21.15.186:3300/0 2024-04-26T04:26:34.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.642+0000 7f0b79a91640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0b740a2f70 addr v1:172.21.15.63:6789/0 2024-04-26T04:26:34.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.642+0000 7f0b79a91640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b74137830 con 0x7f0b740a2f70 2024-04-26T04:26:34.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.642+0000 7f0b79a91640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:34.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.642+0000 7f0b79a91640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7f0b740a3730 con 0x7f0b7409fe60 2024-04-26T04:26:34.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.642+0000 7f0b79a91640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:34.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.642+0000 7f0b79a91640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f0b740a0c60 con 0x7f0b740a0240 2024-04-26T04:26:34.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.642+0000 7f0b79a91640 10 monclient(hunting): _renew_subs 2024-04-26T04:26:34.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.642+0000 7f0b79a91640 10 monclient(hunting): authenticate will time out at 1426.355134s 2024-04-26T04:26:34.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.642+0000 7f0b737fe640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f0b7409fe60 0x7f0b741372f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:34.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.642+0000 7f0b73fff640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f0b740a2f70 0x7f0b740a3eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:54008/0 (socket says 172.21.15.63:54008) 2024-04-26T04:26:34.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.642+0000 7f0b73fff640 1 -- 172.21.15.63:0/3274789683 learned_addr learned my addr 172.21.15.63:0/3274789683 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-04-26T04:26:34.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.642+0000 7f0b737fe640 10 monclient(hunting): get_auth_request con 0x7f0b7409fe60 auth_method 0 2024-04-26T04:26:34.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.642+0000 7f0b737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:34.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.642+0000 7f0b737fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:34.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.642+0000 7f0b737fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:34.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.642+0000 7f0b737fe640 1 -- 172.21.15.63:0/3274789683 >> v2:172.21.15.63:3300/0 conn(0x7f0b7409fe60 msgr2=0x7f0b741372f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T04:26:34.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.642+0000 7f0b737fe640 1 -- 172.21.15.63:0/3274789683 >> v2:172.21.15.63:3300/0 conn(0x7f0b7409fe60 msgr2=0x7f0b741372f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:26:34.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.642+0000 7f0b737fe640 1 --2- 172.21.15.63:0/3274789683 >> v2:172.21.15.63:3300/0 conn(0x7f0b7409fe60 0x7f0b741372f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:26:34.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.642+0000 7f0b72ffd640 1 --2- 172.21.15.63:0/3274789683 >> v2:172.21.15.186:3300/0 conn(0x7f0b740a0240 0x7f0b740a2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:34.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.642+0000 7f0b737fe640 1 --2- 172.21.15.63:0/3274789683 >> v2:172.21.15.63:3300/0 conn(0x7f0b7409fe60 0x7f0b741372f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:26:34.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.642+0000 7f0b72ffd640 10 monclient(hunting): get_auth_request con 0x7f0b740a0240 auth_method 0 2024-04-26T04:26:34.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.642+0000 7f0b72ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:34.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.642+0000 7f0b72ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:34.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.642+0000 7f0b72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:34.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.642+0000 7f0b72ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:26:34.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.642+0000 7f0b72ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:26:34.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.642+0000 7f0b72ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:26:34.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.646+0000 7f0b72ffd640 10 monclient(hunting): handle_auth_done global_id 34537 payload 274 2024-04-26T04:26:34.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.646+0000 7f0b72ffd640 1 -- 172.21.15.63:0/3274789683 >> v2:172.21.15.63:3300/0 conn(0x7f0b7409fe60 msgr2=0x7f0b741372f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T04:26:34.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.646+0000 7f0b72ffd640 1 --2- 172.21.15.63:0/3274789683 >> v2:172.21.15.63:3300/0 conn(0x7f0b7409fe60 0x7f0b741372f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:34.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.646+0000 7f0b72ffd640 1 -- 172.21.15.63:0/3274789683 >> v1:172.21.15.63:6789/0 conn(0x7f0b740a2f70 legacy=0x7f0b740a3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:26:34.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.646+0000 7f0b72ffd640 10 monclient: _finish_hunting 0 2024-04-26T04:26:34.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.646+0000 7f0b72ffd640 1 monclient: found mon.noname-c 2024-04-26T04:26:34.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.646+0000 7f0b72ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:26:34.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.646+0000 7f0b72ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.186:3300/0 2024-04-26T04:26:34.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.646+0000 7f0b72ffd640 1 -- 172.21.15.63:0/3274789683 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b740a0f40 con 0x7f0b740a0240 2024-04-26T04:26:34.654 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.650+0000 7f0b72ffd640 1 -- 172.21.15.63:0/3274789683 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b740a0240 msgr2=0x7f0b740a2a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:26:34.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.650+0000 7f0b72ffd640 1 --2- 172.21.15.63:0/3274789683 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b740a0240 0x7f0b740a2a30 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f0b7406dab0 tx=0x7f0b6801c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f65797c007fcec52 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:26:34.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.650+0000 7f0b727fc640 1 -- 172.21.15.63:0/3274789683 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b680b0070 con 0x7f0b740a0240 2024-04-26T04:26:34.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.650+0000 7f0b727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:34.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.650+0000 7f0b727fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:26:34.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.650+0000 7f0b727fc640 10 monclient: dump: 2024-04-26T04:26:34.656 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:34.657 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:34.657 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:34.657 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:34.657 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:34.657 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:34.658 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:34.658 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:34.658 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:34.658 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:34.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.650+0000 7f0b727fc640 10 monclient: _finish_auth 0 2024-04-26T04:26:34.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.650+0000 7f0b727fc640 10 monclient: _check_auth_tickets 2024-04-26T04:26:34.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.650+0000 7f0b727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:26:34.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.650+0000 7f0b727fc640 1 -- 172.21.15.63:0/3274789683 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b680ab600 con 0x7f0b740a0240 2024-04-26T04:26:34.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.650+0000 7f0b727fc640 10 monclient: handle_config config(0 keys) 2024-04-26T04:26:34.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.650+0000 7f0b727fc640 1 -- 172.21.15.63:0/3274789683 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b680ab9d0 con 0x7f0b740a0240 2024-04-26T04:26:34.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.650+0000 7f0b79a91640 5 monclient: authenticate success, global_id 34537 2024-04-26T04:26:34.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.650+0000 7f0b79a91640 10 monclient: get_monmap_and_config success 2024-04-26T04:26:34.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.650+0000 7f0b79a91640 4 set_mon_vals no callback set 2024-04-26T04:26:34.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.650+0000 7f0b727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:34.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.650+0000 7f0b727fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T04:26:34.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.650+0000 7f0b727fc640 10 monclient: dump: 2024-04-26T04:26:34.659 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:34.659 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:34.659 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:34.659 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:34.659 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:34.659 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:34.660 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:34.660 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:34.660 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:34.660 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:34.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.650+0000 7f0b79a91640 10 monclient: shutdown 2024-04-26T04:26:34.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.650+0000 7f0b79a91640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:26:34.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.650+0000 7f0b79a91640 1 -- 172.21.15.63:0/3274789683 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b740a0240 msgr2=0x7f0b740a2a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:34.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.650+0000 7f0b79a91640 1 --2- 172.21.15.63:0/3274789683 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b740a0240 0x7f0b740a2a30 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f0b7406dab0 tx=0x7f0b6801c2d0 comp rx=0 tx=0).stop 2024-04-26T04:26:34.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.650+0000 7f0b79a91640 1 -- 172.21.15.63:0/3274789683 shutdown_connections 2024-04-26T04:26:34.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.650+0000 7f0b79a91640 1 --2- 172.21.15.63:0/3274789683 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b740a0240 0x7f0b740a2a30 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:34.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.650+0000 7f0b79a91640 1 --2- 172.21.15.63:0/3274789683 >> v2:172.21.15.63:3300/0 conn(0x7f0b7409fe60 0x7f0b741372f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:34.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.650+0000 7f0b79a91640 1 -- 172.21.15.63:0/3274789683 >> 172.21.15.63:0/3274789683 conn(0x7f0b740a5fb0 msgr2=0x7f0b740a63c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:26:34.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.650+0000 7f0b79a91640 1 -- 172.21.15.63:0/3274789683 shutdown_connections 2024-04-26T04:26:34.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.650+0000 7f0b79a91640 1 -- 172.21.15.63:0/3274789683 wait complete. 2024-04-26T04:26:34.660 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.650+0000 7f0b79a91640 10 monclient: build_initial_monmap 2024-04-26T04:26:34.661 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.650+0000 7f0b79a91640 10 monclient: monmap: 2024-04-26T04:26:34.661 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:26:34.661 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:26:34.661 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:26:34.661 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:26:34.661 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:26:34.661 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:34.661 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:26:34.662 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:26:34.662 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:26:34.662 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:34.667 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.650+0000 7f0b79a91640 1 Processor -- start 2024-04-26T04:26:34.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.650+0000 7f0b79a91640 1 -- start start 2024-04-26T04:26:34.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.650+0000 7f0b79a91640 10 monclient: init 2024-04-26T04:26:34.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b79a91640 10 monclient: _reopen_session rank -1 2024-04-26T04:26:34.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b79a91640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T04:26:34.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b79a91640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f0b7409fe60 0x7f0b741cf470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:34.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b79a91640 10 monclient(hunting): picked mon.noname-c con 0x7f0b7409fe60 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:26:34.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b79a91640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b740a0240 0x7f0b7413e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:34.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b79a91640 10 monclient(hunting): picked mon.noname-a con 0x7f0b740a0240 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:26:34.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b79a91640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b740a2f70 0x7f0b7413e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:34.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b79a91640 10 monclient(hunting): picked mon.noname-b con 0x7f0b740a2f70 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:26:34.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b79a91640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:34.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b79a91640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f0b74141d60 con 0x7f0b740a0240 2024-04-26T04:26:34.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b79a91640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:34.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b79a91640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0b74141be0 con 0x7f0b740a2f70 2024-04-26T04:26:34.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b79a91640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:34.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b79a91640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f0b74141ee0 con 0x7f0b7409fe60 2024-04-26T04:26:34.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b79a91640 10 monclient(hunting): _renew_subs 2024-04-26T04:26:34.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b79a91640 10 monclient(hunting): authenticate will time out at 1426.364248s 2024-04-26T04:26:34.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b72ffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b740a0240 0x7f0b7413e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:34.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b737fe640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f0b7409fe60 0x7f0b741cf470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:34.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b72ffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b740a0240 0x7f0b7413e180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:33506/0 (socket says 172.21.15.63:33506) 2024-04-26T04:26:34.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b72ffd640 1 -- 172.21.15.63:0/4269482057 learned_addr learned my addr 172.21.15.63:0/4269482057 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:26:34.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b72ffd640 10 monclient(hunting): get_auth_request con 0x7f0b740a0240 auth_method 0 2024-04-26T04:26:34.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b72ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:34.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b72ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:34.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:34.669 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b737fe640 10 monclient(hunting): get_auth_request con 0x7f0b7409fe60 auth_method 0 2024-04-26T04:26:34.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:34.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b737fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:34.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b737fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:34.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b72ffd640 1 -- 172.21.15.63:0/4269482057 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b740a0240 msgr2=0x7f0b7413e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T04:26:34.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b72ffd640 1 -- 172.21.15.63:0/4269482057 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b740a0240 msgr2=0x7f0b7413e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T04:26:34.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b72ffd640 1 --2- 172.21.15.63:0/4269482057 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b740a0240 0x7f0b7413e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:26:34.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:26:34.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:26:34.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b72ffd640 1 --2- 172.21.15.63:0/4269482057 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b740a0240 0x7f0b7413e180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:26:34.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:26:34.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b73fff640 1 --2- 172.21.15.63:0/4269482057 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b740a2f70 0x7f0b7413e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:34.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b73fff640 10 monclient(hunting): get_auth_request con 0x7f0b740a2f70 auth_method 0 2024-04-26T04:26:34.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b73fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:34.670 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b73fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:34.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b73fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:34.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b737fe640 10 monclient(hunting): handle_auth_done global_id 24662 payload 824 2024-04-26T04:26:34.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b737fe640 1 -- 172.21.15.63:0/4269482057 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b740a2f70 msgr2=0x7f0b7413e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:34.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b737fe640 1 --2- 172.21.15.63:0/4269482057 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b740a2f70 0x7f0b7413e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:34.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b737fe640 1 -- 172.21.15.63:0/4269482057 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b740a0240 msgr2=0x7f0b7413e180 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T04:26:34.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b737fe640 1 --2- 172.21.15.63:0/4269482057 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b740a0240 0x7f0b7413e180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:34.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b737fe640 10 monclient: _finish_hunting 0 2024-04-26T04:26:34.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b737fe640 1 monclient: found mon.noname-c 2024-04-26T04:26:34.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:26:34.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b737fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.63:3301/0 2024-04-26T04:26:34.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b737fe640 1 -- 172.21.15.63:0/4269482057 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b7413b260 con 0x7f0b7409fe60 2024-04-26T04:26:34.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b73fff640 1 --2- 172.21.15.63:0/4269482057 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b740a2f70 0x7f0b7413e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T04:26:34.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b737fe640 1 -- 172.21.15.63:0/4269482057 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f0b7409fe60 msgr2=0x7f0b741cf470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:26:34.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b737fe640 1 --2- 172.21.15.63:0/4269482057 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f0b7409fe60 0x7f0b741cf470 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f0b6400e7e0 tx=0x7f0b6400eca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:26:34.672 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b5bfff640 1 -- 172.21.15.63:0/4269482057 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b64017030 con 0x7f0b7409fe60 2024-04-26T04:26:34.672 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b5bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:34.672 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b5bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:26:34.672 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b5bfff640 10 monclient: dump: 2024-04-26T04:26:34.672 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:34.672 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:34.672 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:34.672 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:34.672 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:34.672 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:34.672 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:34.672 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:34.672 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:34.672 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:34.672 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b5bfff640 10 monclient: _finish_auth 0 2024-04-26T04:26:34.673 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b5bfff640 10 monclient: _check_auth_tickets 2024-04-26T04:26:34.673 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b5bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:26:34.673 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b79a91640 5 monclient: authenticate success, global_id 24662 2024-04-26T04:26:34.673 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b79a91640 10 monclient: _renew_subs 2024-04-26T04:26:34.673 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b79a91640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:26:34.673 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b79a91640 1 -- 172.21.15.63:0/4269482057 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b740a38f0 con 0x7f0b7409fe60 2024-04-26T04:26:34.673 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b5bfff640 1 -- 172.21.15.63:0/4269482057 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b64004020 con 0x7f0b7409fe60 2024-04-26T04:26:34.673 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b5bfff640 10 monclient: handle_config config(0 keys) 2024-04-26T04:26:34.673 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b5bfff640 1 -- 172.21.15.63:0/4269482057 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b6400f040 con 0x7f0b7409fe60 2024-04-26T04:26:34.673 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b5bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:34.673 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b5bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:26:34.673 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b5bfff640 10 monclient: dump: 2024-04-26T04:26:34.673 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:34.673 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:34.673 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:34.674 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:34.674 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:34.674 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:34.674 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:34.674 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:34.674 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:34.674 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:34.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b79a91640 10 monclient: _renew_subs 2024-04-26T04:26:34.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b79a91640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:26:34.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b71ffb640 4 set_mon_vals no callback set 2024-04-26T04:26:34.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.654+0000 7f0b79a91640 1 -- 172.21.15.63:0/4269482057 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b7413cc70 con 0x7f0b7409fe60 2024-04-26T04:26:34.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.658+0000 7f0b79a91640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:26:34.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.658+0000 7f0b79a91640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:26:34.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.658+0000 7f0b79a91640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:26:34.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.658+0000 7f0b79a91640 1 -- 172.21.15.63:0/4269482057 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b74062e10 con 0x7f0b7409fe60 2024-04-26T04:26:34.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.658+0000 7f0b5bfff640 1 -- 172.21.15.63:0/4269482057 <== mon.2 v2:172.21.15.63:3301/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f0b6401a070 con 0x7f0b7409fe60 2024-04-26T04:26:34.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.658+0000 7f0b5bfff640 1 --2- 172.21.15.63:0/4269482057 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f0b60035ec0 0x7f0b60038380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:34.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.658+0000 7f0b5bfff640 1 -- 172.21.15.63:0/4269482057 <== mon.2 v2:172.21.15.63:3301/0 5 ==== osd_map(321..321 src has 1..321) ==== 4738+0+0 (secure 0 0 0) 0x7f0b640487f0 con 0x7f0b7409fe60 2024-04-26T04:26:34.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.658+0000 7f0b72ffd640 1 --2- 172.21.15.63:0/4269482057 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f0b60035ec0 0x7f0b60038380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:34.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.658+0000 7f0b72ffd640 10 monclient: get_auth_request con 0x7f0b60035ec0 auth_method 0 2024-04-26T04:26:34.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.662+0000 7f0b72ffd640 1 --2- 172.21.15.63:0/4269482057 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f0b60035ec0 0x7f0b60038380 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f0b68000c10 tx=0x7f0b680a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:26:34.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.662+0000 7f0b5bfff640 1 -- 172.21.15.63:0/4269482057 <== mon.2 v2:172.21.15.63:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f0b6401ae40 con 0x7f0b7409fe60 2024-04-26T04:26:34.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.662+0000 7f0b5bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:26:34.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.662+0000 7f0b5bfff640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:26:34.695 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:26:34.695 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[32038]: starting. 2024-04-26T04:26:34.695 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[32038]: creating pool ceph_test_rados_delete_pools_parallel.smithi186-32001 2024-04-26T04:26:34.695 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[32038]: created object 0... 2024-04-26T04:26:34.695 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[32038]: created object 25... 2024-04-26T04:26:34.695 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[32038]: created object 49... 2024-04-26T04:26:34.695 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[32038]: finishing. 2024-04-26T04:26:34.695 INFO:teuthology.orchestra.run.smithi186.stdout:process_3_[32038]: shutting down. 2024-04-26T04:26:34.748 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:26:34.748 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[32040]: starting. 2024-04-26T04:26:34.748 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[32040]: listing objects. 2024-04-26T04:26:34.748 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[32040]: listed object 0... 2024-04-26T04:26:34.748 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[32040]: listed object 25... 2024-04-26T04:26:34.748 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[32040]: saw 50 objects 2024-04-26T04:26:34.749 INFO:teuthology.orchestra.run.smithi186.stdout:process_5_[32040]: shutting down. 2024-04-26T04:26:34.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.886+0000 7f0b79a91640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T04:26:34.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.886+0000 7f0b79a91640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T04:26:34.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.886+0000 7f0b79a91640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:26:34.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.886+0000 7f0b79a91640 1 -- 172.21.15.63:0/4269482057 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f0b74068430 con 0x7f0b7409fe60 2024-04-26T04:26:34.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.886+0000 7f0b5bfff640 1 -- 172.21.15.63:0/4269482057 <== mon.2 v2:172.21.15.63:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f0b6401ca50 con 0x7f0b7409fe60 2024-04-26T04:26:34.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.886+0000 7f0b5bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T04:26:34.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.886+0000 7f0b5bfff640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:26:34.891 INFO:teuthology.orchestra.run.smithi063.stdout:{"election_epoch":28,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":98,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cd1c6a0c-4e4b-4014-89c0-2e535479ef39","modified":"2024-04-26T04:18:03.702339Z","created":"2024-04-26T04:18:03.702339Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3300","nonce":0},{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:3300","nonce":0},{"type":"v1","addr":"172.21.15.186:6789","nonce":0}]},"addr":"172.21.15.186:6789/0","public_addr":"172.21.15.186:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3301","nonce":0},{"type":"v1","addr":"172.21.15.63:6790","nonce":0}]},"addr":"172.21.15.63:6790/0","public_addr":"172.21.15.63:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T04:26:34.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.886+0000 7f0b79a91640 1 -- 172.21.15.63:0/4269482057 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f0b60035ec0 msgr2=0x7f0b60038380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:34.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.886+0000 7f0b79a91640 1 --2- 172.21.15.63:0/4269482057 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f0b60035ec0 0x7f0b60038380 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f0b68000c10 tx=0x7f0b680a6000 comp rx=0 tx=0).stop 2024-04-26T04:26:34.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.886+0000 7f0b79a91640 10 monclient: shutdown 2024-04-26T04:26:34.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.886+0000 7f0b79a91640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:26:34.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.886+0000 7f0b79a91640 1 -- 172.21.15.63:0/4269482057 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f0b7409fe60 msgr2=0x7f0b741cf470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:34.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.886+0000 7f0b79a91640 1 --2- 172.21.15.63:0/4269482057 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f0b7409fe60 0x7f0b741cf470 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f0b6400e7e0 tx=0x7f0b6400eca0 comp rx=0 tx=0).stop 2024-04-26T04:26:34.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.886+0000 7f0b79a91640 1 -- 172.21.15.63:0/4269482057 shutdown_connections 2024-04-26T04:26:34.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.886+0000 7f0b79a91640 1 --2- 172.21.15.63:0/4269482057 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f0b60035ec0 0x7f0b60038380 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:34.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.886+0000 7f0b79a91640 1 --2- 172.21.15.63:0/4269482057 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b740a2f70 0x7f0b7413e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:34.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.886+0000 7f0b79a91640 1 --2- 172.21.15.63:0/4269482057 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b740a0240 0x7f0b7413e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:34.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.886+0000 7f0b79a91640 1 --2- 172.21.15.63:0/4269482057 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f0b7409fe60 0x7f0b741cf470 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:34.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.886+0000 7f0b79a91640 1 -- 172.21.15.63:0/4269482057 >> 172.21.15.63:0/4269482057 conn(0x7f0b740a5fb0 msgr2=0x7f0b740a7cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:26:34.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.886+0000 7f0b79a91640 1 -- 172.21.15.63:0/4269482057 shutdown_connections 2024-04-26T04:26:34.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:34.890+0000 7f0b79a91640 1 -- 172.21.15.63:0/4269482057 wait complete. 2024-04-26T04:26:35.295 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:26:35.295 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[32039]: starting. 2024-04-26T04:26:35.295 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[32039]: deleting pool ceph_test_rados_delete_pools_parallel.smithi186-32001 2024-04-26T04:26:35.295 INFO:teuthology.orchestra.run.smithi186.stdout:process_4_[32039]: shutting down. 2024-04-26T04:26:35.298 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:26:35.298 INFO:teuthology.orchestra.run.smithi186.stdout:******************************* 2024-04-26T04:26:35.298 INFO:teuthology.orchestra.run.smithi186.stdout:******* SUCCESS ********** 2024-04-26T04:26:35.301 DEBUG:teuthology.run_tasks:Unwinding manager mon_thrash 2024-04-26T04:26:35.310 INFO:tasks.mon_thrash:joining mon_thrasher 2024-04-26T04:26:37.911 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T04:26:38.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.038+0000 7f4bb3c06640 10 monclient: get_monmap_and_config 2024-04-26T04:26:38.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.038+0000 7f4bb3c06640 10 monclient: build_initial_monmap 2024-04-26T04:26:38.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.038+0000 7f4bb3c06640 10 monclient: monmap: 2024-04-26T04:26:38.043 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:26:38.043 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:38.043 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:26:38.043281+0000 2024-04-26T04:26:38.043 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:26:38.043281+0000 2024-04-26T04:26:38.044 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:26:38.044 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:38.044 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:26:38.044 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:26:38.044 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:26:38.044 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:26:38.044 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:26:38.044 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:38.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.038+0000 7f4bb3c06640 1 Processor -- start 2024-04-26T04:26:38.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.038+0000 7f4bb3c06640 1 -- start start 2024-04-26T04:26:38.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.038+0000 7f4bb3c06640 10 monclient: init 2024-04-26T04:26:38.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.038+0000 7f4bb3c06640 10 monclient: _reopen_session rank -1 2024-04-26T04:26:38.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.038+0000 7f4bb3c06640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-04-26T04:26:38.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.038+0000 7f4bb3c06640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4bac0a2cd0 addr v1:172.21.15.63:6789/0 2024-04-26T04:26:38.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.038+0000 7f4bb3c06640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4bac0a30b0 addr v1:172.21.15.186:6789/0 2024-04-26T04:26:38.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.038+0000 7f4bb3c06640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f4bac13d720 0x7f4bac13fb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:38.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.038+0000 7f4bb3c06640 10 monclient(hunting): picked mon.noname-a con 0x7f4bac13d720 addr v2:172.21.15.63:3300/0 2024-04-26T04:26:38.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.038+0000 7f4bb3c06640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4bac0a4ee0 con 0x7f4bac0a2cd0 2024-04-26T04:26:38.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.038+0000 7f4bb3c06640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4bac0a0fb0 con 0x7f4bac0a30b0 2024-04-26T04:26:38.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.038+0000 7f4bb3c06640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:38.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.038+0000 7f4bb3c06640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7f4bac0a5380 con 0x7f4bac13d720 2024-04-26T04:26:38.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.038+0000 7f4bb3c06640 10 monclient(hunting): _renew_subs 2024-04-26T04:26:38.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.038+0000 7f4bb3c06640 10 monclient(hunting): authenticate will time out at 1429.751743s 2024-04-26T04:26:38.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.038+0000 7f4bb21a3640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f4bac13d720 0x7f4bac13fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:38.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.038+0000 7f4bb19a2640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f4bac0a2cd0 0x7f4bac0a47d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:54016/0 (socket says 172.21.15.63:54016) 2024-04-26T04:26:38.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.038+0000 7f4bb21a3640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f4bac13d720 0x7f4bac13fb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:33536/0 (socket says 172.21.15.63:33536) 2024-04-26T04:26:38.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.038+0000 7f4bb19a2640 1 -- 172.21.15.63:0/2719790934 learned_addr learned my addr 172.21.15.63:0/2719790934 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-04-26T04:26:38.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.038+0000 7f4bb21a3640 10 monclient(hunting): get_auth_request con 0x7f4bac13d720 auth_method 0 2024-04-26T04:26:38.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.038+0000 7f4bb21a3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:38.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.038+0000 7f4bb21a3640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:38.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.038+0000 7f4bb21a3640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:38.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb21a3640 1 -- 172.21.15.63:0/2719790934 >> v2:172.21.15.63:3300/0 conn(0x7f4bac13d720 msgr2=0x7f4bac13fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T04:26:38.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb21a3640 1 -- 172.21.15.63:0/2719790934 >> v2:172.21.15.63:3300/0 conn(0x7f4bac13d720 msgr2=0x7f4bac13fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:26:38.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb21a3640 1 --2- 172.21.15.63:0/2719790934 >> v2:172.21.15.63:3300/0 conn(0x7f4bac13d720 0x7f4bac13fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:26:38.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb21a3640 1 --2- 172.21.15.63:0/2719790934 >> v2:172.21.15.63:3300/0 conn(0x7f4bac13d720 0x7f4bac13fb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:26:38.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb19a2640 1 -- 172.21.15.63:0/2719790934 >> v1:172.21.15.63:6789/0 conn(0x7f4bac0a2cd0 legacy=0x7f4bac0a47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:26:38.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb11a1640 1 -- 172.21.15.63:0/2719790934 >> v1:172.21.15.186:6789/0 conn(0x7f4bac0a30b0 legacy=0x7f4bac13c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:26:38.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb09a0640 1 -- 172.21.15.63:0/2719790934 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3734813041 0 0) 0x7f4bac0a0fb0 con 0x7f4bac0a30b0 2024-04-26T04:26:38.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb09a0640 10 monclient(hunting): _negotiate 2024-04-26T04:26:38.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb09a0640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:38.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb09a0640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:38.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb09a0640 10 monclient(hunting): my global_id is 34543 2024-04-26T04:26:38.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb09a0640 1 -- 172.21.15.63:0/2719790934 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b98003530 con 0x7f4bac0a30b0 2024-04-26T04:26:38.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb09a0640 1 -- 172.21.15.63:0/2719790934 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3792438442 0 0) 0x7f4b98003530 con 0x7f4bac0a30b0 2024-04-26T04:26:38.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb09a0640 1 -- 172.21.15.63:0/2719790934 >> v2:172.21.15.63:3300/0 conn(0x7f4bac13d720 msgr2=0x7f4bac13fb10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T04:26:38.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb09a0640 1 --2- 172.21.15.63:0/2719790934 >> v2:172.21.15.63:3300/0 conn(0x7f4bac13d720 0x7f4bac13fb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:38.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb09a0640 1 -- 172.21.15.63:0/2719790934 >> v1:172.21.15.63:6789/0 conn(0x7f4bac0a2cd0 legacy=0x7f4bac0a47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:38.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb09a0640 10 monclient: _finish_hunting 0 2024-04-26T04:26:38.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb09a0640 1 monclient: found mon.noname-c-legacy 2024-04-26T04:26:38.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb09a0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:26:38.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb09a0640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.186:6789/0 2024-04-26T04:26:38.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb09a0640 1 -- 172.21.15.63:0/2719790934 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bac0a1290 con 0x7f4bac0a30b0 2024-04-26T04:26:38.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb09a0640 10 monclient: _finish_auth 0 2024-04-26T04:26:38.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb09a0640 10 monclient: _check_auth_tickets 2024-04-26T04:26:38.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb09a0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:26:38.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb09a0640 1 -- 172.21.15.63:0/2719790934 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2612289768 0 0) 0x7f4b9c001a40 con 0x7f4bac0a30b0 2024-04-26T04:26:38.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb09a0640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:38.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb09a0640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-04-26T04:26:38.053 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb09a0640 10 monclient: dump: 2024-04-26T04:26:38.053 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:38.053 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:38.053 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:38.053 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:38.053 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:38.053 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:38.053 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:38.053 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:38.053 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:38.053 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:38.053 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb09a0640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] but i'm connected to v1:172.21.15.186:6789/0, reconnecting 2024-04-26T04:26:38.053 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb09a0640 10 monclient: _reopen_session rank -1 2024-04-26T04:26:38.053 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb09a0640 1 -- 172.21.15.63:0/2719790934 >> v1:172.21.15.186:6789/0 conn(0x7f4bac0a30b0 legacy=0x7f4bac13c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:38.053 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb09a0640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T04:26:38.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb09a0640 1 --2- 172.21.15.63:0/2719790934 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b98003aa0 0x7f4b98003ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:38.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb09a0640 10 monclient(hunting): picked mon.a con 0x7f4b98003aa0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:26:38.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb09a0640 1 --2- 172.21.15.63:0/2719790934 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b98004810 0x7f4b98004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:38.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb09a0640 10 monclient(hunting): picked mon.b con 0x7f4b98004810 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:26:38.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb09a0640 1 --2- 172.21.15.63:0/2719790934 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4b980053d0 0x7f4b9800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:38.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb09a0640 10 monclient(hunting): picked mon.c con 0x7f4b980053d0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:26:38.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb09a0640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:38.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb09a0640 1 -- 172.21.15.63:0/2719790934 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f4bac0a5380 con 0x7f4b98003aa0 2024-04-26T04:26:38.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb09a0640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:38.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb09a0640 1 -- 172.21.15.63:0/2719790934 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4b9800c290 con 0x7f4b98004810 2024-04-26T04:26:38.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb09a0640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:38.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb09a0640 1 -- 172.21.15.63:0/2719790934 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f4b9800c490 con 0x7f4b980053d0 2024-04-26T04:26:38.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb09a0640 10 monclient(hunting): _renew_subs 2024-04-26T04:26:38.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb09a0640 10 monclient(hunting): _finish_auth 0 2024-04-26T04:26:38.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb21a3640 1 --2- 172.21.15.63:0/2719790934 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4b980053d0 0x7f4b9800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:38.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb19a2640 1 --2- 172.21.15.63:0/2719790934 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b98003aa0 0x7f4b98003ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:38.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb21a3640 10 monclient(hunting): get_auth_request con 0x7f4b980053d0 auth_method 0 2024-04-26T04:26:38.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb21a3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:38.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb21a3640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:38.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb21a3640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:26:38.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb19a2640 10 monclient(hunting): get_auth_request con 0x7f4b98003aa0 auth_method 0 2024-04-26T04:26:38.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb19a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:38.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb19a2640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:38.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb19a2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:26:38.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb21a3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:26:38.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb21a3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:26:38.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb11a1640 1 --2- 172.21.15.63:0/2719790934 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b98004810 0x7f4b98004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:38.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb21a3640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T04:26:38.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb19a2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:26:38.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb19a2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:26:38.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb19a2640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T04:26:38.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb11a1640 10 monclient(hunting): get_auth_request con 0x7f4b98004810 auth_method 0 2024-04-26T04:26:38.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb11a1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:38.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb11a1640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:38.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb11a1640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:26:38.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb19a2640 10 monclient(hunting): handle_auth_done global_id 34543 payload 293 2024-04-26T04:26:38.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb19a2640 1 -- 172.21.15.63:0/2719790934 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4b980053d0 msgr2=0x7f4b9800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:26:38.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb19a2640 1 --2- 172.21.15.63:0/2719790934 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4b980053d0 0x7f4b9800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:38.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb19a2640 1 -- 172.21.15.63:0/2719790934 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b98004810 msgr2=0x7f4b98004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:26:38.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb19a2640 1 --2- 172.21.15.63:0/2719790934 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b98004810 0x7f4b98004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:38.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb19a2640 10 monclient: _finish_hunting 0 2024-04-26T04:26:38.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb19a2640 1 monclient: found mon.a 2024-04-26T04:26:38.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb19a2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:26:38.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb19a2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:26:38.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb19a2640 1 -- 172.21.15.63:0/2719790934 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b98003530 con 0x7f4b98003aa0 2024-04-26T04:26:38.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb19a2640 10 monclient: _finish_auth 0 2024-04-26T04:26:38.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb19a2640 10 monclient: _check_auth_tickets 2024-04-26T04:26:38.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb19a2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:26:38.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb3c06640 5 monclient: authenticate success, global_id 34543 2024-04-26T04:26:38.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb21a3640 1 --2- 172.21.15.63:0/2719790934 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4b980053d0 0x7f4b9800b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T04:26:38.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb3c06640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T04:26:38.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb11a1640 1 --2- 172.21.15.63:0/2719790934 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b98004810 0x7f4b98004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T04:26:38.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.042+0000 7f4bb21a3640 1 -- 172.21.15.63:0/2719790934 reap_dead start 2024-04-26T04:26:38.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb19a2640 1 -- 172.21.15.63:0/2719790934 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b98003aa0 msgr2=0x7f4b98003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:26:38.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb19a2640 1 --2- 172.21.15.63:0/2719790934 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b98003aa0 0x7f4b98003ec0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4ba8001a30 tx=0x7f4ba801c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b52891232001a421 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:26:38.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb09a0640 1 -- 172.21.15.63:0/2719790934 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ba8001730 con 0x7f4b98003aa0 2024-04-26T04:26:38.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb09a0640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:38.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb09a0640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:26:38.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb09a0640 10 monclient: dump: 2024-04-26T04:26:38.063 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:38.063 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:38.063 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:38.063 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:38.063 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:38.063 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:38.063 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:38.063 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:38.063 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:38.063 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:38.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb3c06640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T04:26:38.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb09a0640 1 -- 172.21.15.63:0/2719790934 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ba8004ab0 con 0x7f4b98003aa0 2024-04-26T04:26:38.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb09a0640 10 monclient: handle_config config(0 keys) 2024-04-26T04:26:38.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb3c06640 10 monclient: get_monmap_and_config success 2024-04-26T04:26:38.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb09a0640 1 -- 172.21.15.63:0/2719790934 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ba8004e60 con 0x7f4b98003aa0 2024-04-26T04:26:38.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb3c06640 4 set_mon_vals no callback set 2024-04-26T04:26:38.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb09a0640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:38.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb09a0640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:26:38.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb09a0640 10 monclient: dump: 2024-04-26T04:26:38.064 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:38.064 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:38.064 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:38.064 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:38.064 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:38.064 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:38.065 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:38.065 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:38.065 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:38.065 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:38.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb3c06640 10 monclient: shutdown 2024-04-26T04:26:38.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb3c06640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:26:38.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb3c06640 1 -- 172.21.15.63:0/2719790934 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b98003aa0 msgr2=0x7f4b98003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:38.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb3c06640 1 --2- 172.21.15.63:0/2719790934 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b98003aa0 0x7f4b98003ec0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4ba8001a30 tx=0x7f4ba801c650 comp rx=0 tx=0).stop 2024-04-26T04:26:38.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb3c06640 1 -- 172.21.15.63:0/2719790934 shutdown_connections 2024-04-26T04:26:38.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb3c06640 1 --2- 172.21.15.63:0/2719790934 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b98003aa0 0x7f4b98003ec0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:38.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb3c06640 1 -- 172.21.15.63:0/2719790934 >> 172.21.15.63:0/2719790934 conn(0x7f4bac0a5f40 msgr2=0x7f4bac0a6320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:26:38.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb3c06640 1 -- 172.21.15.63:0/2719790934 shutdown_connections 2024-04-26T04:26:38.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb3c06640 1 -- 172.21.15.63:0/2719790934 wait complete. 2024-04-26T04:26:38.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb3c06640 10 monclient: build_initial_monmap 2024-04-26T04:26:38.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb3c06640 10 monclient: monmap: 2024-04-26T04:26:38.066 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:26:38.066 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:26:38.066 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:26:38.066 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:26:38.066 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:26:38.066 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:38.066 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:26:38.066 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:26:38.066 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:26:38.066 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:38.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb3c06640 1 Processor -- start 2024-04-26T04:26:38.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb3c06640 1 -- start start 2024-04-26T04:26:38.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb3c06640 10 monclient: init 2024-04-26T04:26:38.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb3c06640 10 monclient: _reopen_session rank -1 2024-04-26T04:26:38.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb3c06640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T04:26:38.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb3c06640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4b98003aa0 0x7f4bac1cef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:38.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb3c06640 10 monclient(hunting): picked mon.noname-c con 0x7f4b98003aa0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:26:38.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb3c06640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4bac135df0 0x7f4bac11fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:38.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb3c06640 10 monclient(hunting): picked mon.noname-a con 0x7f4bac135df0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:26:38.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb3c06640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4bac136210 0x7f4bac11ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:38.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb3c06640 10 monclient(hunting): picked mon.noname-b con 0x7f4bac136210 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:26:38.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb3c06640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:38.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb3c06640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f4bac137670 con 0x7f4bac135df0 2024-04-26T04:26:38.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb3c06640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:38.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb3c06640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4bac1374f0 con 0x7f4bac136210 2024-04-26T04:26:38.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb3c06640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:38.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb3c06640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f4bac1377f0 con 0x7f4b98003aa0 2024-04-26T04:26:38.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb3c06640 10 monclient(hunting): _renew_subs 2024-04-26T04:26:38.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb3c06640 10 monclient(hunting): authenticate will time out at 1429.763143s 2024-04-26T04:26:38.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb19a2640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4b98003aa0 0x7f4bac1cef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:38.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb19a2640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4b98003aa0 0x7f4bac1cef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:35002/0 (socket says 172.21.15.63:35002) 2024-04-26T04:26:38.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb19a2640 1 -- 172.21.15.63:0/3552196767 learned_addr learned my addr 172.21.15.63:0/3552196767 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:26:38.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb19a2640 10 monclient(hunting): get_auth_request con 0x7f4b98003aa0 auth_method 0 2024-04-26T04:26:38.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb11a1640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4bac135df0 0x7f4bac11fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:38.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb19a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:38.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb19a2640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:38.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb19a2640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:38.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb11a1640 10 monclient(hunting): get_auth_request con 0x7f4bac135df0 auth_method 0 2024-04-26T04:26:38.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb11a1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:38.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb11a1640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:38.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb11a1640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:38.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb19a2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:26:38.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb19a2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:26:38.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb19a2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:26:38.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb11a1640 1 -- 172.21.15.63:0/3552196767 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4bac135df0 msgr2=0x7f4bac11fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-04-26T04:26:38.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb11a1640 1 -- 172.21.15.63:0/3552196767 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4bac135df0 msgr2=0x7f4bac11fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T04:26:38.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb11a1640 1 --2- 172.21.15.63:0/3552196767 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4bac135df0 0x7f4bac11fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:26:38.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb11a1640 1 --2- 172.21.15.63:0/3552196767 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4bac135df0 0x7f4bac11fa90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:26:38.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb21a3640 1 --2- 172.21.15.63:0/3552196767 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4bac136210 0x7f4bac11ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:38.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb19a2640 10 monclient(hunting): handle_auth_done global_id 24665 payload 824 2024-04-26T04:26:38.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb19a2640 1 -- 172.21.15.63:0/3552196767 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4bac136210 msgr2=0x7f4bac11ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:38.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb19a2640 1 --2- 172.21.15.63:0/3552196767 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4bac136210 0x7f4bac11ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:38.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb19a2640 1 -- 172.21.15.63:0/3552196767 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4bac135df0 msgr2=0x7f4bac11fa90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-26T04:26:38.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb19a2640 1 --2- 172.21.15.63:0/3552196767 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4bac135df0 0x7f4bac11fa90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:38.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb19a2640 10 monclient: _finish_hunting 0 2024-04-26T04:26:38.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb19a2640 1 monclient: found mon.noname-c 2024-04-26T04:26:38.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb19a2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:26:38.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb19a2640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.63:3301/0 2024-04-26T04:26:38.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb19a2640 1 -- 172.21.15.63:0/3552196767 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bac1214c0 con 0x7f4b98003aa0 2024-04-26T04:26:38.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb21a3640 10 monclient: get_auth_request con 0x7f4bac136210 auth_method 0 2024-04-26T04:26:38.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.050+0000 7f4bb21a3640 1 --2- 172.21.15.63:0/3552196767 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4bac136210 0x7f4bac11ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T04:26:38.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.054+0000 7f4bb19a2640 1 -- 172.21.15.63:0/3552196767 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4b98003aa0 msgr2=0x7f4bac1cef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:26:38.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.054+0000 7f4bb19a2640 1 --2- 172.21.15.63:0/3552196767 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4b98003aa0 0x7f4bac1cef40 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4ba8001a30 tx=0x7f4b98005840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:26:38.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.054+0000 7f4ba27fc640 1 -- 172.21.15.63:0/3552196767 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ba8001730 con 0x7f4b98003aa0 2024-04-26T04:26:38.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.054+0000 7f4ba27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:38.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.054+0000 7f4ba27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:26:38.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.054+0000 7f4ba27fc640 10 monclient: dump: 2024-04-26T04:26:38.071 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:38.071 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:38.071 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:38.071 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:38.071 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:38.071 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:38.071 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:38.071 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:38.071 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:38.071 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:38.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.054+0000 7f4ba27fc640 10 monclient: _finish_auth 0 2024-04-26T04:26:38.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.054+0000 7f4ba27fc640 10 monclient: _check_auth_tickets 2024-04-26T04:26:38.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.054+0000 7f4ba27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:26:38.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.054+0000 7f4ba27fc640 1 -- 172.21.15.63:0/3552196767 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ba8004c60 con 0x7f4b98003aa0 2024-04-26T04:26:38.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.054+0000 7f4bb3c06640 5 monclient: authenticate success, global_id 24665 2024-04-26T04:26:38.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.054+0000 7f4bb3c06640 10 monclient: _renew_subs 2024-04-26T04:26:38.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.054+0000 7f4bb3c06640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:26:38.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.054+0000 7f4bb3c06640 1 -- 172.21.15.63:0/3552196767 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bac122f10 con 0x7f4b98003aa0 2024-04-26T04:26:38.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.054+0000 7f4ba27fc640 10 monclient: handle_config config(0 keys) 2024-04-26T04:26:38.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.054+0000 7f4ba27fc640 1 -- 172.21.15.63:0/3552196767 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ba80b9670 con 0x7f4b98003aa0 2024-04-26T04:26:38.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.054+0000 7f4ba27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:38.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.054+0000 7f4ba27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:26:38.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.054+0000 7f4ba27fc640 10 monclient: dump: 2024-04-26T04:26:38.072 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:38.073 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:38.073 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:38.073 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:38.073 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:38.073 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:38.073 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:38.073 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:38.073 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:38.073 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:38.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.054+0000 7f4ba3fff640 4 set_mon_vals no callback set 2024-04-26T04:26:38.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.054+0000 7f4bb3c06640 10 monclient: _renew_subs 2024-04-26T04:26:38.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.054+0000 7f4bb3c06640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:26:38.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.054+0000 7f4bb3c06640 1 -- 172.21.15.63:0/3552196767 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bac122020 con 0x7f4b98003aa0 2024-04-26T04:26:38.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.054+0000 7f4bb3c06640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:26:38.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.054+0000 7f4bb3c06640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:26:38.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.054+0000 7f4bb3c06640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:26:38.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.054+0000 7f4bb3c06640 1 -- 172.21.15.63:0/3552196767 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bac062e10 con 0x7f4b98003aa0 2024-04-26T04:26:38.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.054+0000 7f4ba27fc640 1 -- 172.21.15.63:0/3552196767 <== mon.2 v2:172.21.15.63:3301/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f4ba80c2440 con 0x7f4b98003aa0 2024-04-26T04:26:38.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.054+0000 7f4ba27fc640 1 --2- 172.21.15.63:0/3552196767 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f4b90035ec0 0x7f4b90038380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:38.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.054+0000 7f4ba27fc640 1 -- 172.21.15.63:0/3552196767 <== mon.2 v2:172.21.15.63:3301/0 5 ==== osd_map(323..323 src has 1..323) ==== 4346+0+0 (secure 0 0 0) 0x7f4ba80e2d30 con 0x7f4b98003aa0 2024-04-26T04:26:38.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.054+0000 7f4bb11a1640 1 --2- 172.21.15.63:0/3552196767 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f4b90035ec0 0x7f4b90038380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:38.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.054+0000 7f4bb11a1640 10 monclient: get_auth_request con 0x7f4b90035ec0 auth_method 0 2024-04-26T04:26:38.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.058+0000 7f4bb11a1640 1 --2- 172.21.15.63:0/3552196767 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f4b90035ec0 0x7f4b90038380 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c001cf0 tx=0x7f4b9c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:26:38.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.058+0000 7f4ba27fc640 1 -- 172.21.15.63:0/3552196767 <== mon.2 v2:172.21.15.63:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f4ba80163b0 con 0x7f4b98003aa0 2024-04-26T04:26:38.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.062+0000 7f4ba27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:26:38.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.062+0000 7f4ba27fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:26:38.287 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.282+0000 7f4bb3c06640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T04:26:38.287 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.282+0000 7f4bb3c06640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T04:26:38.287 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.282+0000 7f4bb3c06640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:26:38.287 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.282+0000 7f4bb3c06640 1 -- 172.21.15.63:0/3552196767 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4bac0683d0 con 0x7f4b98003aa0 2024-04-26T04:26:38.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.282+0000 7f4ba27fc640 1 -- 172.21.15.63:0/3552196767 <== mon.2 v2:172.21.15.63:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f4ba8004e10 con 0x7f4b98003aa0 2024-04-26T04:26:38.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.282+0000 7f4ba27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T04:26:38.288 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.282+0000 7f4ba27fc640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:26:38.288 INFO:teuthology.orchestra.run.smithi063.stdout:{"election_epoch":28,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":102,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cd1c6a0c-4e4b-4014-89c0-2e535479ef39","modified":"2024-04-26T04:18:03.702339Z","created":"2024-04-26T04:18:03.702339Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3300","nonce":0},{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:3300","nonce":0},{"type":"v1","addr":"172.21.15.186:6789","nonce":0}]},"addr":"172.21.15.186:6789/0","public_addr":"172.21.15.186:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3301","nonce":0},{"type":"v1","addr":"172.21.15.63:6790","nonce":0}]},"addr":"172.21.15.63:6790/0","public_addr":"172.21.15.63:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T04:26:38.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.286+0000 7f4bb3c06640 1 -- 172.21.15.63:0/3552196767 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f4b90035ec0 msgr2=0x7f4b90038380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:38.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.286+0000 7f4bb3c06640 1 --2- 172.21.15.63:0/3552196767 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f4b90035ec0 0x7f4b90038380 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c001cf0 tx=0x7f4b9c009040 comp rx=0 tx=0).stop 2024-04-26T04:26:38.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.286+0000 7f4bb3c06640 10 monclient: shutdown 2024-04-26T04:26:38.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.286+0000 7f4bb3c06640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:26:38.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.286+0000 7f4bb3c06640 1 -- 172.21.15.63:0/3552196767 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4b98003aa0 msgr2=0x7f4bac1cef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:38.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.286+0000 7f4bb3c06640 1 --2- 172.21.15.63:0/3552196767 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4b98003aa0 0x7f4bac1cef40 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4ba8001a30 tx=0x7f4b98005840 comp rx=0 tx=0).stop 2024-04-26T04:26:38.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.286+0000 7f4bb3c06640 1 -- 172.21.15.63:0/3552196767 shutdown_connections 2024-04-26T04:26:38.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.286+0000 7f4bb3c06640 1 --2- 172.21.15.63:0/3552196767 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f4b90035ec0 0x7f4b90038380 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:38.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.286+0000 7f4bb3c06640 1 --2- 172.21.15.63:0/3552196767 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4bac136210 0x7f4bac11ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:38.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.286+0000 7f4bb3c06640 1 --2- 172.21.15.63:0/3552196767 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4bac135df0 0x7f4bac11fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:38.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.286+0000 7f4bb3c06640 1 --2- 172.21.15.63:0/3552196767 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f4b98003aa0 0x7f4bac1cef40 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:38.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.286+0000 7f4bb3c06640 1 -- 172.21.15.63:0/3552196767 >> 172.21.15.63:0/3552196767 conn(0x7f4bac0a5f40 msgr2=0x7f4bac1c2810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:26:38.292 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.286+0000 7f4bb3c06640 1 -- 172.21.15.63:0/3552196767 shutdown_connections 2024-04-26T04:26:38.292 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:38.286+0000 7f4bb3c06640 1 -- 172.21.15.63:0/3552196767 wait complete. 2024-04-26T04:26:41.308 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T04:26:41.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:41.434+0000 7f94ad254640 10 monclient: get_monmap_and_config 2024-04-26T04:26:41.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:41.434+0000 7f94ad254640 10 monclient: build_initial_monmap 2024-04-26T04:26:41.439 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:41.434+0000 7f94ad254640 10 monclient: monmap: 2024-04-26T04:26:41.439 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:26:41.439 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:41.440 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:26:41.439492+0000 2024-04-26T04:26:41.440 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:26:41.439492+0000 2024-04-26T04:26:41.440 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:26:41.440 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:41.440 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:26:41.440 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:26:41.440 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:26:41.440 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:26:41.440 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:26:41.440 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:41.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:41.434+0000 7f94ad254640 1 Processor -- start 2024-04-26T04:26:41.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:41.434+0000 7f94ad254640 1 -- start start 2024-04-26T04:26:41.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:41.434+0000 7f94ad254640 10 monclient: init 2024-04-26T04:26:41.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:41.434+0000 7f94ad254640 10 monclient: _reopen_session rank -1 2024-04-26T04:26:41.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:41.434+0000 7f94ad254640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-04-26T04:26:41.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:41.434+0000 7f94ad254640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f94a80a2cd0 0x7f94a80a30b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:41.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:41.434+0000 7f94ad254640 10 monclient(hunting): picked mon.noname-c con 0x7f94a80a2cd0 addr v2:172.21.15.186:3300/0 2024-04-26T04:26:41.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:41.434+0000 7f94ad254640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f94a80a47d0 addr v1:172.21.15.63:6789/0 2024-04-26T04:26:41.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:41.434+0000 7f94ad254640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f94a80a4c00 0x7f94a81373f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:41.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:41.434+0000 7f94ad254640 10 monclient(hunting): picked mon.noname-a con 0x7f94a80a4c00 addr v2:172.21.15.63:3300/0 2024-04-26T04:26:41.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:41.434+0000 7f94ad254640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94a80a0fb0 con 0x7f94a80a47d0 2024-04-26T04:26:41.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:41.434+0000 7f94ad254640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:41.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:41.434+0000 7f94ad254640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7f94a80a35f0 con 0x7f94a80a4c00 2024-04-26T04:26:41.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:41.434+0000 7f94ad254640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:41.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:41.434+0000 7f94ad254640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f94a80a5380 con 0x7f94a80a2cd0 2024-04-26T04:26:41.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:41.434+0000 7f94ad254640 10 monclient(hunting): _renew_subs 2024-04-26T04:26:41.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:41.434+0000 7f94ad254640 10 monclient(hunting): authenticate will time out at 1433.147334s 2024-04-26T04:26:41.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:41.434+0000 7f94a659c640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f94a80a47d0 0x7f94a8136ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:37468/0 (socket says 172.21.15.63:37468) 2024-04-26T04:26:41.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:41.434+0000 7f94a659c640 1 -- 172.21.15.63:0/1550164905 learned_addr learned my addr 172.21.15.63:0/1550164905 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-04-26T04:26:41.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:41.434+0000 7f94a759e640 1 --2- 172.21.15.63:0/1550164905 >> v2:172.21.15.63:3300/0 conn(0x7f94a80a4c00 0x7f94a81373f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:41.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:41.434+0000 7f94a759e640 10 monclient(hunting): get_auth_request con 0x7f94a80a4c00 auth_method 0 2024-04-26T04:26:41.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:41.434+0000 7f94a759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:41.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:41.434+0000 7f94a759e640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:41.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:41.434+0000 7f94a759e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:41.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:41.434+0000 7f94a759e640 1 -- 172.21.15.63:0/1550164905 >> v2:172.21.15.63:3300/0 conn(0x7f94a80a4c00 msgr2=0x7f94a81373f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T04:26:41.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:41.434+0000 7f94a759e640 1 -- 172.21.15.63:0/1550164905 >> v2:172.21.15.63:3300/0 conn(0x7f94a80a4c00 msgr2=0x7f94a81373f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:26:41.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:41.434+0000 7f94a759e640 1 --2- 172.21.15.63:0/1550164905 >> v2:172.21.15.63:3300/0 conn(0x7f94a80a4c00 0x7f94a81373f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:26:41.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:41.434+0000 7f94a759e640 1 --2- 172.21.15.63:0/1550164905 >> v2:172.21.15.63:3300/0 conn(0x7f94a80a4c00 0x7f94a81373f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:26:41.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:41.434+0000 7f94a6d9d640 1 --2- 172.21.15.63:0/1550164905 >> v2:172.21.15.186:3300/0 conn(0x7f94a80a2cd0 0x7f94a80a30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:41.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:41.434+0000 7f94a6d9d640 10 monclient(hunting): get_auth_request con 0x7f94a80a2cd0 auth_method 0 2024-04-26T04:26:41.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:41.434+0000 7f94a6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:41.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:41.434+0000 7f94a6d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:41.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:41.434+0000 7f94a6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:41.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:41.438+0000 7f94a6d9d640 1 -- 172.21.15.63:0/1550164905 >> v2:172.21.15.186:3300/0 conn(0x7f94a80a2cd0 msgr2=0x7f94a80a30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T04:26:41.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:41.438+0000 7f94a6d9d640 1 -- 172.21.15.63:0/1550164905 >> v2:172.21.15.186:3300/0 conn(0x7f94a80a2cd0 msgr2=0x7f94a80a30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:26:41.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:41.438+0000 7f94a6d9d640 1 --2- 172.21.15.63:0/1550164905 >> v2:172.21.15.186:3300/0 conn(0x7f94a80a2cd0 0x7f94a80a30b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:26:41.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:41.438+0000 7f94a6d9d640 1 --2- 172.21.15.63:0/1550164905 >> v2:172.21.15.186:3300/0 conn(0x7f94a80a2cd0 0x7f94a80a30b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:26:41.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:41.438+0000 7f94a659c640 1 -- 172.21.15.63:0/1550164905 >> v1:172.21.15.63:6789/0 conn(0x7f94a80a47d0 legacy=0x7f94a8136ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:26:41.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:41.638+0000 7f94a759e640 1 --2- 172.21.15.63:0/1550164905 >> v2:172.21.15.63:3300/0 conn(0x7f94a80a4c00 0x7f94a81373f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:41.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:41.638+0000 7f94a759e640 10 monclient(hunting): get_auth_request con 0x7f94a80a4c00 auth_method 0 2024-04-26T04:26:41.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:41.638+0000 7f94a759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:41.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:41.638+0000 7f94a759e640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:41.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:41.638+0000 7f94a759e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:26:41.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:41.638+0000 7f94a759e640 1 -- 172.21.15.63:0/1550164905 >> v2:172.21.15.63:3300/0 conn(0x7f94a80a4c00 msgr2=0x7f94a81373f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T04:26:41.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:41.638+0000 7f94a759e640 1 -- 172.21.15.63:0/1550164905 >> v2:172.21.15.63:3300/0 conn(0x7f94a80a4c00 msgr2=0x7f94a81373f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:26:41.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:41.638+0000 7f94a759e640 1 --2- 172.21.15.63:0/1550164905 >> v2:172.21.15.63:3300/0 conn(0x7f94a80a4c00 0x7f94a81373f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:26:41.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:41.638+0000 7f94a759e640 1 --2- 172.21.15.63:0/1550164905 >> v2:172.21.15.63:3300/0 conn(0x7f94a80a4c00 0x7f94a81373f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T04:26:41.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:41.642+0000 7f94a6d9d640 1 --2- 172.21.15.63:0/1550164905 >> v2:172.21.15.186:3300/0 conn(0x7f94a80a2cd0 0x7f94a80a30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:41.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:41.642+0000 7f94a6d9d640 10 monclient(hunting): get_auth_request con 0x7f94a80a2cd0 auth_method 0 2024-04-26T04:26:41.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:41.642+0000 7f94a6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:41.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:41.642+0000 7f94a6d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:41.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:41.642+0000 7f94a6d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:26:41.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:41.642+0000 7f94a6d9d640 1 -- 172.21.15.63:0/1550164905 >> v2:172.21.15.186:3300/0 conn(0x7f94a80a2cd0 msgr2=0x7f94a80a30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T04:26:41.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:41.642+0000 7f94a6d9d640 1 -- 172.21.15.63:0/1550164905 >> v2:172.21.15.186:3300/0 conn(0x7f94a80a2cd0 msgr2=0x7f94a80a30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:26:41.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:41.642+0000 7f94a6d9d640 1 --2- 172.21.15.63:0/1550164905 >> v2:172.21.15.186:3300/0 conn(0x7f94a80a2cd0 0x7f94a80a30b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:26:41.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:41.642+0000 7f94a6d9d640 1 --2- 172.21.15.63:0/1550164905 >> v2:172.21.15.186:3300/0 conn(0x7f94a80a2cd0 0x7f94a80a30b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T04:26:42.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:42.042+0000 7f94a759e640 1 --2- 172.21.15.63:0/1550164905 >> v2:172.21.15.63:3300/0 conn(0x7f94a80a4c00 0x7f94a81373f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:42.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:42.042+0000 7f94a759e640 10 monclient(hunting): get_auth_request con 0x7f94a80a4c00 auth_method 0 2024-04-26T04:26:42.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:42.042+0000 7f94a759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:42.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:42.042+0000 7f94a759e640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:42.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:42.042+0000 7f94a759e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:26:42.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:42.042+0000 7f94a759e640 1 -- 172.21.15.63:0/1550164905 >> v2:172.21.15.63:3300/0 conn(0x7f94a80a4c00 msgr2=0x7f94a81373f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T04:26:42.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:42.042+0000 7f94a759e640 1 -- 172.21.15.63:0/1550164905 >> v2:172.21.15.63:3300/0 conn(0x7f94a80a4c00 msgr2=0x7f94a81373f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:26:42.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:42.042+0000 7f94a759e640 1 --2- 172.21.15.63:0/1550164905 >> v2:172.21.15.63:3300/0 conn(0x7f94a80a4c00 0x7f94a81373f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:26:42.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:42.042+0000 7f94a759e640 1 --2- 172.21.15.63:0/1550164905 >> v2:172.21.15.63:3300/0 conn(0x7f94a80a4c00 0x7f94a81373f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T04:26:42.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:42.046+0000 7f94a6d9d640 1 --2- 172.21.15.63:0/1550164905 >> v2:172.21.15.186:3300/0 conn(0x7f94a80a2cd0 0x7f94a80a30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:42.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:42.046+0000 7f94a6d9d640 10 monclient(hunting): get_auth_request con 0x7f94a80a2cd0 auth_method 0 2024-04-26T04:26:42.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:42.046+0000 7f94a6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:42.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:42.046+0000 7f94a6d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:42.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:42.046+0000 7f94a6d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:26:42.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:42.046+0000 7f94a6d9d640 1 -- 172.21.15.63:0/1550164905 >> v2:172.21.15.186:3300/0 conn(0x7f94a80a2cd0 msgr2=0x7f94a80a30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T04:26:42.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:42.046+0000 7f94a6d9d640 1 -- 172.21.15.63:0/1550164905 >> v2:172.21.15.186:3300/0 conn(0x7f94a80a2cd0 msgr2=0x7f94a80a30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:26:42.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:42.046+0000 7f94a6d9d640 1 --2- 172.21.15.63:0/1550164905 >> v2:172.21.15.186:3300/0 conn(0x7f94a80a2cd0 0x7f94a80a30b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:26:42.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:42.046+0000 7f94a6d9d640 1 --2- 172.21.15.63:0/1550164905 >> v2:172.21.15.186:3300/0 conn(0x7f94a80a2cd0 0x7f94a80a30b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T04:26:42.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:42.817+0000 7f94a659c640 1 -- 172.21.15.63:0/1550164905 >> v1:172.21.15.63:6789/0 conn(0x7f94a80a47d0 legacy=0x7f94a8136ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-04-26T04:26:42.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:42.817+0000 7f94a659c640 1 -- 172.21.15.63:0/1550164905 >> v1:172.21.15.63:6789/0 conn(0x7f94a80a47d0 legacy=0x7f94a8136ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T04:26:42.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:42.817+0000 7f94a659c640 1 --1- 172.21.15.63:0/1550164905 >> v1:172.21.15.63:6789/0 conn(0x7f94a80a47d0 0x7f94a8136ce0 :-1 s=OPENED pgs=34 cs=1 l=1).handle_message read tag failed 2024-04-26T04:26:42.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:42.817+0000 7f94a659c640 1 --1- 172.21.15.63:0/1550164905 >> v1:172.21.15.63:6789/0 conn(0x7f94a80a47d0 0x7f94a8136ce0 :-1 s=OPENED pgs=34 cs=1 l=1).fault on lossy channel, failing 2024-04-26T04:26:42.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:42.817+0000 7f94a5d9b640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.63:6789/0 2024-04-26T04:26:42.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:42.845+0000 7f94a759e640 1 --2- 172.21.15.63:0/1550164905 >> v2:172.21.15.63:3300/0 conn(0x7f94a80a4c00 0x7f94a81373f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:42.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:42.845+0000 7f94a759e640 10 monclient(hunting): get_auth_request con 0x7f94a80a4c00 auth_method 0 2024-04-26T04:26:42.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:42.845+0000 7f94a759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:42.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:42.845+0000 7f94a759e640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:42.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:42.845+0000 7f94a759e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:26:42.851 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:42.845+0000 7f94a759e640 1 -- 172.21.15.63:0/1550164905 >> v2:172.21.15.63:3300/0 conn(0x7f94a80a4c00 msgr2=0x7f94a81373f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T04:26:42.851 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:42.845+0000 7f94a759e640 1 -- 172.21.15.63:0/1550164905 >> v2:172.21.15.63:3300/0 conn(0x7f94a80a4c00 msgr2=0x7f94a81373f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:26:42.851 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:42.845+0000 7f94a759e640 1 --2- 172.21.15.63:0/1550164905 >> v2:172.21.15.63:3300/0 conn(0x7f94a80a4c00 0x7f94a81373f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:26:42.851 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:42.845+0000 7f94a759e640 1 --2- 172.21.15.63:0/1550164905 >> v2:172.21.15.63:3300/0 conn(0x7f94a80a4c00 0x7f94a81373f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T04:26:42.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:42.849+0000 7f94a6d9d640 1 --2- 172.21.15.63:0/1550164905 >> v2:172.21.15.186:3300/0 conn(0x7f94a80a2cd0 0x7f94a80a30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:42.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:42.849+0000 7f94a6d9d640 10 monclient(hunting): get_auth_request con 0x7f94a80a2cd0 auth_method 0 2024-04-26T04:26:42.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:42.849+0000 7f94a6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:42.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:42.849+0000 7f94a6d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:42.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:42.849+0000 7f94a6d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:26:42.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:42.849+0000 7f94a6d9d640 1 -- 172.21.15.63:0/1550164905 >> v2:172.21.15.186:3300/0 conn(0x7f94a80a2cd0 msgr2=0x7f94a80a30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T04:26:42.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:42.849+0000 7f94a6d9d640 1 -- 172.21.15.63:0/1550164905 >> v2:172.21.15.186:3300/0 conn(0x7f94a80a2cd0 msgr2=0x7f94a80a30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:26:42.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:42.849+0000 7f94a6d9d640 1 --2- 172.21.15.63:0/1550164905 >> v2:172.21.15.186:3300/0 conn(0x7f94a80a2cd0 0x7f94a80a30b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:26:42.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:42.849+0000 7f94a6d9d640 1 --2- 172.21.15.63:0/1550164905 >> v2:172.21.15.186:3300/0 conn(0x7f94a80a2cd0 0x7f94a80a30b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T04:26:44.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:44.433+0000 7f94a4d99640 10 monclient(hunting): tick 2024-04-26T04:26:44.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:44.433+0000 7f94a4d99640 10 monclient(hunting): _check_auth_tickets 2024-04-26T04:26:44.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:44.433+0000 7f94a4d99640 1 monclient(hunting): continuing hunt 2024-04-26T04:26:44.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:44.433+0000 7f94a4d99640 10 monclient(hunting): _reopen_session rank -1 2024-04-26T04:26:44.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:44.433+0000 7f94a4d99640 1 -- 172.21.15.63:0/1550164905 >> v2:172.21.15.186:3300/0 conn(0x7f94a80a2cd0 msgr2=0x7f94a80a30b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T04:26:44.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:44.433+0000 7f94a4d99640 1 --2- 172.21.15.63:0/1550164905 >> v2:172.21.15.186:3300/0 conn(0x7f94a80a2cd0 0x7f94a80a30b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:44.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:44.433+0000 7f94a4d99640 1 -- 172.21.15.63:0/1550164905 >> v2:172.21.15.63:3300/0 conn(0x7f94a80a4c00 msgr2=0x7f94a81373f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T04:26:44.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:44.433+0000 7f94a4d99640 1 --2- 172.21.15.63:0/1550164905 >> v2:172.21.15.63:3300/0 conn(0x7f94a80a4c00 0x7f94a81373f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:44.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:44.433+0000 7f94a4d99640 1 -- 172.21.15.63:0/1550164905 >> v1:172.21.15.63:6789/0 conn(0x7f94a80a47d0 legacy=0x7f94a8136ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:26:44.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:44.433+0000 7f94a4d99640 10 monclient: _add_conns ranks=[1,2] 2024-04-26T04:26:44.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:44.433+0000 7f94a4d99640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9494003620 addr v1:172.21.15.186:6789/0 2024-04-26T04:26:44.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:44.433+0000 7f94a4d99640 1 --2- 172.21.15.63:0/1550164905 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9494007220 0x7f94940096c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:44.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:44.433+0000 7f94a4d99640 10 monclient(hunting): picked mon.noname-b con 0x7f9494007220 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:26:44.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:44.433+0000 7f94a4d99640 1 -- 172.21.15.63:0/1550164905 --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94a80a1290 con 0x7f9494003620 2024-04-26T04:26:44.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:44.433+0000 7f94a4d99640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:44.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:44.433+0000 7f94a4d99640 1 -- 172.21.15.63:0/1550164905 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f94a80a35f0 con 0x7f9494007220 2024-04-26T04:26:44.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:44.433+0000 7f94a4d99640 10 monclient(hunting): _renew_subs 2024-04-26T04:26:44.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:44.433+0000 7f94a659c640 1 --2- 172.21.15.63:0/1550164905 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9494007220 0x7f94940096c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:44.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:44.437+0000 7f94a659c640 10 monclient(hunting): get_auth_request con 0x7f9494007220 auth_method 0 2024-04-26T04:26:44.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:44.437+0000 7f94a659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:44.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:44.437+0000 7f94a659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:44.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:44.437+0000 7f94a659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:44.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:44.437+0000 7f94a659c640 1 -- 172.21.15.63:0/1550164905 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9494007220 msgr2=0x7f94940096c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T04:26:44.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:44.437+0000 7f94a659c640 1 -- 172.21.15.63:0/1550164905 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9494007220 msgr2=0x7f94940096c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:26:44.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:44.437+0000 7f94a659c640 1 --2- 172.21.15.63:0/1550164905 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9494007220 0x7f94940096c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:26:44.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:44.437+0000 7f94a659c640 1 --2- 172.21.15.63:0/1550164905 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9494007220 0x7f94940096c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:26:44.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:44.437+0000 7f94a6d9d640 1 -- 172.21.15.63:0/1550164905 >> v1:172.21.15.186:6789/0 conn(0x7f9494003620 legacy=0x7f9494005a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:26:44.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:44.641+0000 7f94a659c640 1 --2- 172.21.15.63:0/1550164905 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9494007220 0x7f94940096c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:44.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:44.641+0000 7f94a659c640 10 monclient(hunting): get_auth_request con 0x7f9494007220 auth_method 0 2024-04-26T04:26:44.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:44.641+0000 7f94a659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:44.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:44.641+0000 7f94a659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:44.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:44.641+0000 7f94a659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:26:44.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:44.641+0000 7f94a659c640 1 -- 172.21.15.63:0/1550164905 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9494007220 msgr2=0x7f94940096c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T04:26:44.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:44.641+0000 7f94a659c640 1 -- 172.21.15.63:0/1550164905 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9494007220 msgr2=0x7f94940096c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:26:44.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:44.641+0000 7f94a659c640 1 --2- 172.21.15.63:0/1550164905 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9494007220 0x7f94940096c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:26:44.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:44.641+0000 7f94a659c640 1 --2- 172.21.15.63:0/1550164905 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9494007220 0x7f94940096c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T04:26:45.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:45.045+0000 7f94a659c640 1 --2- 172.21.15.63:0/1550164905 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9494007220 0x7f94940096c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:45.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:45.045+0000 7f94a659c640 10 monclient(hunting): get_auth_request con 0x7f9494007220 auth_method 0 2024-04-26T04:26:45.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:45.045+0000 7f94a659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:45.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:45.045+0000 7f94a659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:45.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:45.045+0000 7f94a659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:26:45.050 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:45.045+0000 7f94a659c640 1 -- 172.21.15.63:0/1550164905 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9494007220 msgr2=0x7f94940096c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T04:26:45.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:45.045+0000 7f94a659c640 1 -- 172.21.15.63:0/1550164905 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9494007220 msgr2=0x7f94940096c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:26:45.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:45.045+0000 7f94a659c640 1 --2- 172.21.15.63:0/1550164905 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9494007220 0x7f94940096c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:26:45.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:45.045+0000 7f94a659c640 1 --2- 172.21.15.63:0/1550164905 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9494007220 0x7f94940096c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T04:26:45.852 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:45.849+0000 7f94a659c640 1 --2- 172.21.15.63:0/1550164905 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9494007220 0x7f94940096c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:45.852 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:45.849+0000 7f94a659c640 10 monclient(hunting): get_auth_request con 0x7f9494007220 auth_method 0 2024-04-26T04:26:45.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:45.849+0000 7f94a659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:45.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:45.849+0000 7f94a659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:45.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:45.849+0000 7f94a659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:26:45.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:45.849+0000 7f94a659c640 1 -- 172.21.15.63:0/1550164905 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9494007220 msgr2=0x7f94940096c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T04:26:45.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:45.849+0000 7f94a659c640 1 -- 172.21.15.63:0/1550164905 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9494007220 msgr2=0x7f94940096c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:26:45.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:45.849+0000 7f94a659c640 1 --2- 172.21.15.63:0/1550164905 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9494007220 0x7f94940096c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:26:45.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:45.849+0000 7f94a659c640 1 --2- 172.21.15.63:0/1550164905 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9494007220 0x7f94940096c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T04:26:47.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:47.437+0000 7f94a4d99640 10 monclient(hunting): tick 2024-04-26T04:26:47.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:47.437+0000 7f94a4d99640 10 monclient(hunting): _check_auth_tickets 2024-04-26T04:26:47.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:47.437+0000 7f94a4d99640 1 monclient(hunting): continuing hunt 2024-04-26T04:26:47.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:47.437+0000 7f94a4d99640 10 monclient(hunting): _reopen_session rank -1 2024-04-26T04:26:47.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:47.437+0000 7f94a4d99640 1 -- 172.21.15.63:0/1550164905 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9494007220 msgr2=0x7f94940096c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T04:26:47.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:47.437+0000 7f94a4d99640 1 --2- 172.21.15.63:0/1550164905 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9494007220 0x7f94940096c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:47.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:47.437+0000 7f94a4d99640 1 -- 172.21.15.63:0/1550164905 >> v1:172.21.15.186:6789/0 conn(0x7f9494003620 legacy=0x7f9494005a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:47.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:47.437+0000 7f94a759e640 1 -- 172.21.15.63:0/1550164905 reap_dead start 2024-04-26T04:26:47.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:47.437+0000 7f94a4d99640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-04-26T04:26:47.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:47.437+0000 7f94a4d99640 1 --2- 172.21.15.63:0/1550164905 >> v2:172.21.15.186:3300/0 conn(0x7f9494009fa0 0x7f94940096c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:47.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:47.437+0000 7f94a4d99640 10 monclient(hunting): picked mon.noname-c con 0x7f9494009fa0 addr v2:172.21.15.186:3300/0 2024-04-26T04:26:47.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:47.437+0000 7f94a4d99640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f949400a420 addr v1:172.21.15.186:6789/0 2024-04-26T04:26:47.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:47.437+0000 7f94a4d99640 1 --2- 172.21.15.63:0/1550164905 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f949400a970 0x7f949400d260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:47.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:47.437+0000 7f94a4d99640 10 monclient(hunting): picked mon.noname-b con 0x7f949400a970 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:26:47.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:47.437+0000 7f94a4d99640 1 -- 172.21.15.63:0/1550164905 --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f949400aee0 con 0x7f949400a420 2024-04-26T04:26:47.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:47.437+0000 7f94a4d99640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:47.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:47.437+0000 7f94a4d99640 1 -- 172.21.15.63:0/1550164905 --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f94a80a35f0 con 0x7f9494009fa0 2024-04-26T04:26:47.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:47.437+0000 7f94a4d99640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:47.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:47.437+0000 7f94a4d99640 1 -- 172.21.15.63:0/1550164905 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f94a80a5380 con 0x7f949400a970 2024-04-26T04:26:47.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:47.437+0000 7f94a4d99640 10 monclient(hunting): _renew_subs 2024-04-26T04:26:47.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:47.437+0000 7f94a759e640 1 --2- 172.21.15.63:0/1550164905 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f949400a970 0x7f949400d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:47.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:47.437+0000 7f94a6d9d640 1 --2- 172.21.15.63:0/1550164905 >> v2:172.21.15.186:3300/0 conn(0x7f9494009fa0 0x7f94940096c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:47.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:47.437+0000 7f94a759e640 10 monclient(hunting): get_auth_request con 0x7f949400a970 auth_method 0 2024-04-26T04:26:47.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:47.437+0000 7f94a759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:47.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:47.437+0000 7f94a759e640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:47.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:47.437+0000 7f94a759e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:47.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:47.437+0000 7f94a6d9d640 10 monclient(hunting): get_auth_request con 0x7f9494009fa0 auth_method 0 2024-04-26T04:26:47.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:47.437+0000 7f94a6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:47.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:47.437+0000 7f94a6d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:47.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:47.437+0000 7f94a6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:47.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:47.437+0000 7f94a759e640 1 -- 172.21.15.63:0/1550164905 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f949400a970 msgr2=0x7f949400d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T04:26:47.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:47.437+0000 7f94a759e640 1 -- 172.21.15.63:0/1550164905 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f949400a970 msgr2=0x7f949400d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:26:47.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:47.437+0000 7f94a759e640 1 --2- 172.21.15.63:0/1550164905 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f949400a970 0x7f949400d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:26:47.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:47.437+0000 7f94a759e640 1 --2- 172.21.15.63:0/1550164905 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f949400a970 0x7f949400d260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:26:47.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:47.437+0000 7f94a6d9d640 1 -- 172.21.15.63:0/1550164905 >> v2:172.21.15.186:3300/0 conn(0x7f9494009fa0 msgr2=0x7f94940096c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T04:26:47.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:47.437+0000 7f94a6d9d640 1 -- 172.21.15.63:0/1550164905 >> v2:172.21.15.186:3300/0 conn(0x7f9494009fa0 msgr2=0x7f94940096c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:26:47.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:47.437+0000 7f94a6d9d640 1 --2- 172.21.15.63:0/1550164905 >> v2:172.21.15.186:3300/0 conn(0x7f9494009fa0 0x7f94940096c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:26:47.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:47.437+0000 7f94a659c640 1 -- 172.21.15.63:0/1550164905 >> v1:172.21.15.186:6789/0 conn(0x7f949400a420 legacy=0x7f9494005a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:26:47.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:47.437+0000 7f94a6d9d640 1 --2- 172.21.15.63:0/1550164905 >> v2:172.21.15.186:3300/0 conn(0x7f9494009fa0 0x7f94940096c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:26:47.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:47.641+0000 7f94a759e640 1 --2- 172.21.15.63:0/1550164905 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f949400a970 0x7f949400d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:47.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:47.641+0000 7f94a759e640 10 monclient(hunting): get_auth_request con 0x7f949400a970 auth_method 0 2024-04-26T04:26:47.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:47.641+0000 7f94a759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:47.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:47.641+0000 7f94a759e640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:47.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:47.641+0000 7f94a759e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:26:47.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:47.641+0000 7f94a759e640 1 -- 172.21.15.63:0/1550164905 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f949400a970 msgr2=0x7f949400d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T04:26:47.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:47.641+0000 7f94a759e640 1 -- 172.21.15.63:0/1550164905 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f949400a970 msgr2=0x7f949400d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:26:47.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:47.641+0000 7f94a759e640 1 --2- 172.21.15.63:0/1550164905 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f949400a970 0x7f949400d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:26:47.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:47.641+0000 7f94a759e640 1 --2- 172.21.15.63:0/1550164905 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f949400a970 0x7f949400d260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T04:26:47.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:47.641+0000 7f94a6d9d640 1 --2- 172.21.15.63:0/1550164905 >> v2:172.21.15.186:3300/0 conn(0x7f9494009fa0 0x7f94940096c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:47.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:47.641+0000 7f94a6d9d640 10 monclient(hunting): get_auth_request con 0x7f9494009fa0 auth_method 0 2024-04-26T04:26:47.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:47.641+0000 7f94a6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:47.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:47.641+0000 7f94a6d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:47.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:47.641+0000 7f94a6d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:26:47.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:47.641+0000 7f94a6d9d640 1 -- 172.21.15.63:0/1550164905 >> v2:172.21.15.186:3300/0 conn(0x7f9494009fa0 msgr2=0x7f94940096c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T04:26:47.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:47.641+0000 7f94a6d9d640 1 -- 172.21.15.63:0/1550164905 >> v2:172.21.15.186:3300/0 conn(0x7f9494009fa0 msgr2=0x7f94940096c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:26:47.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:47.641+0000 7f94a6d9d640 1 --2- 172.21.15.63:0/1550164905 >> v2:172.21.15.186:3300/0 conn(0x7f9494009fa0 0x7f94940096c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:26:47.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:47.641+0000 7f94a6d9d640 1 --2- 172.21.15.63:0/1550164905 >> v2:172.21.15.186:3300/0 conn(0x7f9494009fa0 0x7f94940096c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T04:26:48.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:48.045+0000 7f94a759e640 1 --2- 172.21.15.63:0/1550164905 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f949400a970 0x7f949400d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:48.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:48.045+0000 7f94a759e640 10 monclient(hunting): get_auth_request con 0x7f949400a970 auth_method 0 2024-04-26T04:26:48.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:48.045+0000 7f94a759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:48.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:48.045+0000 7f94a759e640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:48.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:48.045+0000 7f94a759e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:26:48.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:48.045+0000 7f94a6d9d640 1 --2- 172.21.15.63:0/1550164905 >> v2:172.21.15.186:3300/0 conn(0x7f9494009fa0 0x7f94940096c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:48.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:48.045+0000 7f94a6d9d640 10 monclient(hunting): get_auth_request con 0x7f9494009fa0 auth_method 0 2024-04-26T04:26:48.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:48.045+0000 7f94a6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:48.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:48.045+0000 7f94a6d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:48.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:48.045+0000 7f94a6d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:26:48.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:48.045+0000 7f94a759e640 1 -- 172.21.15.63:0/1550164905 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f949400a970 msgr2=0x7f949400d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T04:26:48.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:48.045+0000 7f94a759e640 1 -- 172.21.15.63:0/1550164905 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f949400a970 msgr2=0x7f949400d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:26:48.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:48.045+0000 7f94a759e640 1 --2- 172.21.15.63:0/1550164905 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f949400a970 0x7f949400d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:26:48.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:48.045+0000 7f94a6d9d640 1 -- 172.21.15.63:0/1550164905 >> v2:172.21.15.186:3300/0 conn(0x7f9494009fa0 msgr2=0x7f94940096c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T04:26:48.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:48.045+0000 7f94a759e640 1 --2- 172.21.15.63:0/1550164905 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f949400a970 0x7f949400d260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T04:26:48.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:48.045+0000 7f94a6d9d640 1 -- 172.21.15.63:0/1550164905 >> v2:172.21.15.186:3300/0 conn(0x7f9494009fa0 msgr2=0x7f94940096c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:26:48.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:48.045+0000 7f94a6d9d640 1 --2- 172.21.15.63:0/1550164905 >> v2:172.21.15.186:3300/0 conn(0x7f9494009fa0 0x7f94940096c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:26:48.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:48.045+0000 7f94a6d9d640 1 --2- 172.21.15.63:0/1550164905 >> v2:172.21.15.186:3300/0 conn(0x7f9494009fa0 0x7f94940096c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T04:26:48.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:48.849+0000 7f94a759e640 1 --2- 172.21.15.63:0/1550164905 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f949400a970 0x7f949400d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:48.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:48.849+0000 7f94a759e640 10 monclient(hunting): get_auth_request con 0x7f949400a970 auth_method 0 2024-04-26T04:26:48.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:48.849+0000 7f94a759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:48.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:48.849+0000 7f94a759e640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:48.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:48.849+0000 7f94a759e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:26:48.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:48.849+0000 7f94a6d9d640 1 --2- 172.21.15.63:0/1550164905 >> v2:172.21.15.186:3300/0 conn(0x7f9494009fa0 0x7f94940096c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:48.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:48.849+0000 7f94a6d9d640 10 monclient(hunting): get_auth_request con 0x7f9494009fa0 auth_method 0 2024-04-26T04:26:48.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:48.849+0000 7f94a6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:48.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:48.849+0000 7f94a6d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:48.855 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:48.849+0000 7f94a6d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:26:48.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:48.849+0000 7f94a759e640 1 -- 172.21.15.63:0/1550164905 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f949400a970 msgr2=0x7f949400d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-26T04:26:48.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:48.849+0000 7f94a759e640 1 -- 172.21.15.63:0/1550164905 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f949400a970 msgr2=0x7f949400d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:26:48.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:48.849+0000 7f94a759e640 1 --2- 172.21.15.63:0/1550164905 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f949400a970 0x7f949400d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:26:48.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:48.849+0000 7f94a6d9d640 1 -- 172.21.15.63:0/1550164905 >> v2:172.21.15.186:3300/0 conn(0x7f9494009fa0 msgr2=0x7f94940096c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-26T04:26:48.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:48.849+0000 7f94a6d9d640 1 -- 172.21.15.63:0/1550164905 >> v2:172.21.15.186:3300/0 conn(0x7f9494009fa0 msgr2=0x7f94940096c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:26:48.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:48.849+0000 7f94a6d9d640 1 --2- 172.21.15.63:0/1550164905 >> v2:172.21.15.186:3300/0 conn(0x7f9494009fa0 0x7f94940096c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:26:48.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:48.849+0000 7f94a759e640 1 --2- 172.21.15.63:0/1550164905 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f949400a970 0x7f949400d260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T04:26:48.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:48.849+0000 7f94a6d9d640 1 --2- 172.21.15.63:0/1550164905 >> v2:172.21.15.186:3300/0 conn(0x7f9494009fa0 0x7f94940096c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T04:26:50.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:50.441+0000 7f94a4d99640 10 monclient(hunting): tick 2024-04-26T04:26:50.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:50.441+0000 7f94a4d99640 10 monclient(hunting): _check_auth_tickets 2024-04-26T04:26:50.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:50.441+0000 7f94a4d99640 1 monclient(hunting): continuing hunt 2024-04-26T04:26:50.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:50.441+0000 7f94a4d99640 10 monclient(hunting): _reopen_session rank -1 2024-04-26T04:26:50.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:50.441+0000 7f94a4d99640 1 -- 172.21.15.63:0/1550164905 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f949400a970 msgr2=0x7f949400d260 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T04:26:50.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:50.441+0000 7f94a4d99640 1 --2- 172.21.15.63:0/1550164905 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f949400a970 0x7f949400d260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:50.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:50.441+0000 7f94a4d99640 1 -- 172.21.15.63:0/1550164905 >> v2:172.21.15.186:3300/0 conn(0x7f9494009fa0 msgr2=0x7f94940096c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T04:26:50.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:50.441+0000 7f94a4d99640 1 --2- 172.21.15.63:0/1550164905 >> v2:172.21.15.186:3300/0 conn(0x7f9494009fa0 0x7f94940096c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:50.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:50.441+0000 7f94a4d99640 1 -- 172.21.15.63:0/1550164905 >> v1:172.21.15.186:6789/0 conn(0x7f949400a420 legacy=0x7f9494005a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:50.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:50.441+0000 7f94a4d99640 10 monclient: _add_conns ranks=[0,3] 2024-04-26T04:26:50.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:50.441+0000 7f94a4d99640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f949400dc60 addr v1:172.21.15.63:6789/0 2024-04-26T04:26:50.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:50.441+0000 7f94a4d99640 1 --2- 172.21.15.63:0/1550164905 >> v2:172.21.15.63:3300/0 conn(0x7f9494011e00 0x7f94940141f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:50.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:50.441+0000 7f94a4d99640 10 monclient(hunting): picked mon.noname-a con 0x7f9494011e00 addr v2:172.21.15.63:3300/0 2024-04-26T04:26:50.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:50.441+0000 7f94a4d99640 1 -- 172.21.15.63:0/1550164905 --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f949400eb70 con 0x7f949400dc60 2024-04-26T04:26:50.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:50.441+0000 7f94a4d99640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:50.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:50.441+0000 7f94a4d99640 1 -- 172.21.15.63:0/1550164905 --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7f94a80a35f0 con 0x7f9494011e00 2024-04-26T04:26:50.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:50.441+0000 7f94a4d99640 10 monclient(hunting): _renew_subs 2024-04-26T04:26:50.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:50.441+0000 7f94a659c640 1 --2- 172.21.15.63:0/1550164905 >> v2:172.21.15.63:3300/0 conn(0x7f9494011e00 0x7f94940141f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:50.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:50.441+0000 7f94a659c640 10 monclient(hunting): get_auth_request con 0x7f9494011e00 auth_method 0 2024-04-26T04:26:50.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:50.441+0000 7f94a659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:50.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:50.441+0000 7f94a659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:50.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:50.441+0000 7f94a659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:50.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:50.441+0000 7f94a659c640 1 -- 172.21.15.63:0/1550164905 >> v2:172.21.15.63:3300/0 conn(0x7f9494011e00 msgr2=0x7f94940141f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T04:26:50.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:50.441+0000 7f94a659c640 1 -- 172.21.15.63:0/1550164905 >> v2:172.21.15.63:3300/0 conn(0x7f9494011e00 msgr2=0x7f94940141f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:26:50.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:50.441+0000 7f94a659c640 1 --2- 172.21.15.63:0/1550164905 >> v2:172.21.15.63:3300/0 conn(0x7f9494011e00 0x7f94940141f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:26:50.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:50.441+0000 7f94a659c640 1 --2- 172.21.15.63:0/1550164905 >> v2:172.21.15.63:3300/0 conn(0x7f9494011e00 0x7f94940141f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-26T04:26:50.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:50.441+0000 7f94a6d9d640 1 -- 172.21.15.63:0/1550164905 >> v1:172.21.15.63:6789/0 conn(0x7f949400dc60 legacy=0x7f949400e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:26:50.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:50.645+0000 7f94a659c640 1 --2- 172.21.15.63:0/1550164905 >> v2:172.21.15.63:3300/0 conn(0x7f9494011e00 0x7f94940141f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:50.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:50.645+0000 7f94a659c640 10 monclient(hunting): get_auth_request con 0x7f9494011e00 auth_method 0 2024-04-26T04:26:50.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:50.645+0000 7f94a659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:50.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:50.645+0000 7f94a659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:50.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:50.645+0000 7f94a659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:26:50.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:50.645+0000 7f94a659c640 1 -- 172.21.15.63:0/1550164905 >> v2:172.21.15.63:3300/0 conn(0x7f9494011e00 msgr2=0x7f94940141f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T04:26:50.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:50.645+0000 7f94a659c640 1 -- 172.21.15.63:0/1550164905 >> v2:172.21.15.63:3300/0 conn(0x7f9494011e00 msgr2=0x7f94940141f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:26:50.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:50.645+0000 7f94a659c640 1 --2- 172.21.15.63:0/1550164905 >> v2:172.21.15.63:3300/0 conn(0x7f9494011e00 0x7f94940141f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:26:50.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:50.645+0000 7f94a659c640 1 --2- 172.21.15.63:0/1550164905 >> v2:172.21.15.63:3300/0 conn(0x7f9494011e00 0x7f94940141f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-26T04:26:51.051 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:51.049+0000 7f94a659c640 1 --2- 172.21.15.63:0/1550164905 >> v2:172.21.15.63:3300/0 conn(0x7f9494011e00 0x7f94940141f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:51.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:51.049+0000 7f94a659c640 10 monclient(hunting): get_auth_request con 0x7f9494011e00 auth_method 0 2024-04-26T04:26:51.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:51.049+0000 7f94a659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:51.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:51.049+0000 7f94a659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:51.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:51.049+0000 7f94a659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:26:51.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:51.049+0000 7f94a659c640 1 -- 172.21.15.63:0/1550164905 >> v2:172.21.15.63:3300/0 conn(0x7f9494011e00 msgr2=0x7f94940141f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T04:26:51.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:51.049+0000 7f94a659c640 1 -- 172.21.15.63:0/1550164905 >> v2:172.21.15.63:3300/0 conn(0x7f9494011e00 msgr2=0x7f94940141f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:26:51.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:51.049+0000 7f94a659c640 1 --2- 172.21.15.63:0/1550164905 >> v2:172.21.15.63:3300/0 conn(0x7f9494011e00 0x7f94940141f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:26:51.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:51.049+0000 7f94a659c640 1 --2- 172.21.15.63:0/1550164905 >> v2:172.21.15.63:3300/0 conn(0x7f9494011e00 0x7f94940141f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-26T04:26:51.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:51.853+0000 7f94a659c640 1 --2- 172.21.15.63:0/1550164905 >> v2:172.21.15.63:3300/0 conn(0x7f9494011e00 0x7f94940141f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:51.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:51.853+0000 7f94a659c640 10 monclient(hunting): get_auth_request con 0x7f9494011e00 auth_method 0 2024-04-26T04:26:51.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:51.853+0000 7f94a659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:51.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:51.853+0000 7f94a659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:51.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:51.853+0000 7f94a659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:26:51.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:51.853+0000 7f94a659c640 1 -- 172.21.15.63:0/1550164905 >> v2:172.21.15.63:3300/0 conn(0x7f9494011e00 msgr2=0x7f94940141f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-26T04:26:51.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:51.853+0000 7f94a659c640 1 -- 172.21.15.63:0/1550164905 >> v2:172.21.15.63:3300/0 conn(0x7f9494011e00 msgr2=0x7f94940141f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-26T04:26:51.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:51.853+0000 7f94a659c640 1 --2- 172.21.15.63:0/1550164905 >> v2:172.21.15.63:3300/0 conn(0x7f9494011e00 0x7f94940141f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:26:51.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:51.853+0000 7f94a659c640 1 --2- 172.21.15.63:0/1550164905 >> v2:172.21.15.63:3300/0 conn(0x7f9494011e00 0x7f94940141f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-26T04:26:53.328 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.321+0000 7f94a5d9b640 1 -- 172.21.15.63:0/1550164905 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3651013257 0 0) 0x7f949400eb70 con 0x7f949400dc60 2024-04-26T04:26:53.329 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.321+0000 7f94a5d9b640 10 monclient(hunting): _negotiate 2024-04-26T04:26:53.329 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.321+0000 7f94a5d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:53.329 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.321+0000 7f94a5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:53.329 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.321+0000 7f94a5d9b640 10 monclient(hunting): my global_id is 44097 2024-04-26T04:26:53.329 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.321+0000 7f94a5d9b640 1 -- 172.21.15.63:0/1550164905 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9490003530 con 0x7f949400dc60 2024-04-26T04:26:53.330 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94a5d9b640 1 -- 172.21.15.63:0/1550164905 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3305227517 0 0) 0x7f9490003530 con 0x7f949400dc60 2024-04-26T04:26:53.330 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94a5d9b640 1 -- 172.21.15.63:0/1550164905 >> v2:172.21.15.63:3300/0 conn(0x7f9494011e00 msgr2=0x7f94940141f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-26T04:26:53.331 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94a5d9b640 1 --2- 172.21.15.63:0/1550164905 >> v2:172.21.15.63:3300/0 conn(0x7f9494011e00 0x7f94940141f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:53.331 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94a5d9b640 10 monclient: _finish_hunting 0 2024-04-26T04:26:53.331 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94a5d9b640 1 monclient: found mon.noname-a-legacy 2024-04-26T04:26:53.331 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94a5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:26:53.331 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94a5d9b640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.63:6789/0 2024-04-26T04:26:53.331 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94a5d9b640 1 -- 172.21.15.63:0/1550164905 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94940157b0 con 0x7f949400dc60 2024-04-26T04:26:53.331 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94a5d9b640 10 monclient: _finish_auth 0 2024-04-26T04:26:53.331 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94a5d9b640 10 monclient: _check_auth_tickets 2024-04-26T04:26:53.331 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94a5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:26:53.331 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94a5d9b640 1 -- 172.21.15.63:0/1550164905 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2612289768 0 0) 0x7f949800f040 con 0x7f949400dc60 2024-04-26T04:26:53.331 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94a5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:53.331 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94a5d9b640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-26T04:26:53.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94a5d9b640 10 monclient: dump: 2024-04-26T04:26:53.332 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:53.332 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:53.332 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:53.332 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:53.332 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:53.332 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:53.332 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:53.332 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:53.332 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:53.332 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:53.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94a5d9b640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] but i'm connected to v1:172.21.15.63:6789/0, reconnecting 2024-04-26T04:26:53.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94a5d9b640 10 monclient: _reopen_session rank -1 2024-04-26T04:26:53.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94a5d9b640 1 -- 172.21.15.63:0/1550164905 >> v1:172.21.15.63:6789/0 conn(0x7f949400dc60 legacy=0x7f949400e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:53.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94a759e640 1 -- 172.21.15.63:0/1550164905 reap_dead start 2024-04-26T04:26:53.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94a5d9b640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T04:26:53.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94a5d9b640 1 --2- 172.21.15.63:0/1550164905 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9490003aa0 0x7f9490003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:53.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94a5d9b640 10 monclient(hunting): picked mon.a con 0x7f9490003aa0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:26:53.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94a5d9b640 1 --2- 172.21.15.63:0/1550164905 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9490004810 0x7f9490004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:53.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94a5d9b640 10 monclient(hunting): picked mon.c con 0x7f9490004810 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:26:53.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94a5d9b640 1 --2- 172.21.15.63:0/1550164905 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f94900053d0 0x7f949000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:53.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94a5d9b640 10 monclient(hunting): picked mon.b con 0x7f94900053d0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:26:53.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94a5d9b640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:53.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94a5d9b640 1 -- 172.21.15.63:0/1550164905 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f94a80a35f0 con 0x7f9490003aa0 2024-04-26T04:26:53.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94a5d9b640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:53.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94a5d9b640 1 -- 172.21.15.63:0/1550164905 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f949000c270 con 0x7f94900053d0 2024-04-26T04:26:53.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94a5d9b640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:53.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94a5d9b640 1 -- 172.21.15.63:0/1550164905 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f949000c470 con 0x7f9490004810 2024-04-26T04:26:53.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94a5d9b640 10 monclient(hunting): _renew_subs 2024-04-26T04:26:53.334 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94a5d9b640 10 monclient(hunting): _finish_auth 0 2024-04-26T04:26:53.335 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94a6d9d640 1 --2- 172.21.15.63:0/1550164905 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9490003aa0 0x7f9490003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:53.335 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94a759e640 1 --2- 172.21.15.63:0/1550164905 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f94900053d0 0x7f949000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:53.335 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94a6d9d640 10 monclient(hunting): get_auth_request con 0x7f9490003aa0 auth_method 0 2024-04-26T04:26:53.335 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94a6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:53.335 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94a6d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:53.335 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94a6d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:26:53.335 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94a759e640 10 monclient(hunting): get_auth_request con 0x7f94900053d0 auth_method 0 2024-04-26T04:26:53.335 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94a759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:53.335 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94a759e640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:53.335 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94a759e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:26:53.335 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94a6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:26:53.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94a6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:26:53.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94a6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T04:26:53.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94a759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:26:53.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94a759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:26:53.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94a759e640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T04:26:53.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94a659c640 1 --2- 172.21.15.63:0/1550164905 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9490004810 0x7f9490004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:53.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94a659c640 10 monclient(hunting): get_auth_request con 0x7f9490004810 auth_method 0 2024-04-26T04:26:53.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94a659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:53.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94a659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:53.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94a659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:26:53.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94a759e640 10 monclient(hunting): handle_auth_done global_id 44097 payload 293 2024-04-26T04:26:53.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94a759e640 1 -- 172.21.15.63:0/1550164905 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9490004810 msgr2=0x7f9490004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:26:53.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94a759e640 1 --2- 172.21.15.63:0/1550164905 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9490004810 0x7f9490004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:53.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94a759e640 1 -- 172.21.15.63:0/1550164905 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9490003aa0 msgr2=0x7f9490003ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:26:53.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94a759e640 1 --2- 172.21.15.63:0/1550164905 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9490003aa0 0x7f9490003ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:53.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94a759e640 10 monclient: _finish_hunting 0 2024-04-26T04:26:53.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94a759e640 1 monclient: found mon.b 2024-04-26T04:26:53.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94a759e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:26:53.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94a759e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.186:3300/0 2024-04-26T04:26:53.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94a759e640 1 -- 172.21.15.63:0/1550164905 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9490003530 con 0x7f94900053d0 2024-04-26T04:26:53.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94a759e640 10 monclient: _finish_auth 0 2024-04-26T04:26:53.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94a759e640 10 monclient: _check_auth_tickets 2024-04-26T04:26:53.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94a759e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:26:53.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94a6d9d640 1 --2- 172.21.15.63:0/1550164905 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9490003aa0 0x7f9490003ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T04:26:53.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94ad254640 5 monclient: authenticate success, global_id 44097 2024-04-26T04:26:53.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.325+0000 7f94ad254640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T04:26:53.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.333+0000 7f94a759e640 1 -- 172.21.15.63:0/1550164905 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f94900053d0 msgr2=0x7f949000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:26:53.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.333+0000 7f94a759e640 1 --2- 172.21.15.63:0/1550164905 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f94900053d0 0x7f949000b9d0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f94a80a4fe0 tx=0x7f94a001dab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c387a0cfec820388 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:26:53.339 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.333+0000 7f94a5d9b640 1 -- 172.21.15.63:0/1550164905 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94a000f040 con 0x7f94900053d0 2024-04-26T04:26:53.339 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.333+0000 7f94a5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:53.339 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.333+0000 7f94a5d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T04:26:53.339 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.333+0000 7f94a5d9b640 10 monclient: dump: 2024-04-26T04:26:53.339 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:53.339 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:53.339 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:53.339 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:53.339 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:53.339 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:53.339 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:53.339 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:53.340 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:53.340 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:53.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.333+0000 7f94a5d9b640 1 -- 172.21.15.63:0/1550164905 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94a00af070 con 0x7f94900053d0 2024-04-26T04:26:53.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.333+0000 7f94ad254640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T04:26:53.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.333+0000 7f94a5d9b640 10 monclient: handle_config config(0 keys) 2024-04-26T04:26:53.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.333+0000 7f94ad254640 10 monclient: get_monmap_and_config success 2024-04-26T04:26:53.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.333+0000 7f94a5d9b640 1 -- 172.21.15.63:0/1550164905 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94a00a7940 con 0x7f94900053d0 2024-04-26T04:26:53.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.333+0000 7f94ad254640 4 set_mon_vals no callback set 2024-04-26T04:26:53.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.333+0000 7f94a5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:53.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.333+0000 7f94a5d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T04:26:53.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.333+0000 7f94a5d9b640 10 monclient: dump: 2024-04-26T04:26:53.340 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:53.340 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:53.340 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:53.341 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:53.341 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:53.341 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:53.341 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:53.341 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:53.341 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:53.341 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:53.344 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.333+0000 7f94ad254640 10 monclient: shutdown 2024-04-26T04:26:53.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.333+0000 7f94ad254640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:26:53.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.333+0000 7f94ad254640 1 -- 172.21.15.63:0/1550164905 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f94900053d0 msgr2=0x7f949000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:53.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.333+0000 7f94ad254640 1 --2- 172.21.15.63:0/1550164905 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f94900053d0 0x7f949000b9d0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f94a80a4fe0 tx=0x7f94a001dab0 comp rx=0 tx=0).stop 2024-04-26T04:26:53.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.333+0000 7f94ad254640 1 -- 172.21.15.63:0/1550164905 shutdown_connections 2024-04-26T04:26:53.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.333+0000 7f94ad254640 1 --2- 172.21.15.63:0/1550164905 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f94900053d0 0x7f949000b9d0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:53.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.333+0000 7f94ad254640 1 --2- 172.21.15.63:0/1550164905 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9490004810 0x7f9490004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:53.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.333+0000 7f94ad254640 1 --2- 172.21.15.63:0/1550164905 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9490003aa0 0x7f9490003ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:53.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.333+0000 7f94ad254640 1 -- 172.21.15.63:0/1550164905 >> 172.21.15.63:0/1550164905 conn(0x7f94a80a5f40 msgr2=0x7f94a80a6320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:26:53.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.333+0000 7f94ad254640 1 -- 172.21.15.63:0/1550164905 shutdown_connections 2024-04-26T04:26:53.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.333+0000 7f94ad254640 1 -- 172.21.15.63:0/1550164905 wait complete. 2024-04-26T04:26:53.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.333+0000 7f94ad254640 10 monclient: build_initial_monmap 2024-04-26T04:26:53.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.333+0000 7f94ad254640 10 monclient: monmap: 2024-04-26T04:26:53.345 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:26:53.346 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:26:53.346 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:26:53.346 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:26:53.346 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:26:53.346 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:53.346 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:26:53.346 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:26:53.346 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:26:53.346 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:53.346 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.337+0000 7f94ad254640 1 Processor -- start 2024-04-26T04:26:53.346 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.337+0000 7f94ad254640 1 -- start start 2024-04-26T04:26:53.346 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.337+0000 7f94ad254640 10 monclient: init 2024-04-26T04:26:53.346 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.337+0000 7f94ad254640 10 monclient: _reopen_session rank -1 2024-04-26T04:26:53.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.337+0000 7f94ad254640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T04:26:53.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.337+0000 7f94ad254640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9490003aa0 0x7f94a811fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:53.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.337+0000 7f94ad254640 10 monclient(hunting): picked mon.noname-b con 0x7f9490003aa0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:26:53.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.337+0000 7f94ad254640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9490004810 0x7f94a811ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:53.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.337+0000 7f94ad254640 10 monclient(hunting): picked mon.noname-a con 0x7f9490004810 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:26:53.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.337+0000 7f94ad254640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f94900053d0 0x7f94a8120510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:53.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.337+0000 7f94ad254640 10 monclient(hunting): picked mon.noname-c con 0x7f94900053d0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:26:53.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.337+0000 7f94ad254640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:53.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.337+0000 7f94ad254640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f94a80a3230 con 0x7f9490004810 2024-04-26T04:26:53.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.337+0000 7f94ad254640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:53.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.337+0000 7f94ad254640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f94a80a30b0 con 0x7f9490003aa0 2024-04-26T04:26:53.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.337+0000 7f94ad254640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:53.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.337+0000 7f94ad254640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f94a80a33b0 con 0x7f94900053d0 2024-04-26T04:26:53.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.337+0000 7f94ad254640 10 monclient(hunting): _renew_subs 2024-04-26T04:26:53.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.337+0000 7f94ad254640 10 monclient(hunting): authenticate will time out at 1445.049407s 2024-04-26T04:26:53.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.337+0000 7f94a659c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9490004810 0x7f94a811ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:53.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.337+0000 7f94a659c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9490004810 0x7f94a811ffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:55184/0 (socket says 172.21.15.63:55184) 2024-04-26T04:26:53.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.337+0000 7f94a759e640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f94900053d0 0x7f94a8120510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:53.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.337+0000 7f94a659c640 1 -- 172.21.15.63:0/1661173790 learned_addr learned my addr 172.21.15.63:0/1661173790 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:26:53.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.337+0000 7f94a659c640 10 monclient(hunting): get_auth_request con 0x7f9490004810 auth_method 0 2024-04-26T04:26:53.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.337+0000 7f94a659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:53.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.337+0000 7f94a659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:53.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.337+0000 7f94a659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:53.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.337+0000 7f94a759e640 10 monclient(hunting): get_auth_request con 0x7f94900053d0 auth_method 0 2024-04-26T04:26:53.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.337+0000 7f94a759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:53.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.337+0000 7f94a759e640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:53.349 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.337+0000 7f94a759e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:53.349 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.337+0000 7f94a6d9d640 1 --2- 172.21.15.63:0/1661173790 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9490003aa0 0x7f94a811fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:53.349 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.337+0000 7f94a659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:26:53.349 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.337+0000 7f94a659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:26:53.349 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.337+0000 7f94a659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:26:53.349 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.337+0000 7f94a6d9d640 10 monclient(hunting): get_auth_request con 0x7f9490003aa0 auth_method 0 2024-04-26T04:26:53.349 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.337+0000 7f94a6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:53.349 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.337+0000 7f94a6d9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:53.349 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.337+0000 7f94a6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:53.349 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.337+0000 7f94a759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:26:53.349 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.337+0000 7f94a759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:26:53.349 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.337+0000 7f94a759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:26:53.349 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.337+0000 7f94a6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:26:53.349 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.337+0000 7f94a6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:26:53.349 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.337+0000 7f94a6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:26:53.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.337+0000 7f94a759e640 10 monclient(hunting): handle_auth_done global_id 24668 payload 824 2024-04-26T04:26:53.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.337+0000 7f94a759e640 1 -- 172.21.15.63:0/1661173790 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9490003aa0 msgr2=0x7f94a811fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:53.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.337+0000 7f94a759e640 1 --2- 172.21.15.63:0/1661173790 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9490003aa0 0x7f94a811fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:53.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.337+0000 7f94a759e640 1 -- 172.21.15.63:0/1661173790 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9490004810 msgr2=0x7f94a811ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:53.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.337+0000 7f94a759e640 1 --2- 172.21.15.63:0/1661173790 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9490004810 0x7f94a811ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:53.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.337+0000 7f94a759e640 10 monclient: _finish_hunting 0 2024-04-26T04:26:53.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.337+0000 7f94a759e640 1 monclient: found mon.noname-c 2024-04-26T04:26:53.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.337+0000 7f94a759e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:26:53.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.337+0000 7f94a759e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.63:3301/0 2024-04-26T04:26:53.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.337+0000 7f94a759e640 1 -- 172.21.15.63:0/1661173790 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94a8122cd0 con 0x7f94900053d0 2024-04-26T04:26:53.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.337+0000 7f94a659c640 1 --2- 172.21.15.63:0/1661173790 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9490004810 0x7f94a811ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T04:26:53.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.341+0000 7f94a759e640 1 -- 172.21.15.63:0/1661173790 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f94900053d0 msgr2=0x7f94a8120510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:26:53.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.341+0000 7f94a759e640 1 --2- 172.21.15.63:0/1661173790 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f94900053d0 0x7f94a8120510 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f949000c440 tx=0x7f94a001b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:26:53.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.341+0000 7f948f7fe640 1 -- 172.21.15.63:0/1661173790 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94a000f040 con 0x7f94900053d0 2024-04-26T04:26:53.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.341+0000 7f948f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:53.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.341+0000 7f948f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:26:53.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.341+0000 7f948f7fe640 10 monclient: dump: 2024-04-26T04:26:53.353 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:53.353 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:53.353 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:53.353 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:53.353 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:53.353 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:53.353 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:53.353 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:53.353 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:53.353 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:53.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.341+0000 7f948f7fe640 10 monclient: _finish_auth 0 2024-04-26T04:26:53.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.341+0000 7f948f7fe640 10 monclient: _check_auth_tickets 2024-04-26T04:26:53.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.341+0000 7f948f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:26:53.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.341+0000 7f948f7fe640 1 -- 172.21.15.63:0/1661173790 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94a00a7ca0 con 0x7f94900053d0 2024-04-26T04:26:53.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.341+0000 7f948f7fe640 10 monclient: handle_config config(0 keys) 2024-04-26T04:26:53.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.341+0000 7f948f7fe640 1 -- 172.21.15.63:0/1661173790 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94a0006be0 con 0x7f94900053d0 2024-04-26T04:26:53.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.341+0000 7f948f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:53.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.341+0000 7f948f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:26:53.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.341+0000 7f948f7fe640 10 monclient: dump: 2024-04-26T04:26:53.354 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:53.354 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:53.354 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:53.354 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:53.354 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:53.355 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:53.355 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:53.355 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:53.355 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:53.355 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:53.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.341+0000 7f94a559a640 4 set_mon_vals no callback set 2024-04-26T04:26:53.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.341+0000 7f94ad254640 5 monclient: authenticate success, global_id 24668 2024-04-26T04:26:53.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.341+0000 7f94ad254640 10 monclient: _renew_subs 2024-04-26T04:26:53.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.341+0000 7f94ad254640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:26:53.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.341+0000 7f94ad254640 1 -- 172.21.15.63:0/1661173790 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94a8121b00 con 0x7f94900053d0 2024-04-26T04:26:53.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.341+0000 7f94ad254640 10 monclient: _renew_subs 2024-04-26T04:26:53.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.341+0000 7f94ad254640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:26:53.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.341+0000 7f94ad254640 1 -- 172.21.15.63:0/1661173790 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f94a8123c90 con 0x7f94900053d0 2024-04-26T04:26:53.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.341+0000 7f94ad254640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:26:53.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.341+0000 7f94ad254640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:26:53.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.341+0000 7f94ad254640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:26:53.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.341+0000 7f948f7fe640 1 -- 172.21.15.63:0/1661173790 <== mon.2 v2:172.21.15.63:3301/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f94a00bc020 con 0x7f94900053d0 2024-04-26T04:26:53.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.341+0000 7f94ad254640 1 -- 172.21.15.63:0/1661173790 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94a8062e10 con 0x7f94900053d0 2024-04-26T04:26:53.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.341+0000 7f948f7fe640 1 --2- 172.21.15.63:0/1661173790 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f949403b450 0x7f949403d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:53.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.341+0000 7f948f7fe640 1 -- 172.21.15.63:0/1661173790 <== mon.2 v2:172.21.15.63:3301/0 5 ==== osd_map(323..323 src has 1..323) ==== 4346+0+0 (secure 0 0 0) 0x7f94a00e3370 con 0x7f94900053d0 2024-04-26T04:26:53.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.341+0000 7f94a6d9d640 1 --2- 172.21.15.63:0/1661173790 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f949403b450 0x7f949403d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:53.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.341+0000 7f94a6d9d640 10 monclient: get_auth_request con 0x7f949403b450 auth_method 0 2024-04-26T04:26:53.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.345+0000 7f94a6d9d640 1 --2- 172.21.15.63:0/1661173790 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f949403b450 0x7f949403d910 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f949800e590 tx=0x7f9498008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:26:53.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.349+0000 7f948f7fe640 1 -- 172.21.15.63:0/1661173790 <== mon.2 v2:172.21.15.63:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f94a00c5070 con 0x7f94900053d0 2024-04-26T04:26:53.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.349+0000 7f948f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:26:53.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.349+0000 7f948f7fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:26:53.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.577+0000 7f94ad254640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T04:26:53.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.577+0000 7f94ad254640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T04:26:53.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.577+0000 7f94ad254640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:26:53.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.577+0000 7f94ad254640 1 -- 172.21.15.63:0/1661173790 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f94a80683d0 con 0x7f94900053d0 2024-04-26T04:26:53.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.577+0000 7f948f7fe640 1 -- 172.21.15.63:0/1661173790 <== mon.2 v2:172.21.15.63:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f94a00e3030 con 0x7f94900053d0 2024-04-26T04:26:53.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.577+0000 7f948f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T04:26:53.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.577+0000 7f948f7fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:26:53.583 INFO:teuthology.orchestra.run.smithi063.stdout:{"election_epoch":42,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cd1c6a0c-4e4b-4014-89c0-2e535479ef39","modified":"2024-04-26T04:18:03.702339Z","created":"2024-04-26T04:18:03.702339Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3300","nonce":0},{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:3300","nonce":0},{"type":"v1","addr":"172.21.15.186:6789","nonce":0}]},"addr":"172.21.15.186:6789/0","public_addr":"172.21.15.186:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3301","nonce":0},{"type":"v1","addr":"172.21.15.63:6790","nonce":0}]},"addr":"172.21.15.63:6790/0","public_addr":"172.21.15.63:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T04:26:53.585 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.577+0000 7f94ad254640 1 -- 172.21.15.63:0/1661173790 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f949403b450 msgr2=0x7f949403d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:53.585 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.577+0000 7f94ad254640 1 --2- 172.21.15.63:0/1661173790 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f949403b450 0x7f949403d910 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f949800e590 tx=0x7f9498008040 comp rx=0 tx=0).stop 2024-04-26T04:26:53.585 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.577+0000 7f94ad254640 10 monclient: shutdown 2024-04-26T04:26:53.585 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.577+0000 7f94ad254640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:26:53.586 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.577+0000 7f94ad254640 1 -- 172.21.15.63:0/1661173790 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f94900053d0 msgr2=0x7f94a8120510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:53.586 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.577+0000 7f94ad254640 1 --2- 172.21.15.63:0/1661173790 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f94900053d0 0x7f94a8120510 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f949000c440 tx=0x7f94a001b7a0 comp rx=0 tx=0).stop 2024-04-26T04:26:53.586 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.577+0000 7f94ad254640 1 -- 172.21.15.63:0/1661173790 shutdown_connections 2024-04-26T04:26:53.586 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.577+0000 7f94ad254640 1 --2- 172.21.15.63:0/1661173790 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f949403b450 0x7f949403d910 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:53.586 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.577+0000 7f94ad254640 1 --2- 172.21.15.63:0/1661173790 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f94900053d0 0x7f94a8120510 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:53.586 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.577+0000 7f94ad254640 1 --2- 172.21.15.63:0/1661173790 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9490004810 0x7f94a811ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:53.586 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.577+0000 7f94ad254640 1 --2- 172.21.15.63:0/1661173790 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9490003aa0 0x7f94a811fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:53.586 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.577+0000 7f94ad254640 1 -- 172.21.15.63:0/1661173790 >> 172.21.15.63:0/1661173790 conn(0x7f94a80a5f40 msgr2=0x7f94a80a1a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:26:53.587 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.577+0000 7f94ad254640 1 -- 172.21.15.63:0/1661173790 shutdown_connections 2024-04-26T04:26:53.587 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.577+0000 7f94ad254640 1 -- 172.21.15.63:0/1661173790 wait complete. 2024-04-26T04:26:53.601 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-26T04:26:53.601 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-04-26T04:26:53.602 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-04-26T04:26:53.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.713+0000 7f9f39e32640 10 monclient: get_monmap_and_config 2024-04-26T04:26:53.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.713+0000 7f9f39e32640 10 monclient: build_initial_monmap 2024-04-26T04:26:53.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.713+0000 7f9f39e32640 10 monclient: monmap: 2024-04-26T04:26:53.720 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:26:53.720 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:53.720 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:26:53.720084+0000 2024-04-26T04:26:53.720 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:26:53.720084+0000 2024-04-26T04:26:53.720 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:26:53.720 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:53.720 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:26:53.720 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:26:53.720 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:26:53.721 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:26:53.721 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:26:53.721 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:53.723 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.713+0000 7f9f39e32640 1 Processor -- start 2024-04-26T04:26:53.723 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.713+0000 7f9f39e32640 1 -- start start 2024-04-26T04:26:53.723 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.713+0000 7f9f39e32640 10 monclient: init 2024-04-26T04:26:53.723 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.717+0000 7f9f39e32640 10 monclient: _reopen_session rank -1 2024-04-26T04:26:53.723 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.717+0000 7f9f39e32640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-04-26T04:26:53.723 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.717+0000 7f9f39e32640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f9f3409fe60 0x7f9f341372f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:53.723 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.717+0000 7f9f39e32640 10 monclient(hunting): picked mon.noname-a con 0x7f9f3409fe60 addr v2:172.21.15.63:3300/0 2024-04-26T04:26:53.723 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.717+0000 7f9f39e32640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9f340a0240 addr v1:172.21.15.186:6789/0 2024-04-26T04:26:53.723 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.717+0000 7f9f39e32640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9f3413c100 0x7f9f3413e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:53.723 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.717+0000 7f9f39e32640 10 monclient(hunting): picked mon.noname-b con 0x7f9f3413c100 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:26:53.724 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.717+0000 7f9f39e32640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f34137830 con 0x7f9f340a0240 2024-04-26T04:26:53.724 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.717+0000 7f9f39e32640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:53.724 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.717+0000 7f9f39e32640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7f9f340a3730 con 0x7f9f3409fe60 2024-04-26T04:26:53.724 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.717+0000 7f9f39e32640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:53.724 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.717+0000 7f9f39e32640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f9f340a0c60 con 0x7f9f3413c100 2024-04-26T04:26:53.724 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.717+0000 7f9f39e32640 10 monclient(hunting): _renew_subs 2024-04-26T04:26:53.724 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.717+0000 7f9f39e32640 10 monclient(hunting): authenticate will time out at 1445.428536s 2024-04-26T04:26:53.724 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.717+0000 7f9f337fe640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f9f3409fe60 0x7f9f341372f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:53.724 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.717+0000 7f9f33fff640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9f3413c100 0x7f9f3413e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:53.724 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.717+0000 7f9f33fff640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9f3413c100 0x7f9f3413e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:40974/0 (socket says 172.21.15.63:40974) 2024-04-26T04:26:53.724 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.717+0000 7f9f337fe640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f9f3409fe60 0x7f9f341372f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:55188/0 (socket says 172.21.15.63:55188) 2024-04-26T04:26:53.724 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.717+0000 7f9f33fff640 1 -- 172.21.15.63:0/4225341330 learned_addr learned my addr 172.21.15.63:0/4225341330 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:26:53.724 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.717+0000 7f9f33fff640 10 monclient(hunting): get_auth_request con 0x7f9f3413c100 auth_method 0 2024-04-26T04:26:53.724 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.717+0000 7f9f33fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:53.724 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.717+0000 7f9f33fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:53.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.717+0000 7f9f33fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:53.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.717+0000 7f9f337fe640 10 monclient(hunting): get_auth_request con 0x7f9f3409fe60 auth_method 0 2024-04-26T04:26:53.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.717+0000 7f9f337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:53.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.717+0000 7f9f337fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:53.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.717+0000 7f9f337fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:53.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.717+0000 7f9f33fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:26:53.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.717+0000 7f9f33fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:26:53.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.717+0000 7f9f33fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:26:53.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.717+0000 7f9f337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:26:53.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.717+0000 7f9f337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:26:53.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.717+0000 7f9f337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:26:53.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.717+0000 7f9f33fff640 10 monclient(hunting): handle_auth_done global_id 24671 payload 274 2024-04-26T04:26:53.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.717+0000 7f9f33fff640 1 -- 172.21.15.63:0/4225341330 >> v2:172.21.15.63:3300/0 conn(0x7f9f3409fe60 msgr2=0x7f9f341372f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:26:53.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.717+0000 7f9f33fff640 1 --2- 172.21.15.63:0/4225341330 >> v2:172.21.15.63:3300/0 conn(0x7f9f3409fe60 0x7f9f341372f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:53.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.717+0000 7f9f33fff640 1 -- 172.21.15.63:0/4225341330 >> v1:172.21.15.186:6789/0 conn(0x7f9f340a0240 legacy=0x7f9f340a2a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:26:53.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.717+0000 7f9f33fff640 10 monclient: _finish_hunting 0 2024-04-26T04:26:53.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.717+0000 7f9f33fff640 1 monclient: found mon.noname-b 2024-04-26T04:26:53.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.717+0000 7f9f33fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:26:53.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.717+0000 7f9f33fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.63:3301/0 2024-04-26T04:26:53.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.717+0000 7f9f33fff640 1 -- 172.21.15.63:0/4225341330 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f340a0f40 con 0x7f9f3413c100 2024-04-26T04:26:53.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.721+0000 7f9f33fff640 1 -- 172.21.15.63:0/4225341330 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9f3413c100 msgr2=0x7f9f3413e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:26:53.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.721+0000 7f9f33fff640 1 --2- 172.21.15.63:0/4225341330 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9f3413c100 0x7f9f3413e4f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f9f2c009e50 tx=0x7f9f2c01c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86d169c8a1ac834a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:26:53.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.721+0000 7f9f32ffd640 1 -- 172.21.15.63:0/4225341330 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f2c0a5720 con 0x7f9f3413c100 2024-04-26T04:26:53.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.721+0000 7f9f32ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:53.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.721+0000 7f9f32ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T04:26:53.728 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.721+0000 7f9f32ffd640 10 monclient: dump: 2024-04-26T04:26:53.728 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:53.728 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:53.729 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:53.729 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:53.729 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:53.729 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:53.729 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:53.729 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:53.729 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:53.729 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:53.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.721+0000 7f9f32ffd640 10 monclient: _finish_auth 0 2024-04-26T04:26:53.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.721+0000 7f9f32ffd640 10 monclient: _check_auth_tickets 2024-04-26T04:26:53.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.721+0000 7f9f32ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:26:53.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.721+0000 7f9f32ffd640 1 -- 172.21.15.63:0/4225341330 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f2c0a58c0 con 0x7f9f3413c100 2024-04-26T04:26:53.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.721+0000 7f9f39e32640 5 monclient: authenticate success, global_id 24671 2024-04-26T04:26:53.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.721+0000 7f9f39e32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T04:26:53.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.721+0000 7f9f32ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T04:26:53.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.721+0000 7f9f39e32640 10 monclient: get_monmap_and_config success 2024-04-26T04:26:53.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.721+0000 7f9f32ffd640 1 -- 172.21.15.63:0/4225341330 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f2c0a5c90 con 0x7f9f3413c100 2024-04-26T04:26:53.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.721+0000 7f9f32ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:53.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.721+0000 7f9f39e32640 4 set_mon_vals no callback set 2024-04-26T04:26:53.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.721+0000 7f9f32ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:26:53.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.721+0000 7f9f32ffd640 10 monclient: dump: 2024-04-26T04:26:53.730 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:53.730 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:53.730 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:53.730 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:53.730 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:53.730 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:53.730 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:53.730 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:53.731 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:53.731 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:53.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.721+0000 7f9f39e32640 10 monclient: shutdown 2024-04-26T04:26:53.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.721+0000 7f9f39e32640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:26:53.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.721+0000 7f9f39e32640 1 -- 172.21.15.63:0/4225341330 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9f3413c100 msgr2=0x7f9f3413e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:53.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.721+0000 7f9f39e32640 1 --2- 172.21.15.63:0/4225341330 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9f3413c100 0x7f9f3413e4f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f9f2c009e50 tx=0x7f9f2c01c880 comp rx=0 tx=0).stop 2024-04-26T04:26:53.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.721+0000 7f9f39e32640 1 -- 172.21.15.63:0/4225341330 shutdown_connections 2024-04-26T04:26:53.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.721+0000 7f9f39e32640 1 --2- 172.21.15.63:0/4225341330 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9f3413c100 0x7f9f3413e4f0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:53.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.721+0000 7f9f39e32640 1 --2- 172.21.15.63:0/4225341330 >> v2:172.21.15.63:3300/0 conn(0x7f9f3409fe60 0x7f9f341372f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:53.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.721+0000 7f9f39e32640 1 -- 172.21.15.63:0/4225341330 >> 172.21.15.63:0/4225341330 conn(0x7f9f340a5fb0 msgr2=0x7f9f340a63c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:26:53.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.721+0000 7f9f39e32640 1 -- 172.21.15.63:0/4225341330 shutdown_connections 2024-04-26T04:26:53.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.721+0000 7f9f39e32640 1 -- 172.21.15.63:0/4225341330 wait complete. 2024-04-26T04:26:53.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.721+0000 7f9f39e32640 10 monclient: build_initial_monmap 2024-04-26T04:26:53.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.721+0000 7f9f39e32640 10 monclient: monmap: 2024-04-26T04:26:53.732 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:26:53.732 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:26:53.732 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:26:53.732 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:26:53.732 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:26:53.732 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:53.732 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:26:53.732 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:26:53.732 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:26:53.732 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:53.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.721+0000 7f9f39e32640 1 Processor -- start 2024-04-26T04:26:53.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.721+0000 7f9f39e32640 1 -- start start 2024-04-26T04:26:53.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.721+0000 7f9f39e32640 10 monclient: init 2024-04-26T04:26:53.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.721+0000 7f9f39e32640 10 monclient: _reopen_session rank -1 2024-04-26T04:26:53.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.721+0000 7f9f39e32640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T04:26:53.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.721+0000 7f9f39e32640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9f3409fe60 0x7f9f34072350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:53.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.721+0000 7f9f39e32640 10 monclient(hunting): picked mon.noname-a con 0x7f9f3409fe60 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:26:53.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.721+0000 7f9f39e32640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9f340a0240 0x7f9f34072890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:53.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.721+0000 7f9f39e32640 10 monclient(hunting): picked mon.noname-c con 0x7f9f340a0240 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:26:53.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.721+0000 7f9f39e32640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f3413c100 0x7f9f3406e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:53.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.721+0000 7f9f39e32640 10 monclient(hunting): picked mon.noname-b con 0x7f9f3413c100 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:26:53.738 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.721+0000 7f9f39e32640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:53.738 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.721+0000 7f9f39e32640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f9f341416a0 con 0x7f9f3409fe60 2024-04-26T04:26:53.738 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.721+0000 7f9f39e32640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:53.738 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.721+0000 7f9f39e32640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9f34141520 con 0x7f9f3413c100 2024-04-26T04:26:53.738 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.721+0000 7f9f39e32640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:53.738 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.721+0000 7f9f39e32640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f9f34141820 con 0x7f9f340a0240 2024-04-26T04:26:53.738 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.721+0000 7f9f39e32640 10 monclient(hunting): _renew_subs 2024-04-26T04:26:53.738 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.721+0000 7f9f39e32640 10 monclient(hunting): authenticate will time out at 1445.435539s 2024-04-26T04:26:53.738 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.721+0000 7f9f337fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9f3409fe60 0x7f9f34072350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:53.738 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.721+0000 7f9f2affd640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9f340a0240 0x7f9f34072890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:53.738 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.721+0000 7f9f337fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9f3409fe60 0x7f9f34072350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:55202/0 (socket says 172.21.15.63:55202) 2024-04-26T04:26:53.738 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.721+0000 7f9f337fe640 1 -- 172.21.15.63:0/34829886 learned_addr learned my addr 172.21.15.63:0/34829886 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:26:53.738 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.721+0000 7f9f337fe640 10 monclient(hunting): get_auth_request con 0x7f9f3409fe60 auth_method 0 2024-04-26T04:26:53.739 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.721+0000 7f9f337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:53.739 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.721+0000 7f9f337fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:53.739 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.721+0000 7f9f337fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:53.739 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.721+0000 7f9f2affd640 10 monclient(hunting): get_auth_request con 0x7f9f340a0240 auth_method 0 2024-04-26T04:26:53.739 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.721+0000 7f9f2affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:53.739 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.721+0000 7f9f2affd640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:53.739 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.721+0000 7f9f2affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:53.739 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.721+0000 7f9f337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:26:53.739 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.721+0000 7f9f337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:26:53.739 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.721+0000 7f9f337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:26:53.739 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.721+0000 7f9f2affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:26:53.739 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.721+0000 7f9f2affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:26:53.739 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.721+0000 7f9f2affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:26:53.739 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.721+0000 7f9f33fff640 1 --2- 172.21.15.63:0/34829886 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f3413c100 0x7f9f3406e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:53.739 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.725+0000 7f9f337fe640 10 monclient(hunting): handle_auth_done global_id 44106 payload 824 2024-04-26T04:26:53.740 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.725+0000 7f9f337fe640 1 -- 172.21.15.63:0/34829886 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9f340a0240 msgr2=0x7f9f34072890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:53.740 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.725+0000 7f9f337fe640 1 --2- 172.21.15.63:0/34829886 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9f340a0240 0x7f9f34072890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:53.740 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.725+0000 7f9f337fe640 1 -- 172.21.15.63:0/34829886 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f3413c100 msgr2=0x7f9f3406e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:53.740 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.725+0000 7f9f337fe640 1 --2- 172.21.15.63:0/34829886 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f3413c100 0x7f9f3406e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:53.740 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.725+0000 7f9f337fe640 10 monclient: _finish_hunting 0 2024-04-26T04:26:53.740 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.725+0000 7f9f337fe640 1 monclient: found mon.noname-a 2024-04-26T04:26:53.740 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.725+0000 7f9f337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:26:53.740 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.725+0000 7f9f337fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:26:53.740 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.725+0000 7f9f337fe640 1 -- 172.21.15.63:0/34829886 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f340a7660 con 0x7f9f3409fe60 2024-04-26T04:26:53.740 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.725+0000 7f9f2affd640 1 --2- 172.21.15.63:0/34829886 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f9f340a0240 0x7f9f34072890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T04:26:53.740 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.725+0000 7f9f33fff640 10 monclient: get_auth_request con 0x7f9f3413c100 auth_method 0 2024-04-26T04:26:53.740 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.725+0000 7f9f33fff640 1 --2- 172.21.15.63:0/34829886 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f3413c100 0x7f9f3406e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T04:26:53.740 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.725+0000 7f9f337fe640 1 -- 172.21.15.63:0/34829886 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9f3409fe60 msgr2=0x7f9f34072350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:26:53.740 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.725+0000 7f9f337fe640 1 --2- 172.21.15.63:0/34829886 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9f3409fe60 0x7f9f34072350 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9f2400e930 tx=0x7f9f2400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:26:53.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.725+0000 7f9f30ff9640 1 -- 172.21.15.63:0/34829886 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f24017070 con 0x7f9f3409fe60 2024-04-26T04:26:53.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.725+0000 7f9f30ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:53.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.725+0000 7f9f30ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:26:53.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.725+0000 7f9f30ff9640 10 monclient: dump: 2024-04-26T04:26:53.741 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:53.741 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:53.741 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:53.741 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:53.741 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:53.741 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:53.741 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:53.741 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:53.741 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:53.741 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:53.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.725+0000 7f9f30ff9640 10 monclient: _finish_auth 0 2024-04-26T04:26:53.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.725+0000 7f9f30ff9640 10 monclient: _check_auth_tickets 2024-04-26T04:26:53.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.725+0000 7f9f30ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:26:53.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.725+0000 7f9f30ff9640 1 -- 172.21.15.63:0/34829886 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f2400cc10 con 0x7f9f3409fe60 2024-04-26T04:26:53.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.725+0000 7f9f39e32640 5 monclient: authenticate success, global_id 44106 2024-04-26T04:26:53.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.725+0000 7f9f39e32640 10 monclient: _renew_subs 2024-04-26T04:26:53.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.725+0000 7f9f39e32640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:26:53.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.725+0000 7f9f39e32640 1 -- 172.21.15.63:0/34829886 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f3413b6b0 con 0x7f9f3409fe60 2024-04-26T04:26:53.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.725+0000 7f9f30ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T04:26:53.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.725+0000 7f9f30ff9640 1 -- 172.21.15.63:0/34829886 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f24013470 con 0x7f9f3409fe60 2024-04-26T04:26:53.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.725+0000 7f9f30ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:53.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.725+0000 7f9f30ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:26:53.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.725+0000 7f9f30ff9640 10 monclient: dump: 2024-04-26T04:26:53.742 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:53.743 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:53.743 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:53.743 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:53.743 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:53.743 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:53.743 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:53.743 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:53.743 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:53.743 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:53.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.725+0000 7f9f327fc640 4 set_mon_vals no callback set 2024-04-26T04:26:53.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.725+0000 7f9f39e32640 10 monclient: _renew_subs 2024-04-26T04:26:53.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.725+0000 7f9f39e32640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:26:53.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.725+0000 7f9f39e32640 1 -- 172.21.15.63:0/34829886 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f3406f8b0 con 0x7f9f3409fe60 2024-04-26T04:26:53.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.725+0000 7f9f39e32640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:26:53.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.725+0000 7f9f39e32640 1 --2- 172.21.15.63:0/34829886 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9f341cef60 0x7f9f3406e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:53.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.725+0000 7f9f39e32640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:53.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.725+0000 7f9f39e32640 1 -- 172.21.15.63:0/34829886 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f9f34062e10 con 0x7f9f341cef60 2024-04-26T04:26:53.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.725+0000 7f9f2affd640 1 --2- 172.21.15.63:0/34829886 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9f341cef60 0x7f9f3406e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:53.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.725+0000 7f9f2affd640 10 monclient: get_auth_request con 0x7f9f341cef60 auth_method 0 2024-04-26T04:26:53.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.725+0000 7f9f2affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:53.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.725+0000 7f9f2affd640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:53.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.725+0000 7f9f2affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:53.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.725+0000 7f9f2affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:26:53.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.725+0000 7f9f2affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:26:53.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.725+0000 7f9f2affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:26:53.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.725+0000 7f9f337fe640 1 --2- 172.21.15.63:0/34829886 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9f3409fe60 0x7f9f34072350 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9f2400e930 tx=0x7f9f2400edf0 comp rx=0 tx=0).queue_received will delay after 1.71411e+09 on 0x7f9f2401a070 mgrmap(e 5) 2024-04-26T04:26:53.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.725+0000 7f9f30ff9640 1 -- 172.21.15.63:0/34829886 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f9f2401a070 con 0x7f9f3409fe60 2024-04-26T04:26:53.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.725+0000 7f9f2affd640 10 monclient(hunting): handle_auth_done global_id 44109 payload 824 2024-04-26T04:26:53.744 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.725+0000 7f9f2affd640 1 -- 172.21.15.63:0/34829886 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9f340008e0 con 0x7f9f341cef60 2024-04-26T04:26:53.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.725+0000 7f9f30ff9640 1 --2- 172.21.15.63:0/34829886 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f9f14035ec0 0x7f9f14038380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:53.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.729+0000 7f9f33fff640 1 --2- 172.21.15.63:0/34829886 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f9f14035ec0 0x7f9f14038380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:53.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.729+0000 7f9f33fff640 10 monclient: get_auth_request con 0x7f9f14035ec0 auth_method 0 2024-04-26T04:26:53.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.729+0000 7f9f2affd640 1 -- 172.21.15.63:0/34829886 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9f341cef60 msgr2=0x7f9f3406e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:26:53.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.729+0000 7f9f2affd640 1 --2- 172.21.15.63:0/34829886 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9f341cef60 0x7f9f3406e860 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9f340733b0 tx=0x7f9f1c00ff70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:26:53.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.729+0000 7f9f30ff9640 1 -- 172.21.15.63:0/34829886 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f1c018020 con 0x7f9f341cef60 2024-04-26T04:26:53.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.729+0000 7f9f30ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:53.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.729+0000 7f9f30ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:26:53.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.729+0000 7f9f30ff9640 10 monclient: dump: 2024-04-26T04:26:53.745 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:53.745 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:53.745 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:53.745 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:53.745 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:53.745 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:53.746 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:53.746 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:53.746 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:53.746 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:53.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.729+0000 7f9f30ff9640 1 -- 172.21.15.63:0/34829886 <== mon.0 v2:172.21.15.63:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9f1c00dd30 con 0x7f9f341cef60 2024-04-26T04:26:53.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.729+0000 7f9f30ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:26:53.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.729+0000 7f9f30ff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:26:53.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.729+0000 7f9f30ff9640 1 -- 172.21.15.63:0/34829886 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9f341cef60 msgr2=0x7f9f3406e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:53.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.729+0000 7f9f30ff9640 1 --2- 172.21.15.63:0/34829886 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9f341cef60 0x7f9f3406e860 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9f340733b0 tx=0x7f9f1c00ff70 comp rx=0 tx=0).stop 2024-04-26T04:26:53.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.729+0000 7f9f30ff9640 1 -- 172.21.15.63:0/34829886 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9f341cef60 msgr2=0x7f9f3406e860 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:26:53.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.729+0000 7f9f30ff9640 1 --2- 172.21.15.63:0/34829886 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9f341cef60 0x7f9f3406e860 secure :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9f340733b0 tx=0x7f9f1c00ff70 comp rx=0 tx=0).stop 2024-04-26T04:26:53.746 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.729+0000 7f9f33fff640 1 --2- 172.21.15.63:0/34829886 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f9f14035ec0 0x7f9f14038380 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f9f3406dab0 tx=0x7f9f2c0a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:26:53.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.737+0000 7f9f39e32640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T04:26:53.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.737+0000 7f9f39e32640 1 --2- 172.21.15.63:0/34829886 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9f3406eda0 0x7f9f3413c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:53.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.737+0000 7f9f39e32640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:53.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.737+0000 7f9f39e32640 1 -- 172.21.15.63:0/34829886 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f9f3406f1b0 con 0x7f9f3406eda0 2024-04-26T04:26:53.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.737+0000 7f9f2affd640 1 --2- 172.21.15.63:0/34829886 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9f3406eda0 0x7f9f3413c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:53.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.737+0000 7f9f2affd640 10 monclient: get_auth_request con 0x7f9f3406eda0 auth_method 0 2024-04-26T04:26:53.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.737+0000 7f9f2affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:53.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.737+0000 7f9f2affd640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:53.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.737+0000 7f9f2affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:53.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.737+0000 7f9f2affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:26:53.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.737+0000 7f9f2affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:26:53.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.737+0000 7f9f2affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:26:53.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.737+0000 7f9f2affd640 10 monclient(hunting): handle_auth_done global_id 44112 payload 824 2024-04-26T04:26:53.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.737+0000 7f9f2affd640 1 -- 172.21.15.63:0/34829886 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9f34070580 con 0x7f9f3406eda0 2024-04-26T04:26:53.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.737+0000 7f9f2affd640 1 -- 172.21.15.63:0/34829886 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9f3406eda0 msgr2=0x7f9f3413c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:26:53.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.737+0000 7f9f2affd640 1 --2- 172.21.15.63:0/34829886 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9f3406eda0 0x7f9f3413c570 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9f3413b4e0 tx=0x7f9f1c003700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:26:53.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.737+0000 7f9f30ff9640 1 -- 172.21.15.63:0/34829886 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f1c018030 con 0x7f9f3406eda0 2024-04-26T04:26:53.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.737+0000 7f9f30ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:53.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.737+0000 7f9f30ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:26:53.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.737+0000 7f9f30ff9640 10 monclient: dump: 2024-04-26T04:26:53.749 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:53.749 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:53.750 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:53.750 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:53.750 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:53.750 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:53.750 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:53.750 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:53.750 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:53.750 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:53.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.741+0000 7f9f30ff9640 1 -- 172.21.15.63:0/34829886 <== mon.0 v2:172.21.15.63:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4552 (secure 0 0 0) 0x7f9f1c014b00 con 0x7f9f3406eda0 2024-04-26T04:26:53.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.741+0000 7f9f30ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T04:26:53.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.741+0000 7f9f30ff9640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:26:53.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.741+0000 7f9f30ff9640 1 -- 172.21.15.63:0/34829886 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9f3406eda0 msgr2=0x7f9f3413c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:53.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.741+0000 7f9f30ff9640 1 --2- 172.21.15.63:0/34829886 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9f3406eda0 0x7f9f3413c570 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9f3413b4e0 tx=0x7f9f1c003700 comp rx=0 tx=0).stop 2024-04-26T04:26:53.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.741+0000 7f9f30ff9640 1 -- 172.21.15.63:0/34829886 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9f3406eda0 msgr2=0x7f9f3413c570 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:26:53.751 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.741+0000 7f9f30ff9640 1 --2- 172.21.15.63:0/34829886 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9f3406eda0 0x7f9f3413c570 secure :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9f3413b4e0 tx=0x7f9f1c003700 comp rx=0 tx=0).stop 2024-04-26T04:26:53.751 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-26T04:26:53.751 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a", 2024-04-26T04:26:53.751 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 0, 2024-04-26T04:26:53.751 INFO:teuthology.orchestra.run.smithi063.stdout: "state": "leader", 2024-04-26T04:26:53.751 INFO:teuthology.orchestra.run.smithi063.stdout: "election_epoch": 42, 2024-04-26T04:26:53.751 INFO:teuthology.orchestra.run.smithi063.stdout: "uptime": 25927, 2024-04-26T04:26:53.751 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum": [ 2024-04-26T04:26:53.751 INFO:teuthology.orchestra.run.smithi063.stdout: 0, 2024-04-26T04:26:53.751 INFO:teuthology.orchestra.run.smithi063.stdout: 1, 2024-04-26T04:26:53.752 INFO:teuthology.orchestra.run.smithi063.stdout: 2 2024-04-26T04:26:53.752 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:26:53.752 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_age": 0, 2024-04-26T04:26:53.752 INFO:teuthology.orchestra.run.smithi063.stdout: "features": { 2024-04-26T04:26:53.752 INFO:teuthology.orchestra.run.smithi063.stdout: "required_con": "2450521708007866372", 2024-04-26T04:26:53.752 INFO:teuthology.orchestra.run.smithi063.stdout: "required_mon": [ 2024-04-26T04:26:53.752 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:26:53.752 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:26:53.752 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:26:53.752 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:26:53.752 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:26:53.752 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:26:53.752 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:26:53.752 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:26:53.752 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:26:53.753 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:26:53.753 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:26:53.753 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:26:53.753 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_con": "4540701547738038271", 2024-04-26T04:26:53.753 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_mon": [ 2024-04-26T04:26:53.753 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:26:53.753 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:26:53.753 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:26:53.753 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:26:53.753 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:26:53.753 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:26:53.753 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:26:53.753 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:26:53.753 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:26:53.753 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:26:53.754 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:26:53.754 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:26:53.754 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:26:53.754 INFO:teuthology.orchestra.run.smithi063.stdout: "outside_quorum": [], 2024-04-26T04:26:53.754 INFO:teuthology.orchestra.run.smithi063.stdout: "extra_probe_peers": [], 2024-04-26T04:26:53.754 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_provider": [], 2024-04-26T04:26:53.754 INFO:teuthology.orchestra.run.smithi063.stdout: "monmap": { 2024-04-26T04:26:53.754 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 1, 2024-04-26T04:26:53.754 INFO:teuthology.orchestra.run.smithi063.stdout: "fsid": "cd1c6a0c-4e4b-4014-89c0-2e535479ef39", 2024-04-26T04:26:53.754 INFO:teuthology.orchestra.run.smithi063.stdout: "modified": "2024-04-26T04:18:03.702339Z", 2024-04-26T04:26:53.754 INFO:teuthology.orchestra.run.smithi063.stdout: "created": "2024-04-26T04:18:03.702339Z", 2024-04-26T04:26:53.754 INFO:teuthology.orchestra.run.smithi063.stdout: "min_mon_release": 19, 2024-04-26T04:26:53.754 INFO:teuthology.orchestra.run.smithi063.stdout: "min_mon_release_name": "squid", 2024-04-26T04:26:53.754 INFO:teuthology.orchestra.run.smithi063.stdout: "election_strategy": 1, 2024-04-26T04:26:53.755 INFO:teuthology.orchestra.run.smithi063.stdout: "disallowed_leaders: ": "", 2024-04-26T04:26:53.755 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": false, 2024-04-26T04:26:53.755 INFO:teuthology.orchestra.run.smithi063.stdout: "tiebreaker_mon": "", 2024-04-26T04:26:53.755 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_ranks: ": "", 2024-04-26T04:26:53.755 INFO:teuthology.orchestra.run.smithi063.stdout: "features": { 2024-04-26T04:26:53.755 INFO:teuthology.orchestra.run.smithi063.stdout: "persistent": [ 2024-04-26T04:26:53.755 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:26:53.755 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:26:53.755 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:26:53.755 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:26:53.755 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:26:53.755 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:26:53.755 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:26:53.755 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:26:53.756 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:26:53.756 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:26:53.756 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:26:53.756 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:26:53.756 INFO:teuthology.orchestra.run.smithi063.stdout: "optional": [] 2024-04-26T04:26:53.756 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:26:53.756 INFO:teuthology.orchestra.run.smithi063.stdout: "mons": [ 2024-04-26T04:26:53.756 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:26:53.756 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 0, 2024-04-26T04:26:53.756 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a", 2024-04-26T04:26:53.756 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:26:53.756 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:26:53.756 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:26:53.756 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:26:53.756 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:3300", 2024-04-26T04:26:53.757 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:26:53.757 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:26:53.757 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:26:53.757 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:26:53.757 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6789", 2024-04-26T04:26:53.757 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:26:53.757 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:26:53.757 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:26:53.757 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:26:53.757 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6789/0", 2024-04-26T04:26:53.757 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6789/0", 2024-04-26T04:26:53.757 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:26:53.758 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:26:53.758 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:26:53.758 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:26:53.758 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:26:53.758 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 1, 2024-04-26T04:26:53.758 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b", 2024-04-26T04:26:53.758 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:26:53.758 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:26:53.758 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:26:53.758 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:26:53.758 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:3300", 2024-04-26T04:26:53.758 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:26:53.758 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:26:53.759 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:26:53.759 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:26:53.759 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:6789", 2024-04-26T04:26:53.759 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:26:53.759 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:26:53.759 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:26:53.759 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:26:53.759 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:6789/0", 2024-04-26T04:26:53.759 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.186:6789/0", 2024-04-26T04:26:53.759 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:26:53.759 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:26:53.759 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:26:53.759 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:26:53.759 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:26:53.760 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 2, 2024-04-26T04:26:53.760 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "c", 2024-04-26T04:26:53.760 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:26:53.760 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:26:53.760 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:26:53.760 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:26:53.760 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:3301", 2024-04-26T04:26:53.760 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:26:53.760 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:26:53.760 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:26:53.760 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:26:53.760 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6790", 2024-04-26T04:26:53.760 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:26:53.760 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:26:53.761 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:26:53.761 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:26:53.761 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6790/0", 2024-04-26T04:26:53.761 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6790/0", 2024-04-26T04:26:53.761 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:26:53.761 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:26:53.761 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:26:53.761 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:26:53.761 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:26:53.761 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:26:53.761 INFO:teuthology.orchestra.run.smithi063.stdout: "feature_map": { 2024-04-26T04:26:53.761 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": [ 2024-04-26T04:26:53.761 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:26:53.761 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:26:53.762 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:26:53.762 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 1 2024-04-26T04:26:53.762 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:26:53.762 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:26:53.762 INFO:teuthology.orchestra.run.smithi063.stdout: "client": [ 2024-04-26T04:26:53.762 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:26:53.762 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:26:53.762 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:26:53.762 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 2 2024-04-26T04:26:53.762 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:26:53.762 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:26:53.762 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:26:53.762 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": false 2024-04-26T04:26:53.762 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-26T04:26:53.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.741+0000 7f9f39e32640 1 -- 172.21.15.63:0/34829886 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f9f14035ec0 msgr2=0x7f9f14038380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:53.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.741+0000 7f9f39e32640 1 --2- 172.21.15.63:0/34829886 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f9f14035ec0 0x7f9f14038380 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f9f3406dab0 tx=0x7f9f2c0a7070 comp rx=0 tx=0).stop 2024-04-26T04:26:53.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.741+0000 7f9f39e32640 10 monclient: shutdown 2024-04-26T04:26:53.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.741+0000 7f9f39e32640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:26:53.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.741+0000 7f9f39e32640 1 -- 172.21.15.63:0/34829886 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9f3409fe60 msgr2=0x7f9f34072350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:53.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.741+0000 7f9f39e32640 1 --2- 172.21.15.63:0/34829886 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9f3409fe60 0x7f9f34072350 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9f2400e930 tx=0x7f9f2400edf0 comp rx=0 tx=0).stop 2024-04-26T04:26:53.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.741+0000 7f9f33fff640 1 -- 172.21.15.63:0/34829886 reap_dead start 2024-04-26T04:26:53.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.741+0000 7f9f33fff640 1 -- 172.21.15.63:0/34829886 reap_dead start 2024-04-26T04:26:53.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.741+0000 7f9f30ff9640 1 -- 172.21.15.63:0/34829886 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(323..323 src has 1..323) ==== 4346+0+0 (unknown 0 0 0) 0x7f9f24048e20 con 0x7f9f3409fe60 2024-04-26T04:26:53.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.741+0000 7f9f39e32640 1 -- 172.21.15.63:0/34829886 shutdown_connections 2024-04-26T04:26:53.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.741+0000 7f9f39e32640 1 -- 172.21.15.63:0/34829886 >> 172.21.15.63:0/34829886 conn(0x7f9f340a5fb0 msgr2=0x7f9f340a8860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:26:53.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.741+0000 7f9f39e32640 1 -- 172.21.15.63:0/34829886 shutdown_connections 2024-04-26T04:26:53.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.741+0000 7f9f39e32640 1 -- 172.21.15.63:0/34829886 wait complete. 2024-04-26T04:26:53.765 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T04:26:53.765 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-04-26T04:26:53.765 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-04-26T04:26:53.882 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.877+0000 7fd59121e640 10 monclient: get_monmap_and_config 2024-04-26T04:26:53.883 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.877+0000 7fd59121e640 10 monclient: build_initial_monmap 2024-04-26T04:26:53.883 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.877+0000 7fd59121e640 10 monclient: monmap: 2024-04-26T04:26:53.883 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:26:53.883 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:53.883 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:26:53.882964+0000 2024-04-26T04:26:53.883 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:26:53.882964+0000 2024-04-26T04:26:53.883 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:26:53.883 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:53.883 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:26:53.883 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:26:53.883 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:26:53.883 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:26:53.883 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:26:53.883 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:53.885 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.877+0000 7fd59121e640 1 Processor -- start 2024-04-26T04:26:53.886 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.877+0000 7fd59121e640 1 -- start start 2024-04-26T04:26:53.886 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.877+0000 7fd59121e640 10 monclient: init 2024-04-26T04:26:53.886 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.877+0000 7fd59121e640 10 monclient: _reopen_session rank -1 2024-04-26T04:26:53.886 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.877+0000 7fd59121e640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-04-26T04:26:53.886 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.877+0000 7fd59121e640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fd58c0a2cd0 0x7fd58c0a30b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:53.886 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.877+0000 7fd59121e640 10 monclient(hunting): picked mon.noname-b con 0x7fd58c0a2cd0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:26:53.886 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.877+0000 7fd59121e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd58c0a47d0 addr v1:172.21.15.186:6789/0 2024-04-26T04:26:53.886 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.877+0000 7fd59121e640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7fd58c0a4bd0 0x7fd58c1373f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:53.886 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.877+0000 7fd59121e640 10 monclient(hunting): picked mon.noname-a con 0x7fd58c0a4bd0 addr v2:172.21.15.63:3300/0 2024-04-26T04:26:53.886 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.877+0000 7fd59121e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd58c0a0fb0 con 0x7fd58c0a47d0 2024-04-26T04:26:53.886 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.877+0000 7fd59121e640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:53.886 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.877+0000 7fd59121e640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7fd58c0a35f0 con 0x7fd58c0a4bd0 2024-04-26T04:26:53.886 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.877+0000 7fd59121e640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:53.886 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.877+0000 7fd59121e640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7fd58c0a5380 con 0x7fd58c0a2cd0 2024-04-26T04:26:53.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.877+0000 7fd59121e640 10 monclient(hunting): _renew_subs 2024-04-26T04:26:53.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.877+0000 7fd59121e640 10 monclient(hunting): authenticate will time out at 1445.591167s 2024-04-26T04:26:53.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.877+0000 7fd58ad9d640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fd58c0a2cd0 0x7fd58c0a30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:53.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.877+0000 7fd58b59e640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7fd58c0a4bd0 0x7fd58c1373f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:53.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.877+0000 7fd58ad9d640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fd58c0a2cd0 0x7fd58c0a30b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:40996/0 (socket says 172.21.15.63:40996) 2024-04-26T04:26:53.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.877+0000 7fd58b59e640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7fd58c0a4bd0 0x7fd58c1373f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:55232/0 (socket says 172.21.15.63:55232) 2024-04-26T04:26:53.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.877+0000 7fd58ad9d640 1 -- 172.21.15.63:0/3209819114 learned_addr learned my addr 172.21.15.63:0/3209819114 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:26:53.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.877+0000 7fd58ad9d640 10 monclient(hunting): get_auth_request con 0x7fd58c0a2cd0 auth_method 0 2024-04-26T04:26:53.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.877+0000 7fd58ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:53.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.877+0000 7fd58ad9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:53.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.877+0000 7fd58ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:53.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.877+0000 7fd58b59e640 10 monclient(hunting): get_auth_request con 0x7fd58c0a4bd0 auth_method 0 2024-04-26T04:26:53.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.877+0000 7fd58b59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:53.887 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.877+0000 7fd58b59e640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:53.888 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.877+0000 7fd58b59e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:53.888 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.877+0000 7fd58ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:26:53.888 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.877+0000 7fd58ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:26:53.888 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.877+0000 7fd58ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:26:53.888 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.877+0000 7fd58b59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:26:53.888 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.877+0000 7fd58b59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:26:53.888 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.877+0000 7fd58b59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:26:53.888 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.877+0000 7fd58ad9d640 10 monclient(hunting): handle_auth_done global_id 24677 payload 274 2024-04-26T04:26:53.888 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.877+0000 7fd58ad9d640 1 -- 172.21.15.63:0/3209819114 >> v2:172.21.15.63:3300/0 conn(0x7fd58c0a4bd0 msgr2=0x7fd58c1373f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:26:53.888 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.877+0000 7fd58ad9d640 1 --2- 172.21.15.63:0/3209819114 >> v2:172.21.15.63:3300/0 conn(0x7fd58c0a4bd0 0x7fd58c1373f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:53.888 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.877+0000 7fd58ad9d640 1 -- 172.21.15.63:0/3209819114 >> v1:172.21.15.186:6789/0 conn(0x7fd58c0a47d0 legacy=0x7fd58c136ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:26:53.888 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.877+0000 7fd58ad9d640 10 monclient: _finish_hunting 0 2024-04-26T04:26:53.888 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.877+0000 7fd58ad9d640 1 monclient: found mon.noname-b 2024-04-26T04:26:53.888 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.877+0000 7fd58ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:26:53.888 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.877+0000 7fd58ad9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.63:3301/0 2024-04-26T04:26:53.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.877+0000 7fd58ad9d640 1 -- 172.21.15.63:0/3209819114 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd58c0a1290 con 0x7fd58c0a2cd0 2024-04-26T04:26:53.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.877+0000 7fd58b59e640 1 --2- 172.21.15.63:0/3209819114 >> v2:172.21.15.63:3300/0 conn(0x7fd58c0a4bd0 0x7fd58c1373f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T04:26:53.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.885+0000 7fd58ad9d640 1 -- 172.21.15.63:0/3209819114 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fd58c0a2cd0 msgr2=0x7fd58c0a30b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:26:53.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.885+0000 7fd58ad9d640 1 --2- 172.21.15.63:0/3209819114 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fd58c0a2cd0 0x7fd58c0a30b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd57c009870 tx=0x7fd57c01c470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c189545c37e0cd9a server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:26:53.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.885+0000 7fd589d9b640 1 -- 172.21.15.63:0/3209819114 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd57c0b0070 con 0x7fd58c0a2cd0 2024-04-26T04:26:53.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.885+0000 7fd589d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:53.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.885+0000 7fd589d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T04:26:53.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.885+0000 7fd589d9b640 10 monclient: dump: 2024-04-26T04:26:53.891 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:53.891 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:53.891 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:53.891 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:53.891 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:53.891 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:53.891 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:53.891 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:53.891 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:53.892 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:53.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.885+0000 7fd589d9b640 10 monclient: _finish_auth 0 2024-04-26T04:26:53.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.885+0000 7fd589d9b640 10 monclient: _check_auth_tickets 2024-04-26T04:26:53.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.885+0000 7fd589d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:26:53.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.885+0000 7fd589d9b640 1 -- 172.21.15.63:0/3209819114 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd57c0ab600 con 0x7fd58c0a2cd0 2024-04-26T04:26:53.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.885+0000 7fd589d9b640 10 monclient: handle_config config(0 keys) 2024-04-26T04:26:53.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.885+0000 7fd589d9b640 1 -- 172.21.15.63:0/3209819114 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd57c0ab9d0 con 0x7fd58c0a2cd0 2024-04-26T04:26:53.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.885+0000 7fd589d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:53.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.885+0000 7fd589d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:26:53.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.885+0000 7fd589d9b640 10 monclient: dump: 2024-04-26T04:26:53.892 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:53.892 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:53.892 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:53.892 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:53.892 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:53.892 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:53.893 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:53.893 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:53.893 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:53.893 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:53.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.885+0000 7fd59121e640 5 monclient: authenticate success, global_id 24677 2024-04-26T04:26:53.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.885+0000 7fd59121e640 10 monclient: get_monmap_and_config success 2024-04-26T04:26:53.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.885+0000 7fd59121e640 4 set_mon_vals no callback set 2024-04-26T04:26:53.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.885+0000 7fd59121e640 10 monclient: shutdown 2024-04-26T04:26:53.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.885+0000 7fd59121e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:26:53.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.885+0000 7fd59121e640 1 -- 172.21.15.63:0/3209819114 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fd58c0a2cd0 msgr2=0x7fd58c0a30b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:53.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.885+0000 7fd59121e640 1 --2- 172.21.15.63:0/3209819114 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fd58c0a2cd0 0x7fd58c0a30b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd57c009870 tx=0x7fd57c01c470 comp rx=0 tx=0).stop 2024-04-26T04:26:53.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.885+0000 7fd59121e640 1 -- 172.21.15.63:0/3209819114 shutdown_connections 2024-04-26T04:26:53.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.885+0000 7fd59121e640 1 --2- 172.21.15.63:0/3209819114 >> v2:172.21.15.63:3300/0 conn(0x7fd58c0a4bd0 0x7fd58c1373f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:53.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.885+0000 7fd59121e640 1 --2- 172.21.15.63:0/3209819114 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fd58c0a2cd0 0x7fd58c0a30b0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:53.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.885+0000 7fd59121e640 1 -- 172.21.15.63:0/3209819114 >> 172.21.15.63:0/3209819114 conn(0x7fd58c0a5f40 msgr2=0x7fd58c0a6320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:26:53.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.885+0000 7fd59121e640 1 -- 172.21.15.63:0/3209819114 shutdown_connections 2024-04-26T04:26:53.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.885+0000 7fd59121e640 1 -- 172.21.15.63:0/3209819114 wait complete. 2024-04-26T04:26:53.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.885+0000 7fd59121e640 10 monclient: build_initial_monmap 2024-04-26T04:26:53.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.885+0000 7fd59121e640 10 monclient: monmap: 2024-04-26T04:26:53.900 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:26:53.900 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:26:53.900 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:26:53.900 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:26:53.900 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:26:53.900 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:53.900 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:26:53.900 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:26:53.901 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:26:53.901 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:53.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.885+0000 7fd59121e640 1 Processor -- start 2024-04-26T04:26:53.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.885+0000 7fd59121e640 1 -- start start 2024-04-26T04:26:53.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.885+0000 7fd59121e640 10 monclient: init 2024-04-26T04:26:53.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.885+0000 7fd59121e640 10 monclient: _reopen_session rank -1 2024-04-26T04:26:53.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.885+0000 7fd59121e640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T04:26:53.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.885+0000 7fd59121e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd58c0a2cd0 0x7fd58c1cf530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:53.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.885+0000 7fd59121e640 10 monclient(hunting): picked mon.noname-a con 0x7fd58c0a2cd0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:26:53.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.885+0000 7fd59121e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd58c0a47d0 0x7fd58c13bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:53.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.885+0000 7fd59121e640 10 monclient(hunting): picked mon.noname-b con 0x7fd58c0a47d0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:26:53.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.885+0000 7fd59121e640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fd58c0a4bd0 0x7fd58c13c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:53.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd59121e640 10 monclient(hunting): picked mon.noname-c con 0x7fd58c0a4bd0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:26:53.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd59121e640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:53.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd59121e640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fd58c141a50 con 0x7fd58c0a2cd0 2024-04-26T04:26:53.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd59121e640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:53.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd59121e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd58c137b40 con 0x7fd58c0a47d0 2024-04-26T04:26:53.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd59121e640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:53.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd59121e640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7fd58c141bd0 con 0x7fd58c0a4bd0 2024-04-26T04:26:53.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd59121e640 10 monclient(hunting): _renew_subs 2024-04-26T04:26:53.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd59121e640 10 monclient(hunting): authenticate will time out at 1445.600040s 2024-04-26T04:26:53.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd58ad9d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd58c0a2cd0 0x7fd58c1cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:53.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd58b59e640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fd58c0a4bd0 0x7fd58c13c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:53.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd58ad9d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd58c0a2cd0 0x7fd58c1cf530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:55242/0 (socket says 172.21.15.63:55242) 2024-04-26T04:26:53.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd58ad9d640 1 -- 172.21.15.63:0/2718054739 learned_addr learned my addr 172.21.15.63:0/2718054739 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:26:53.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd58ad9d640 10 monclient(hunting): get_auth_request con 0x7fd58c0a2cd0 auth_method 0 2024-04-26T04:26:53.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd58ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:53.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd58ad9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:53.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd58ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:53.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd58b59e640 10 monclient(hunting): get_auth_request con 0x7fd58c0a4bd0 auth_method 0 2024-04-26T04:26:53.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd58b59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:53.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd58b59e640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:53.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd58b59e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:53.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd58ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:26:53.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd58ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:26:53.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd58ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:26:53.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd58b59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:26:53.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd58b59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:26:53.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd58b59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:26:53.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd58a59c640 1 --2- 172.21.15.63:0/2718054739 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd58c0a47d0 0x7fd58c13bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:53.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd58a59c640 10 monclient(hunting): get_auth_request con 0x7fd58c0a47d0 auth_method 0 2024-04-26T04:26:53.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd58a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:53.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd58a59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:53.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd58a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:53.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd58ad9d640 10 monclient(hunting): handle_auth_done global_id 44118 payload 824 2024-04-26T04:26:53.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd58ad9d640 1 -- 172.21.15.63:0/2718054739 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fd58c0a4bd0 msgr2=0x7fd58c13c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:53.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd58ad9d640 1 --2- 172.21.15.63:0/2718054739 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fd58c0a4bd0 0x7fd58c13c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:53.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd58ad9d640 1 -- 172.21.15.63:0/2718054739 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd58c0a47d0 msgr2=0x7fd58c13bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:53.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd58ad9d640 1 --2- 172.21.15.63:0/2718054739 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd58c0a47d0 0x7fd58c13bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:53.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd58ad9d640 10 monclient: _finish_hunting 0 2024-04-26T04:26:53.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd58ad9d640 1 monclient: found mon.noname-a 2024-04-26T04:26:53.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd58ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:26:53.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd58ad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:26:53.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd58ad9d640 1 -- 172.21.15.63:0/2718054739 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd58c120480 con 0x7fd58c0a2cd0 2024-04-26T04:26:53.905 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd58b59e640 1 --2- 172.21.15.63:0/2718054739 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fd58c0a4bd0 0x7fd58c13c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T04:26:53.905 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd58ad9d640 1 -- 172.21.15.63:0/2718054739 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd58c0a2cd0 msgr2=0x7fd58c1cf530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:26:53.905 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd58ad9d640 1 --2- 172.21.15.63:0/2718054739 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd58c0a2cd0 0x7fd58c1cf530 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd57c01a0a0 tx=0x7fd57c019210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:26:53.905 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd5737fe640 1 -- 172.21.15.63:0/2718054739 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd57c0b0070 con 0x7fd58c0a2cd0 2024-04-26T04:26:53.905 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd5737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:53.905 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd5737fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:26:53.905 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd5737fe640 10 monclient: dump: 2024-04-26T04:26:53.905 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:53.905 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:53.905 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:53.905 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:53.905 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:53.905 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:53.905 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:53.905 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:53.906 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:53.906 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:53.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd5737fe640 10 monclient: _finish_auth 0 2024-04-26T04:26:53.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd5737fe640 10 monclient: _check_auth_tickets 2024-04-26T04:26:53.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd5737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:26:53.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd5737fe640 1 -- 172.21.15.63:0/2718054739 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd57c0b1040 con 0x7fd58c0a2cd0 2024-04-26T04:26:53.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd59121e640 5 monclient: authenticate success, global_id 44118 2024-04-26T04:26:53.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd59121e640 10 monclient: _renew_subs 2024-04-26T04:26:53.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd59121e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:26:53.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd59121e640 1 -- 172.21.15.63:0/2718054739 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd58c0a37d0 con 0x7fd58c0a2cd0 2024-04-26T04:26:53.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd5737fe640 10 monclient: handle_config config(0 keys) 2024-04-26T04:26:53.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd5737fe640 1 -- 172.21.15.63:0/2718054739 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd57c0aaa00 con 0x7fd58c0a2cd0 2024-04-26T04:26:53.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd5737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:53.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd5737fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:26:53.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd5737fe640 10 monclient: dump: 2024-04-26T04:26:53.907 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:53.907 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:53.907 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:53.907 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:53.907 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:53.907 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:53.907 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:53.907 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:53.907 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:53.907 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:53.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd59121e640 10 monclient: _renew_subs 2024-04-26T04:26:53.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd59121e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:26:53.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd58959a640 4 set_mon_vals no callback set 2024-04-26T04:26:53.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd59121e640 1 -- 172.21.15.63:0/2718054739 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd58c1201a0 con 0x7fd58c0a2cd0 2024-04-26T04:26:53.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd59121e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:26:53.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd59121e640 1 --2- 172.21.15.63:0/2718054739 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fd58c13c5a0 0x7fd58c13c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:53.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd59121e640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:53.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd59121e640 1 -- 172.21.15.63:0/2718054739 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7fd58c13d720 con 0x7fd58c13c5a0 2024-04-26T04:26:53.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd58a59c640 1 --2- 172.21.15.63:0/2718054739 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fd58c13c5a0 0x7fd58c13c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:53.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.889+0000 7fd58a59c640 10 monclient: get_auth_request con 0x7fd58c13c5a0 auth_method 0 2024-04-26T04:26:53.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.893+0000 7fd58a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:53.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.893+0000 7fd58a59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:53.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.893+0000 7fd58a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:53.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.893+0000 7fd5737fe640 1 -- 172.21.15.63:0/2718054739 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fd57c0aac20 con 0x7fd58c0a2cd0 2024-04-26T04:26:53.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.893+0000 7fd58a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:26:53.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.893+0000 7fd58a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:26:53.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.893+0000 7fd58a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:26:53.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.893+0000 7fd5737fe640 1 --2- 172.21.15.63:0/2718054739 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fd578035ec0 0x7fd578038380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:53.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.893+0000 7fd58a59c640 10 monclient(hunting): handle_auth_done global_id 24683 payload 824 2024-04-26T04:26:53.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.893+0000 7fd58a59c640 1 -- 172.21.15.63:0/2718054739 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd58c11fe20 con 0x7fd58c13c5a0 2024-04-26T04:26:53.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.893+0000 7fd5737fe640 1 -- 172.21.15.63:0/2718054739 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(323..323 src has 1..323) ==== 4346+0+0 (secure 0 0 0) 0x7fd57c0e2ac0 con 0x7fd58c0a2cd0 2024-04-26T04:26:53.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.893+0000 7fd58b59e640 1 --2- 172.21.15.63:0/2718054739 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fd578035ec0 0x7fd578038380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:53.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.893+0000 7fd58b59e640 10 monclient: get_auth_request con 0x7fd578035ec0 auth_method 0 2024-04-26T04:26:53.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.893+0000 7fd58a59c640 1 -- 172.21.15.63:0/2718054739 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fd58c13c5a0 msgr2=0x7fd58c13c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:26:53.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.893+0000 7fd58a59c640 1 --2- 172.21.15.63:0/2718054739 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fd58c13c5a0 0x7fd58c13c980 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd58000e420 tx=0x7fd58000e8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:26:53.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.893+0000 7fd5737fe640 1 -- 172.21.15.63:0/2718054739 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd580007450 con 0x7fd58c13c5a0 2024-04-26T04:26:53.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.893+0000 7fd5737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:53.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.893+0000 7fd5737fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:26:53.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.893+0000 7fd5737fe640 10 monclient: dump: 2024-04-26T04:26:53.912 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:53.912 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:53.912 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:53.912 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:53.912 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:53.912 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:53.912 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:53.912 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:53.912 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:53.912 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:53.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.893+0000 7fd5737fe640 1 -- 172.21.15.63:0/2718054739 <== mon.2 v2:172.21.15.63:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd580007450 con 0x7fd58c13c5a0 2024-04-26T04:26:53.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.893+0000 7fd5737fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:26:53.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.893+0000 7fd5737fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:26:53.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.893+0000 7fd5737fe640 1 -- 172.21.15.63:0/2718054739 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fd58c13c5a0 msgr2=0x7fd58c13c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:53.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.893+0000 7fd5737fe640 1 --2- 172.21.15.63:0/2718054739 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fd58c13c5a0 0x7fd58c13c980 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd58000e420 tx=0x7fd58000e8e0 comp rx=0 tx=0).stop 2024-04-26T04:26:53.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.893+0000 7fd5737fe640 1 -- 172.21.15.63:0/2718054739 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fd58c13c5a0 msgr2=0x7fd58c13c980 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:26:53.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.893+0000 7fd5737fe640 1 --2- 172.21.15.63:0/2718054739 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fd58c13c5a0 0x7fd58c13c980 secure :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd58000e420 tx=0x7fd58000e8e0 comp rx=0 tx=0).stop 2024-04-26T04:26:53.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.893+0000 7fd58b59e640 1 --2- 172.21.15.63:0/2718054739 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fd578035ec0 0x7fd578038380 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fd584003c10 tx=0x7fd58400b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:26:53.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.901+0000 7fd59121e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T04:26:53.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.901+0000 7fd59121e640 1 --2- 172.21.15.63:0/2718054739 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fd58c137cf0 0x7fd58c138100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:53.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.901+0000 7fd59121e640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:53.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.901+0000 7fd59121e640 1 -- 172.21.15.63:0/2718054739 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7fd58c13e0b0 con 0x7fd58c137cf0 2024-04-26T04:26:53.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.901+0000 7fd58a59c640 1 --2- 172.21.15.63:0/2718054739 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fd58c137cf0 0x7fd58c138100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:53.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.901+0000 7fd58a59c640 10 monclient: get_auth_request con 0x7fd58c137cf0 auth_method 0 2024-04-26T04:26:53.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.901+0000 7fd58a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:53.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.901+0000 7fd58a59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:53.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.901+0000 7fd58a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:53.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.901+0000 7fd58a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:26:53.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.901+0000 7fd58a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:26:53.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.901+0000 7fd58a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:26:53.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.901+0000 7fd58a59c640 10 monclient(hunting): handle_auth_done global_id 24686 payload 824 2024-04-26T04:26:53.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.901+0000 7fd58a59c640 1 -- 172.21.15.63:0/2718054739 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd58c062e10 con 0x7fd58c137cf0 2024-04-26T04:26:53.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.905+0000 7fd58a59c640 1 -- 172.21.15.63:0/2718054739 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fd58c137cf0 msgr2=0x7fd58c138100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:26:53.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.905+0000 7fd58a59c640 1 --2- 172.21.15.63:0/2718054739 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fd58c137cf0 0x7fd58c138100 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd58000e420 tx=0x7fd580003770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:26:53.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.905+0000 7fd5737fe640 1 -- 172.21.15.63:0/2718054739 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd58001eb70 con 0x7fd58c137cf0 2024-04-26T04:26:53.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.905+0000 7fd5737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:53.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.905+0000 7fd5737fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:26:53.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.905+0000 7fd5737fe640 10 monclient: dump: 2024-04-26T04:26:53.914 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:53.914 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:53.915 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:53.915 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:53.915 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:53.915 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:53.915 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:53.915 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:53.915 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:53.915 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:53.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.905+0000 7fd5737fe640 1 -- 172.21.15.63:0/2718054739 <== mon.2 v2:172.21.15.63:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4721 (secure 0 0 0) 0x7fd58001eb70 con 0x7fd58c137cf0 2024-04-26T04:26:53.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.905+0000 7fd5737fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T04:26:53.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.905+0000 7fd5737fe640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:26:53.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.905+0000 7fd5737fe640 1 -- 172.21.15.63:0/2718054739 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fd58c137cf0 msgr2=0x7fd58c138100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:53.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.905+0000 7fd5737fe640 1 --2- 172.21.15.63:0/2718054739 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fd58c137cf0 0x7fd58c138100 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd58000e420 tx=0x7fd580003770 comp rx=0 tx=0).stop 2024-04-26T04:26:53.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.905+0000 7fd5737fe640 1 -- 172.21.15.63:0/2718054739 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fd58c137cf0 msgr2=0x7fd58c138100 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:26:53.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.905+0000 7fd5737fe640 1 --2- 172.21.15.63:0/2718054739 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fd58c137cf0 0x7fd58c138100 secure :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd58000e420 tx=0x7fd580003770 comp rx=0 tx=0).stop 2024-04-26T04:26:53.916 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-26T04:26:53.916 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "c", 2024-04-26T04:26:53.916 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 2, 2024-04-26T04:26:53.916 INFO:teuthology.orchestra.run.smithi063.stdout: "state": "peon", 2024-04-26T04:26:53.916 INFO:teuthology.orchestra.run.smithi063.stdout: "election_epoch": 42, 2024-04-26T04:26:53.916 INFO:teuthology.orchestra.run.smithi063.stdout: "uptime": 261505, 2024-04-26T04:26:53.916 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum": [ 2024-04-26T04:26:53.916 INFO:teuthology.orchestra.run.smithi063.stdout: 0, 2024-04-26T04:26:53.916 INFO:teuthology.orchestra.run.smithi063.stdout: 1, 2024-04-26T04:26:53.916 INFO:teuthology.orchestra.run.smithi063.stdout: 2 2024-04-26T04:26:53.917 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:26:53.917 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_age": 0, 2024-04-26T04:26:53.917 INFO:teuthology.orchestra.run.smithi063.stdout: "features": { 2024-04-26T04:26:53.917 INFO:teuthology.orchestra.run.smithi063.stdout: "required_con": "2450521708007866372", 2024-04-26T04:26:53.917 INFO:teuthology.orchestra.run.smithi063.stdout: "required_mon": [ 2024-04-26T04:26:53.917 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:26:53.917 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:26:53.917 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:26:53.917 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:26:53.917 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:26:53.917 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:26:53.917 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:26:53.917 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:26:53.917 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:26:53.917 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:26:53.918 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:26:53.918 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:26:53.918 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_con": "4540701547738038271", 2024-04-26T04:26:53.918 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_mon": [ 2024-04-26T04:26:53.918 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:26:53.918 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:26:53.918 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:26:53.918 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:26:53.918 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:26:53.918 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:26:53.918 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:26:53.918 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:26:53.918 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:26:53.918 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:26:53.919 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:26:53.919 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:26:53.919 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:26:53.919 INFO:teuthology.orchestra.run.smithi063.stdout: "outside_quorum": [], 2024-04-26T04:26:53.919 INFO:teuthology.orchestra.run.smithi063.stdout: "extra_probe_peers": [], 2024-04-26T04:26:53.919 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_provider": [], 2024-04-26T04:26:53.919 INFO:teuthology.orchestra.run.smithi063.stdout: "monmap": { 2024-04-26T04:26:53.919 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 1, 2024-04-26T04:26:53.919 INFO:teuthology.orchestra.run.smithi063.stdout: "fsid": "cd1c6a0c-4e4b-4014-89c0-2e535479ef39", 2024-04-26T04:26:53.919 INFO:teuthology.orchestra.run.smithi063.stdout: "modified": "2024-04-26T04:18:03.702339Z", 2024-04-26T04:26:53.919 INFO:teuthology.orchestra.run.smithi063.stdout: "created": "2024-04-26T04:18:03.702339Z", 2024-04-26T04:26:53.919 INFO:teuthology.orchestra.run.smithi063.stdout: "min_mon_release": 19, 2024-04-26T04:26:53.919 INFO:teuthology.orchestra.run.smithi063.stdout: "min_mon_release_name": "squid", 2024-04-26T04:26:53.919 INFO:teuthology.orchestra.run.smithi063.stdout: "election_strategy": 1, 2024-04-26T04:26:53.919 INFO:teuthology.orchestra.run.smithi063.stdout: "disallowed_leaders: ": "", 2024-04-26T04:26:53.920 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": false, 2024-04-26T04:26:53.920 INFO:teuthology.orchestra.run.smithi063.stdout: "tiebreaker_mon": "", 2024-04-26T04:26:53.920 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_ranks: ": "", 2024-04-26T04:26:53.920 INFO:teuthology.orchestra.run.smithi063.stdout: "features": { 2024-04-26T04:26:53.920 INFO:teuthology.orchestra.run.smithi063.stdout: "persistent": [ 2024-04-26T04:26:53.920 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:26:53.920 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:26:53.920 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:26:53.920 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:26:53.920 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:26:53.920 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:26:53.920 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:26:53.920 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:26:53.920 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:26:53.921 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:26:53.921 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:26:53.921 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:26:53.921 INFO:teuthology.orchestra.run.smithi063.stdout: "optional": [] 2024-04-26T04:26:53.921 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:26:53.921 INFO:teuthology.orchestra.run.smithi063.stdout: "mons": [ 2024-04-26T04:26:53.921 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:26:53.921 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 0, 2024-04-26T04:26:53.921 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a", 2024-04-26T04:26:53.921 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:26:53.921 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:26:53.921 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:26:53.921 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:26:53.921 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:3300", 2024-04-26T04:26:53.921 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:26:53.922 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:26:53.922 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:26:53.922 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:26:53.922 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6789", 2024-04-26T04:26:53.922 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:26:53.922 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:26:53.922 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:26:53.922 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:26:53.922 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6789/0", 2024-04-26T04:26:53.922 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6789/0", 2024-04-26T04:26:53.922 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:26:53.922 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:26:53.922 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:26:53.922 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:26:53.923 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:26:53.923 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 1, 2024-04-26T04:26:53.923 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b", 2024-04-26T04:26:53.923 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:26:53.923 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:26:53.923 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:26:53.923 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:26:53.923 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:3300", 2024-04-26T04:26:53.923 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:26:53.923 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:26:53.923 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:26:53.923 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:26:53.923 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:6789", 2024-04-26T04:26:53.923 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:26:53.923 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:26:53.924 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:26:53.924 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:26:53.924 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:6789/0", 2024-04-26T04:26:53.924 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.186:6789/0", 2024-04-26T04:26:53.924 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:26:53.924 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:26:53.924 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:26:53.924 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:26:53.924 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:26:53.924 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 2, 2024-04-26T04:26:53.924 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "c", 2024-04-26T04:26:53.924 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:26:53.924 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:26:53.924 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:26:53.924 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:26:53.925 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:3301", 2024-04-26T04:26:53.925 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:26:53.925 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:26:53.925 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:26:53.925 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:26:53.925 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6790", 2024-04-26T04:26:53.925 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:26:53.925 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:26:53.925 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:26:53.925 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:26:53.925 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6790/0", 2024-04-26T04:26:53.925 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6790/0", 2024-04-26T04:26:53.925 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:26:53.925 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:26:53.925 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:26:53.926 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:26:53.926 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:26:53.926 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:26:53.926 INFO:teuthology.orchestra.run.smithi063.stdout: "feature_map": { 2024-04-26T04:26:53.926 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": [ 2024-04-26T04:26:53.926 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:26:53.926 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:26:53.926 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:26:53.926 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 1 2024-04-26T04:26:53.926 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:26:53.926 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:26:53.926 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": [ 2024-04-26T04:26:53.926 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:26:53.926 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:26:53.927 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:26:53.927 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 3 2024-04-26T04:26:53.927 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:26:53.927 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:26:53.927 INFO:teuthology.orchestra.run.smithi063.stdout: "client": [ 2024-04-26T04:26:53.927 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:26:53.927 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:26:53.927 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:26:53.927 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 4 2024-04-26T04:26:53.927 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:26:53.927 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:26:53.927 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:26:53.927 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": false 2024-04-26T04:26:53.927 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-26T04:26:53.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.905+0000 7fd59121e640 1 -- 172.21.15.63:0/2718054739 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fd578035ec0 msgr2=0x7fd578038380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:53.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.905+0000 7fd59121e640 1 --2- 172.21.15.63:0/2718054739 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fd578035ec0 0x7fd578038380 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fd584003c10 tx=0x7fd58400b040 comp rx=0 tx=0).stop 2024-04-26T04:26:53.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.905+0000 7fd59121e640 10 monclient: shutdown 2024-04-26T04:26:53.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.905+0000 7fd59121e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:26:53.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.905+0000 7fd59121e640 1 -- 172.21.15.63:0/2718054739 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd58c0a2cd0 msgr2=0x7fd58c1cf530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:53.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.905+0000 7fd59121e640 1 --2- 172.21.15.63:0/2718054739 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd58c0a2cd0 0x7fd58c1cf530 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd57c01a0a0 tx=0x7fd57c019210 comp rx=0 tx=0).stop 2024-04-26T04:26:53.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.905+0000 7fd58b59e640 1 -- 172.21.15.63:0/2718054739 reap_dead start 2024-04-26T04:26:53.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.905+0000 7fd58b59e640 1 -- 172.21.15.63:0/2718054739 reap_dead start 2024-04-26T04:26:53.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.905+0000 7fd59121e640 1 -- 172.21.15.63:0/2718054739 shutdown_connections 2024-04-26T04:26:53.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.905+0000 7fd59121e640 1 -- 172.21.15.63:0/2718054739 >> 172.21.15.63:0/2718054739 conn(0x7fd58c0a5f40 msgr2=0x7fd58c0a1a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:26:53.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.905+0000 7fd59121e640 1 -- 172.21.15.63:0/2718054739 shutdown_connections 2024-04-26T04:26:53.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:53.905+0000 7fd59121e640 1 -- 172.21.15.63:0/2718054739 wait complete. 2024-04-26T04:26:53.930 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T04:26:53.930 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-04-26T04:26:53.930 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-04-26T04:26:54.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.037+0000 7fa26f784640 10 monclient: get_monmap_and_config 2024-04-26T04:26:54.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.037+0000 7fa26f784640 10 monclient: build_initial_monmap 2024-04-26T04:26:54.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.037+0000 7fa26f784640 10 monclient: monmap: 2024-04-26T04:26:54.043 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:26:54.043 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:54.043 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:26:54.042757+0000 2024-04-26T04:26:54.043 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:26:54.042757+0000 2024-04-26T04:26:54.043 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:26:54.043 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:54.043 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:26:54.043 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:26:54.043 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:26:54.044 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:26:54.044 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:26:54.044 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:54.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.037+0000 7fa26f784640 1 Processor -- start 2024-04-26T04:26:54.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.037+0000 7fa26f784640 1 -- start start 2024-04-26T04:26:54.044 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.037+0000 7fa26f784640 10 monclient: init 2024-04-26T04:26:54.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.037+0000 7fa26f784640 10 monclient: _reopen_session rank -1 2024-04-26T04:26:54.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.037+0000 7fa26f784640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-04-26T04:26:54.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.037+0000 7fa26f784640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa26809c6a0 0x7fa2680996b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:54.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.037+0000 7fa26f784640 10 monclient(hunting): picked mon.noname-c con 0x7fa26809c6a0 addr v2:172.21.15.186:3300/0 2024-04-26T04:26:54.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.037+0000 7fa26f784640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa26809ca80 addr v1:172.21.15.186:6789/0 2024-04-26T04:26:54.046 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.037+0000 7fa26f784640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7fa268138740 0x7fa26813ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:54.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.037+0000 7fa26f784640 10 monclient(hunting): picked mon.noname-a con 0x7fa268138740 addr v2:172.21.15.63:3300/0 2024-04-26T04:26:54.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.037+0000 7fa26f784640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa26809d9f0 con 0x7fa26809ca80 2024-04-26T04:26:54.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.037+0000 7fa26f784640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:54.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.037+0000 7fa26f784640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7fa26809c1d0 con 0x7fa268138740 2024-04-26T04:26:54.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.037+0000 7fa26f784640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:54.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.037+0000 7fa26f784640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fa2680992d0 con 0x7fa26809c6a0 2024-04-26T04:26:54.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.037+0000 7fa26f784640 10 monclient(hunting): _renew_subs 2024-04-26T04:26:54.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.037+0000 7fa26f784640 10 monclient(hunting): authenticate will time out at 1445.750145s 2024-04-26T04:26:54.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.037+0000 7fa26dd21640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7fa268138740 0x7fa26813ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:54.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.037+0000 7fa26dd21640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7fa268138740 0x7fa26813ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:55248/0 (socket says 172.21.15.63:55248) 2024-04-26T04:26:54.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.037+0000 7fa26dd21640 1 -- 172.21.15.63:0/2268528023 learned_addr learned my addr 172.21.15.63:0/2268528023 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:26:54.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.037+0000 7fa26dd21640 10 monclient(hunting): get_auth_request con 0x7fa268138740 auth_method 0 2024-04-26T04:26:54.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.037+0000 7fa26dd21640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:54.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.037+0000 7fa26dd21640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:54.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.037+0000 7fa26dd21640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:54.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.037+0000 7fa26dd21640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:26:54.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.037+0000 7fa26dd21640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:26:54.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.037+0000 7fa26dd21640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:26:54.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.037+0000 7fa26d520640 1 --2- 172.21.15.63:0/2268528023 >> v2:172.21.15.186:3300/0 conn(0x7fa26809c6a0 0x7fa2680996b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:54.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.037+0000 7fa26d520640 10 monclient(hunting): get_auth_request con 0x7fa26809c6a0 auth_method 0 2024-04-26T04:26:54.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.037+0000 7fa26d520640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:54.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.037+0000 7fa26d520640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:54.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.037+0000 7fa26d520640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:54.048 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.037+0000 7fa26dd21640 10 monclient(hunting): handle_auth_done global_id 44121 payload 274 2024-04-26T04:26:54.049 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.037+0000 7fa26dd21640 1 -- 172.21.15.63:0/2268528023 >> v2:172.21.15.186:3300/0 conn(0x7fa26809c6a0 msgr2=0x7fa2680996b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:26:54.049 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.037+0000 7fa26dd21640 1 --2- 172.21.15.63:0/2268528023 >> v2:172.21.15.186:3300/0 conn(0x7fa26809c6a0 0x7fa2680996b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:54.049 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.037+0000 7fa26dd21640 1 -- 172.21.15.63:0/2268528023 >> v1:172.21.15.186:6789/0 conn(0x7fa26809ca80 legacy=0x7fa26809ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:26:54.049 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.037+0000 7fa26dd21640 10 monclient: _finish_hunting 0 2024-04-26T04:26:54.049 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.037+0000 7fa26dd21640 1 monclient: found mon.noname-a 2024-04-26T04:26:54.049 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.037+0000 7fa26dd21640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:26:54.049 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.037+0000 7fa26dd21640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:26:54.049 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.037+0000 7fa26dd21640 1 -- 172.21.15.63:0/2268528023 --> v2:172.21.15.63:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa26809dc30 con 0x7fa268138740 2024-04-26T04:26:54.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.041+0000 7fa26dd21640 1 -- 172.21.15.63:0/2268528023 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa268138740 msgr2=0x7fa26813ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:26:54.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.041+0000 7fa26dd21640 1 --2- 172.21.15.63:0/2268528023 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa268138740 0x7fa26813ab30 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa2680a1a00 tx=0x7fa26401c370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=361bbdf8563bce22 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:26:54.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.041+0000 7fa25bfff640 1 -- 172.21.15.63:0/2268528023 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2640b0070 con 0x7fa268138740 2024-04-26T04:26:54.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.041+0000 7fa25bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:54.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.041+0000 7fa25bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:26:54.052 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.041+0000 7fa25bfff640 10 monclient: dump: 2024-04-26T04:26:54.052 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:54.053 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:54.053 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:54.053 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:54.053 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:54.053 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:54.053 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:54.053 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:54.053 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:54.053 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:54.053 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.041+0000 7fa25bfff640 10 monclient: _finish_auth 0 2024-04-26T04:26:54.053 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.041+0000 7fa25bfff640 10 monclient: _check_auth_tickets 2024-04-26T04:26:54.053 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.041+0000 7fa25bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:26:54.053 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.041+0000 7fa26f784640 5 monclient: authenticate success, global_id 44121 2024-04-26T04:26:54.053 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.041+0000 7fa25bfff640 1 -- 172.21.15.63:0/2268528023 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2640ab600 con 0x7fa268138740 2024-04-26T04:26:54.053 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.041+0000 7fa25bfff640 10 monclient: handle_config config(0 keys) 2024-04-26T04:26:54.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.041+0000 7fa26f784640 10 monclient: get_monmap_and_config success 2024-04-26T04:26:54.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.041+0000 7fa26f784640 4 set_mon_vals no callback set 2024-04-26T04:26:54.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.041+0000 7fa25bfff640 1 -- 172.21.15.63:0/2268528023 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2640ab9d0 con 0x7fa268138740 2024-04-26T04:26:54.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.041+0000 7fa25bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:54.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.041+0000 7fa25bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:26:54.054 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.041+0000 7fa25bfff640 10 monclient: dump: 2024-04-26T04:26:54.054 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:54.054 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:54.054 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:54.054 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:54.054 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:54.054 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:54.054 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:54.054 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:54.055 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:54.055 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:54.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.041+0000 7fa26f784640 10 monclient: shutdown 2024-04-26T04:26:54.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.041+0000 7fa26f784640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:26:54.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.041+0000 7fa26f784640 1 -- 172.21.15.63:0/2268528023 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa268138740 msgr2=0x7fa26813ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:54.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.041+0000 7fa26f784640 1 --2- 172.21.15.63:0/2268528023 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa268138740 0x7fa26813ab30 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa2680a1a00 tx=0x7fa26401c370 comp rx=0 tx=0).stop 2024-04-26T04:26:54.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.041+0000 7fa26f784640 1 -- 172.21.15.63:0/2268528023 shutdown_connections 2024-04-26T04:26:54.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.041+0000 7fa26f784640 1 --2- 172.21.15.63:0/2268528023 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa268138740 0x7fa26813ab30 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:54.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.041+0000 7fa26f784640 1 --2- 172.21.15.63:0/2268528023 >> v2:172.21.15.186:3300/0 conn(0x7fa26809c6a0 0x7fa2680996b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:54.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.041+0000 7fa26f784640 1 -- 172.21.15.63:0/2268528023 >> 172.21.15.63:0/2268528023 conn(0x7fa2680a1e70 msgr2=0x7fa2680a2280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:26:54.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.041+0000 7fa26f784640 1 -- 172.21.15.63:0/2268528023 shutdown_connections 2024-04-26T04:26:54.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.041+0000 7fa26f784640 1 -- 172.21.15.63:0/2268528023 wait complete. 2024-04-26T04:26:54.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.045+0000 7fa26f784640 10 monclient: build_initial_monmap 2024-04-26T04:26:54.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.045+0000 7fa26f784640 10 monclient: monmap: 2024-04-26T04:26:54.055 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:26:54.056 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:26:54.056 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:26:54.056 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:26:54.056 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:26:54.056 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:54.056 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:26:54.056 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:26:54.056 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:26:54.056 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:54.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.045+0000 7fa26f784640 1 Processor -- start 2024-04-26T04:26:54.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.045+0000 7fa26f784640 1 -- start start 2024-04-26T04:26:54.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.045+0000 7fa26f784640 10 monclient: init 2024-04-26T04:26:54.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.045+0000 7fa26f784640 10 monclient: _reopen_session rank -1 2024-04-26T04:26:54.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.045+0000 7fa26f784640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T04:26:54.056 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.045+0000 7fa26f784640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa26809c6a0 0x7fa2681ccb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:54.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.045+0000 7fa26f784640 10 monclient(hunting): picked mon.noname-b con 0x7fa26809c6a0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:26:54.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.045+0000 7fa26f784640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fa26809ca80 0x7fa268134400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:54.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.045+0000 7fa26f784640 10 monclient(hunting): picked mon.noname-c con 0x7fa26809ca80 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:26:54.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.045+0000 7fa26f784640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa268138740 0x7fa268134940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:54.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.045+0000 7fa26f784640 10 monclient(hunting): picked mon.noname-a con 0x7fa268138740 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:26:54.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.045+0000 7fa26f784640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:54.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.045+0000 7fa26f784640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fa26813f520 con 0x7fa268138740 2024-04-26T04:26:54.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.045+0000 7fa26f784640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:54.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.045+0000 7fa26f784640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa26813f3a0 con 0x7fa26809c6a0 2024-04-26T04:26:54.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.045+0000 7fa26f784640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:54.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.045+0000 7fa26f784640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7fa26813f6a0 con 0x7fa26809ca80 2024-04-26T04:26:54.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.045+0000 7fa26f784640 10 monclient(hunting): _renew_subs 2024-04-26T04:26:54.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.045+0000 7fa26f784640 10 monclient(hunting): authenticate will time out at 1445.757268s 2024-04-26T04:26:54.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.045+0000 7fa26dd21640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa268138740 0x7fa268134940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:54.057 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.045+0000 7fa26dd21640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa268138740 0x7fa268134940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:55264/0 (socket says 172.21.15.63:55264) 2024-04-26T04:26:54.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.045+0000 7fa26cd1f640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fa26809ca80 0x7fa268134400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:54.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.045+0000 7fa26dd21640 1 -- 172.21.15.63:0/464213185 learned_addr learned my addr 172.21.15.63:0/464213185 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:26:54.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.045+0000 7fa26dd21640 10 monclient(hunting): get_auth_request con 0x7fa268138740 auth_method 0 2024-04-26T04:26:54.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.045+0000 7fa26dd21640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:54.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.045+0000 7fa26dd21640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:54.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.045+0000 7fa26dd21640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:54.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.045+0000 7fa26cd1f640 10 monclient(hunting): get_auth_request con 0x7fa26809ca80 auth_method 0 2024-04-26T04:26:54.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.045+0000 7fa26cd1f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:54.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.045+0000 7fa26cd1f640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:54.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.045+0000 7fa26cd1f640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:54.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.045+0000 7fa26dd21640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:26:54.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.045+0000 7fa26dd21640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:26:54.058 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.045+0000 7fa26dd21640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:26:54.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.045+0000 7fa26cd1f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:26:54.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.045+0000 7fa26cd1f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:26:54.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.045+0000 7fa26cd1f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:26:54.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.045+0000 7fa26d520640 1 --2- 172.21.15.63:0/464213185 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa26809c6a0 0x7fa2681ccb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:54.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.045+0000 7fa26dd21640 10 monclient(hunting): handle_auth_done global_id 44124 payload 824 2024-04-26T04:26:54.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.045+0000 7fa26dd21640 1 -- 172.21.15.63:0/464213185 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fa26809ca80 msgr2=0x7fa268134400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:54.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.045+0000 7fa26dd21640 1 --2- 172.21.15.63:0/464213185 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fa26809ca80 0x7fa268134400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:54.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.045+0000 7fa26dd21640 1 -- 172.21.15.63:0/464213185 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa26809c6a0 msgr2=0x7fa2681ccb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:54.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.045+0000 7fa26dd21640 1 --2- 172.21.15.63:0/464213185 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa26809c6a0 0x7fa2681ccb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:54.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.045+0000 7fa26dd21640 10 monclient: _finish_hunting 0 2024-04-26T04:26:54.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.045+0000 7fa26dd21640 1 monclient: found mon.noname-a 2024-04-26T04:26:54.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.045+0000 7fa26dd21640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:26:54.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.045+0000 7fa26dd21640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:26:54.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.045+0000 7fa26dd21640 1 -- 172.21.15.63:0/464213185 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa26812b5c0 con 0x7fa268138740 2024-04-26T04:26:54.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.045+0000 7fa26d520640 10 monclient: get_auth_request con 0x7fa26809c6a0 auth_method 0 2024-04-26T04:26:54.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.045+0000 7fa26d520640 1 --2- 172.21.15.63:0/464213185 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa26809c6a0 0x7fa2681ccb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T04:26:54.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.049+0000 7fa26dd21640 1 -- 172.21.15.63:0/464213185 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa268138740 msgr2=0x7fa268134940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:26:54.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.049+0000 7fa26dd21640 1 --2- 172.21.15.63:0/464213185 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa268138740 0x7fa268134940 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa264019fa0 tx=0x7fa264019110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:26:54.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.049+0000 7fa259ffb640 1 -- 172.21.15.63:0/464213185 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2640b0040 con 0x7fa268138740 2024-04-26T04:26:54.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.049+0000 7fa259ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:54.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.049+0000 7fa259ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:26:54.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.049+0000 7fa259ffb640 10 monclient: dump: 2024-04-26T04:26:54.063 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:54.063 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:54.063 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:54.063 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:54.063 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:54.063 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:54.063 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:54.063 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:54.063 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:54.063 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:54.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.049+0000 7fa259ffb640 10 monclient: _finish_auth 0 2024-04-26T04:26:54.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.049+0000 7fa259ffb640 10 monclient: _check_auth_tickets 2024-04-26T04:26:54.063 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.049+0000 7fa259ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:26:54.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.049+0000 7fa259ffb640 1 -- 172.21.15.63:0/464213185 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2640aa5b0 con 0x7fa268138740 2024-04-26T04:26:54.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.049+0000 7fa26f784640 5 monclient: authenticate success, global_id 44124 2024-04-26T04:26:54.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.049+0000 7fa26f784640 10 monclient: _renew_subs 2024-04-26T04:26:54.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.049+0000 7fa26f784640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:26:54.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.049+0000 7fa26f784640 1 -- 172.21.15.63:0/464213185 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa26812b8a0 con 0x7fa268138740 2024-04-26T04:26:54.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.049+0000 7fa259ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T04:26:54.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.049+0000 7fa259ffb640 1 -- 172.21.15.63:0/464213185 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2640aa960 con 0x7fa268138740 2024-04-26T04:26:54.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.049+0000 7fa259ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:54.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.049+0000 7fa259ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:26:54.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.049+0000 7fa259ffb640 10 monclient: dump: 2024-04-26T04:26:54.064 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:54.064 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:54.064 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:54.064 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:54.064 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:54.065 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:54.065 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:54.065 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:54.065 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:54.065 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:54.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.049+0000 7fa26f784640 10 monclient: _renew_subs 2024-04-26T04:26:54.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.049+0000 7fa25affd640 4 set_mon_vals no callback set 2024-04-26T04:26:54.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.049+0000 7fa26f784640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:26:54.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.049+0000 7fa26f784640 1 -- 172.21.15.63:0/464213185 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa268134e80 con 0x7fa268138740 2024-04-26T04:26:54.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.049+0000 7fa26f784640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:26:54.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.049+0000 7fa26f784640 1 --2- 172.21.15.63:0/464213185 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa268070be0 0x7fa268070fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:54.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.049+0000 7fa26f784640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:54.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.049+0000 7fa26f784640 1 -- 172.21.15.63:0/464213185 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa2680504d0 con 0x7fa268070be0 2024-04-26T04:26:54.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.049+0000 7fa259ffb640 1 -- 172.21.15.63:0/464213185 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fa2640aab00 con 0x7fa268138740 2024-04-26T04:26:54.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.049+0000 7fa26d520640 1 --2- 172.21.15.63:0/464213185 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa268070be0 0x7fa268070fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:54.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.049+0000 7fa259ffb640 1 --2- 172.21.15.63:0/464213185 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fa24c035ec0 0x7fa24c038380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:54.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.049+0000 7fa26d520640 10 monclient: get_auth_request con 0x7fa268070be0 auth_method 0 2024-04-26T04:26:54.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.049+0000 7fa26d520640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:54.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.049+0000 7fa26d520640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:54.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.049+0000 7fa26d520640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:54.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.049+0000 7fa259ffb640 1 -- 172.21.15.63:0/464213185 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(323..323 src has 1..323) ==== 4346+0+0 (secure 0 0 0) 0x7fa2640e2950 con 0x7fa268138740 2024-04-26T04:26:54.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.049+0000 7fa26d520640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:26:54.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.049+0000 7fa26d520640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:26:54.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.049+0000 7fa26d520640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:26:54.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.049+0000 7fa26cd1f640 1 --2- 172.21.15.63:0/464213185 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fa24c035ec0 0x7fa24c038380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:54.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.049+0000 7fa26cd1f640 10 monclient: get_auth_request con 0x7fa24c035ec0 auth_method 0 2024-04-26T04:26:54.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.049+0000 7fa26d520640 10 monclient(hunting): handle_auth_done global_id 34555 payload 824 2024-04-26T04:26:54.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.049+0000 7fa26d520640 1 -- 172.21.15.63:0/464213185 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa2680725a0 con 0x7fa268070be0 2024-04-26T04:26:54.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.053+0000 7fa26d520640 1 -- 172.21.15.63:0/464213185 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa268070be0 msgr2=0x7fa268070fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:26:54.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.053+0000 7fa26d520640 1 --2- 172.21.15.63:0/464213185 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa268070be0 0x7fa268070fc0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fa260009ec0 tx=0x7fa26000f520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:26:54.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.053+0000 7fa26cd1f640 1 --2- 172.21.15.63:0/464213185 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fa24c035ec0 0x7fa24c038380 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fa26809ed70 tx=0x7fa25c007e40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:26:54.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.053+0000 7fa259ffb640 1 -- 172.21.15.63:0/464213185 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa260003db0 con 0x7fa268070be0 2024-04-26T04:26:54.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.053+0000 7fa259ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:54.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.053+0000 7fa259ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T04:26:54.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.053+0000 7fa259ffb640 10 monclient: dump: 2024-04-26T04:26:54.067 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:54.067 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:54.067 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:54.067 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:54.067 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:54.067 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:54.067 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:54.068 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:54.068 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:54.068 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:54.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.053+0000 7fa259ffb640 1 -- 172.21.15.63:0/464213185 <== mon.1 v2:172.21.15.186:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa260019070 con 0x7fa268070be0 2024-04-26T04:26:54.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.053+0000 7fa259ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:26:54.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.053+0000 7fa259ffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:26:54.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.053+0000 7fa259ffb640 1 -- 172.21.15.63:0/464213185 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa268070be0 msgr2=0x7fa268070fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:54.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.053+0000 7fa259ffb640 1 --2- 172.21.15.63:0/464213185 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa268070be0 0x7fa268070fc0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fa260009ec0 tx=0x7fa26000f520 comp rx=0 tx=0).stop 2024-04-26T04:26:54.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.053+0000 7fa259ffb640 1 -- 172.21.15.63:0/464213185 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa268070be0 msgr2=0x7fa268070fc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:26:54.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.053+0000 7fa259ffb640 1 --2- 172.21.15.63:0/464213185 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa268070be0 0x7fa268070fc0 secure :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fa260009ec0 tx=0x7fa26000f520 comp rx=0 tx=0).stop 2024-04-26T04:26:54.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.057+0000 7fa26f784640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-04-26T04:26:54.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.057+0000 7fa26f784640 1 --2- 172.21.15.63:0/464213185 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa268071500 0x7fa268137530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:54.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.057+0000 7fa26f784640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:54.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.057+0000 7fa26f784640 1 -- 172.21.15.63:0/464213185 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa2681cd820 con 0x7fa268071500 2024-04-26T04:26:54.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.061+0000 7fa26d520640 1 --2- 172.21.15.63:0/464213185 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa268071500 0x7fa268137530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:54.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.061+0000 7fa26d520640 10 monclient: get_auth_request con 0x7fa268071500 auth_method 0 2024-04-26T04:26:54.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.061+0000 7fa26d520640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:54.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.061+0000 7fa26d520640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:54.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.061+0000 7fa26d520640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:54.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.061+0000 7fa26d520640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:26:54.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.061+0000 7fa26d520640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:26:54.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.061+0000 7fa26d520640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:26:54.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.061+0000 7fa26d520640 10 monclient(hunting): handle_auth_done global_id 34558 payload 824 2024-04-26T04:26:54.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.061+0000 7fa26d520640 1 -- 172.21.15.63:0/464213185 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa268062e10 con 0x7fa268071500 2024-04-26T04:26:54.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.061+0000 7fa26d520640 1 -- 172.21.15.63:0/464213185 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa268071500 msgr2=0x7fa268137530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:26:54.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.061+0000 7fa26d520640 1 --2- 172.21.15.63:0/464213185 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa268071500 0x7fa268137530 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fa260018000 tx=0x7fa260004a50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:26:54.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.065+0000 7fa259ffb640 1 -- 172.21.15.63:0/464213185 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa260004450 con 0x7fa268071500 2024-04-26T04:26:54.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.065+0000 7fa259ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:54.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.065+0000 7fa259ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T04:26:54.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.065+0000 7fa259ffb640 10 monclient: dump: 2024-04-26T04:26:54.071 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:54.071 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:54.071 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:54.072 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:54.072 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:54.072 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:54.072 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:54.072 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:54.072 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:54.072 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:54.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.065+0000 7fa259ffb640 1 -- 172.21.15.63:0/464213185 <== mon.1 v2:172.21.15.186:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4891 (secure 0 0 0) 0x7fa260004450 con 0x7fa268071500 2024-04-26T04:26:54.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.065+0000 7fa259ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-04-26T04:26:54.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.065+0000 7fa259ffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:26:54.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.065+0000 7fa259ffb640 1 -- 172.21.15.63:0/464213185 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa268071500 msgr2=0x7fa268137530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:54.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.065+0000 7fa259ffb640 1 --2- 172.21.15.63:0/464213185 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa268071500 0x7fa268137530 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fa260018000 tx=0x7fa260004a50 comp rx=0 tx=0).stop 2024-04-26T04:26:54.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.065+0000 7fa259ffb640 1 -- 172.21.15.63:0/464213185 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa268071500 msgr2=0x7fa268137530 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:26:54.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.065+0000 7fa259ffb640 1 --2- 172.21.15.63:0/464213185 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa268071500 0x7fa268137530 secure :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fa260018000 tx=0x7fa260004a50 comp rx=0 tx=0).stop 2024-04-26T04:26:54.073 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-26T04:26:54.073 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b", 2024-04-26T04:26:54.073 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 1, 2024-04-26T04:26:54.073 INFO:teuthology.orchestra.run.smithi063.stdout: "state": "peon", 2024-04-26T04:26:54.073 INFO:teuthology.orchestra.run.smithi063.stdout: "election_epoch": 42, 2024-04-26T04:26:54.073 INFO:teuthology.orchestra.run.smithi063.stdout: "uptime": 146846, 2024-04-26T04:26:54.073 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum": [ 2024-04-26T04:26:54.073 INFO:teuthology.orchestra.run.smithi063.stdout: 0, 2024-04-26T04:26:54.073 INFO:teuthology.orchestra.run.smithi063.stdout: 1, 2024-04-26T04:26:54.073 INFO:teuthology.orchestra.run.smithi063.stdout: 2 2024-04-26T04:26:54.073 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:26:54.073 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_age": 0, 2024-04-26T04:26:54.074 INFO:teuthology.orchestra.run.smithi063.stdout: "features": { 2024-04-26T04:26:54.074 INFO:teuthology.orchestra.run.smithi063.stdout: "required_con": "2450521708007866372", 2024-04-26T04:26:54.074 INFO:teuthology.orchestra.run.smithi063.stdout: "required_mon": [ 2024-04-26T04:26:54.074 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:26:54.074 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:26:54.074 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:26:54.074 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:26:54.074 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:26:54.074 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:26:54.074 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:26:54.074 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:26:54.074 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:26:54.074 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:26:54.074 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:26:54.075 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:26:54.075 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_con": "4540701547738038271", 2024-04-26T04:26:54.075 INFO:teuthology.orchestra.run.smithi063.stdout: "quorum_mon": [ 2024-04-26T04:26:54.075 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:26:54.075 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:26:54.075 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:26:54.075 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:26:54.075 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:26:54.075 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:26:54.075 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:26:54.075 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:26:54.075 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:26:54.075 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:26:54.075 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:26:54.076 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:26:54.076 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:26:54.076 INFO:teuthology.orchestra.run.smithi063.stdout: "outside_quorum": [], 2024-04-26T04:26:54.076 INFO:teuthology.orchestra.run.smithi063.stdout: "extra_probe_peers": [], 2024-04-26T04:26:54.076 INFO:teuthology.orchestra.run.smithi063.stdout: "sync_provider": [], 2024-04-26T04:26:54.076 INFO:teuthology.orchestra.run.smithi063.stdout: "monmap": { 2024-04-26T04:26:54.076 INFO:teuthology.orchestra.run.smithi063.stdout: "epoch": 1, 2024-04-26T04:26:54.076 INFO:teuthology.orchestra.run.smithi063.stdout: "fsid": "cd1c6a0c-4e4b-4014-89c0-2e535479ef39", 2024-04-26T04:26:54.076 INFO:teuthology.orchestra.run.smithi063.stdout: "modified": "2024-04-26T04:18:03.702339Z", 2024-04-26T04:26:54.076 INFO:teuthology.orchestra.run.smithi063.stdout: "created": "2024-04-26T04:18:03.702339Z", 2024-04-26T04:26:54.076 INFO:teuthology.orchestra.run.smithi063.stdout: "min_mon_release": 19, 2024-04-26T04:26:54.076 INFO:teuthology.orchestra.run.smithi063.stdout: "min_mon_release_name": "squid", 2024-04-26T04:26:54.076 INFO:teuthology.orchestra.run.smithi063.stdout: "election_strategy": 1, 2024-04-26T04:26:54.076 INFO:teuthology.orchestra.run.smithi063.stdout: "disallowed_leaders: ": "", 2024-04-26T04:26:54.076 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": false, 2024-04-26T04:26:54.077 INFO:teuthology.orchestra.run.smithi063.stdout: "tiebreaker_mon": "", 2024-04-26T04:26:54.077 INFO:teuthology.orchestra.run.smithi063.stdout: "removed_ranks: ": "", 2024-04-26T04:26:54.077 INFO:teuthology.orchestra.run.smithi063.stdout: "features": { 2024-04-26T04:26:54.077 INFO:teuthology.orchestra.run.smithi063.stdout: "persistent": [ 2024-04-26T04:26:54.077 INFO:teuthology.orchestra.run.smithi063.stdout: "kraken", 2024-04-26T04:26:54.077 INFO:teuthology.orchestra.run.smithi063.stdout: "luminous", 2024-04-26T04:26:54.077 INFO:teuthology.orchestra.run.smithi063.stdout: "mimic", 2024-04-26T04:26:54.077 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap-prune", 2024-04-26T04:26:54.077 INFO:teuthology.orchestra.run.smithi063.stdout: "nautilus", 2024-04-26T04:26:54.077 INFO:teuthology.orchestra.run.smithi063.stdout: "octopus", 2024-04-26T04:26:54.077 INFO:teuthology.orchestra.run.smithi063.stdout: "pacific", 2024-04-26T04:26:54.077 INFO:teuthology.orchestra.run.smithi063.stdout: "elector-pinging", 2024-04-26T04:26:54.077 INFO:teuthology.orchestra.run.smithi063.stdout: "quincy", 2024-04-26T04:26:54.077 INFO:teuthology.orchestra.run.smithi063.stdout: "reef", 2024-04-26T04:26:54.077 INFO:teuthology.orchestra.run.smithi063.stdout: "squid" 2024-04-26T04:26:54.078 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:26:54.078 INFO:teuthology.orchestra.run.smithi063.stdout: "optional": [] 2024-04-26T04:26:54.078 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:26:54.078 INFO:teuthology.orchestra.run.smithi063.stdout: "mons": [ 2024-04-26T04:26:54.078 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:26:54.078 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 0, 2024-04-26T04:26:54.078 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "a", 2024-04-26T04:26:54.078 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:26:54.078 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:26:54.078 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:26:54.078 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:26:54.078 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:3300", 2024-04-26T04:26:54.078 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:26:54.078 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:26:54.079 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:26:54.079 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:26:54.079 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6789", 2024-04-26T04:26:54.079 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:26:54.079 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:26:54.079 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:26:54.079 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:26:54.079 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6789/0", 2024-04-26T04:26:54.079 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6789/0", 2024-04-26T04:26:54.079 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:26:54.079 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:26:54.079 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:26:54.079 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:26:54.079 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:26:54.079 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 1, 2024-04-26T04:26:54.080 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "b", 2024-04-26T04:26:54.080 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:26:54.080 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:26:54.080 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:26:54.080 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:26:54.080 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:3300", 2024-04-26T04:26:54.080 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:26:54.080 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:26:54.080 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:26:54.080 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:26:54.080 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:6789", 2024-04-26T04:26:54.080 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:26:54.080 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:26:54.080 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:26:54.080 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:26:54.081 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.186:6789/0", 2024-04-26T04:26:54.081 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.186:6789/0", 2024-04-26T04:26:54.081 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:26:54.081 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:26:54.081 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:26:54.081 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:26:54.081 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:26:54.081 INFO:teuthology.orchestra.run.smithi063.stdout: "rank": 2, 2024-04-26T04:26:54.081 INFO:teuthology.orchestra.run.smithi063.stdout: "name": "c", 2024-04-26T04:26:54.081 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addrs": { 2024-04-26T04:26:54.081 INFO:teuthology.orchestra.run.smithi063.stdout: "addrvec": [ 2024-04-26T04:26:54.081 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:26:54.081 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v2", 2024-04-26T04:26:54.081 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:3301", 2024-04-26T04:26:54.082 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:26:54.082 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:26:54.082 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:26:54.082 INFO:teuthology.orchestra.run.smithi063.stdout: "type": "v1", 2024-04-26T04:26:54.082 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6790", 2024-04-26T04:26:54.082 INFO:teuthology.orchestra.run.smithi063.stdout: "nonce": 0 2024-04-26T04:26:54.082 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:26:54.082 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:26:54.082 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:26:54.082 INFO:teuthology.orchestra.run.smithi063.stdout: "addr": "172.21.15.63:6790/0", 2024-04-26T04:26:54.083 INFO:teuthology.orchestra.run.smithi063.stdout: "public_addr": "172.21.15.63:6790/0", 2024-04-26T04:26:54.083 INFO:teuthology.orchestra.run.smithi063.stdout: "priority": 0, 2024-04-26T04:26:54.083 INFO:teuthology.orchestra.run.smithi063.stdout: "weight": 0, 2024-04-26T04:26:54.083 INFO:teuthology.orchestra.run.smithi063.stdout: "crush_location": "{}" 2024-04-26T04:26:54.083 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:26:54.083 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:26:54.083 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:26:54.083 INFO:teuthology.orchestra.run.smithi063.stdout: "feature_map": { 2024-04-26T04:26:54.083 INFO:teuthology.orchestra.run.smithi063.stdout: "mon": [ 2024-04-26T04:26:54.083 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:26:54.083 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:26:54.083 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:26:54.083 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 1 2024-04-26T04:26:54.083 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:26:54.084 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:26:54.084 INFO:teuthology.orchestra.run.smithi063.stdout: "osd": [ 2024-04-26T04:26:54.084 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:26:54.084 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:26:54.084 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:26:54.084 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 3 2024-04-26T04:26:54.084 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:26:54.084 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:26:54.084 INFO:teuthology.orchestra.run.smithi063.stdout: "client": [ 2024-04-26T04:26:54.084 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:26:54.084 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:26:54.084 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:26:54.084 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 2 2024-04-26T04:26:54.084 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:26:54.084 INFO:teuthology.orchestra.run.smithi063.stdout: ], 2024-04-26T04:26:54.085 INFO:teuthology.orchestra.run.smithi063.stdout: "mgr": [ 2024-04-26T04:26:54.085 INFO:teuthology.orchestra.run.smithi063.stdout: { 2024-04-26T04:26:54.085 INFO:teuthology.orchestra.run.smithi063.stdout: "features": "0x3f03cffffffdffff", 2024-04-26T04:26:54.085 INFO:teuthology.orchestra.run.smithi063.stdout: "release": "luminous", 2024-04-26T04:26:54.085 INFO:teuthology.orchestra.run.smithi063.stdout: "num": 1 2024-04-26T04:26:54.085 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-26T04:26:54.085 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-26T04:26:54.085 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-26T04:26:54.085 INFO:teuthology.orchestra.run.smithi063.stdout: "stretch_mode": false 2024-04-26T04:26:54.085 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-26T04:26:54.086 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.065+0000 7fa26f784640 1 -- 172.21.15.63:0/464213185 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fa24c035ec0 msgr2=0x7fa24c038380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:54.086 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.065+0000 7fa26f784640 1 --2- 172.21.15.63:0/464213185 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fa24c035ec0 0x7fa24c038380 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fa26809ed70 tx=0x7fa25c007e40 comp rx=0 tx=0).stop 2024-04-26T04:26:54.086 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.065+0000 7fa26f784640 10 monclient: shutdown 2024-04-26T04:26:54.086 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.065+0000 7fa26f784640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:26:54.086 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.065+0000 7fa26f784640 1 -- 172.21.15.63:0/464213185 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa268138740 msgr2=0x7fa268134940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:54.086 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.065+0000 7fa26f784640 1 --2- 172.21.15.63:0/464213185 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa268138740 0x7fa268134940 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa264019fa0 tx=0x7fa264019110 comp rx=0 tx=0).stop 2024-04-26T04:26:54.087 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.065+0000 7fa26dd21640 1 -- 172.21.15.63:0/464213185 reap_dead start 2024-04-26T04:26:54.087 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.065+0000 7fa26dd21640 1 -- 172.21.15.63:0/464213185 reap_dead start 2024-04-26T04:26:54.087 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.065+0000 7fa26f784640 1 -- 172.21.15.63:0/464213185 shutdown_connections 2024-04-26T04:26:54.087 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.065+0000 7fa26f784640 1 -- 172.21.15.63:0/464213185 >> 172.21.15.63:0/464213185 conn(0x7fa2680a1e70 msgr2=0x7fa2680a2fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:26:54.087 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.065+0000 7fa26f784640 1 -- 172.21.15.63:0/464213185 shutdown_connections 2024-04-26T04:26:54.087 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.065+0000 7fa26f784640 1 -- 172.21.15.63:0/464213185 wait complete. 2024-04-26T04:26:54.088 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-04-26T04:26:54.088 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-04-26T04:26:54.088 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-04-26T04:26:54.197 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.189+0000 7ffb2b47a640 10 monclient: get_monmap_and_config 2024-04-26T04:26:54.197 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.189+0000 7ffb2b47a640 10 monclient: build_initial_monmap 2024-04-26T04:26:54.197 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.189+0000 7ffb2b47a640 10 monclient: monmap: 2024-04-26T04:26:54.197 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:26:54.197 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:54.197 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:26:54.197220+0000 2024-04-26T04:26:54.197 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:26:54.197220+0000 2024-04-26T04:26:54.197 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:26:54.197 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:54.197 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:26:54.198 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:26:54.198 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:26:54.198 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:26:54.198 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:26:54.198 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:54.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.193+0000 7ffb2b47a640 1 Processor -- start 2024-04-26T04:26:54.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.193+0000 7ffb2b47a640 1 -- start start 2024-04-26T04:26:54.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.193+0000 7ffb2b47a640 10 monclient: init 2024-04-26T04:26:54.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.193+0000 7ffb2b47a640 10 monclient: _reopen_session rank -1 2024-04-26T04:26:54.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.193+0000 7ffb2b47a640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-04-26T04:26:54.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.193+0000 7ffb2b47a640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7ffb24072490 0x7ffb24072870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:54.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.193+0000 7ffb2b47a640 10 monclient(hunting): picked mon.noname-a con 0x7ffb24072490 addr v2:172.21.15.63:3300/0 2024-04-26T04:26:54.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.193+0000 7ffb2b47a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ffb24133be0 addr v1:172.21.15.186:6789/0 2024-04-26T04:26:54.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.193+0000 7ffb2b47a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ffb240a1bb0 addr v1:172.21.15.63:6789/0 2024-04-26T04:26:54.200 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.193+0000 7ffb2b47a640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb2406e750 con 0x7ffb240a1bb0 2024-04-26T04:26:54.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.193+0000 7ffb2b47a640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb2409f930 con 0x7ffb24133be0 2024-04-26T04:26:54.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.193+0000 7ffb2b47a640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:54.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.193+0000 7ffb2b47a640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7ffb2409f4c0 con 0x7ffb24072490 2024-04-26T04:26:54.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.193+0000 7ffb2b47a640 10 monclient(hunting): _renew_subs 2024-04-26T04:26:54.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.193+0000 7ffb2b47a640 10 monclient(hunting): authenticate will time out at 1445.905417s 2024-04-26T04:26:54.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.193+0000 7ffb29216640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7ffb24072490 0x7ffb24072870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:54.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.193+0000 7ffb29a17640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7ffb240a1bb0 0x7ffb2413efe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:36742/0 (socket says 172.21.15.63:36742) 2024-04-26T04:26:54.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.193+0000 7ffb29a17640 1 -- 172.21.15.63:0/3853672618 learned_addr learned my addr 172.21.15.63:0/3853672618 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-04-26T04:26:54.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.193+0000 7ffb29216640 10 monclient(hunting): get_auth_request con 0x7ffb24072490 auth_method 0 2024-04-26T04:26:54.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.193+0000 7ffb29216640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:54.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.193+0000 7ffb29216640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:54.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.193+0000 7ffb29216640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:54.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.193+0000 7ffb29216640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:26:54.202 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.193+0000 7ffb29216640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:26:54.202 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.193+0000 7ffb29216640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:26:54.202 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.193+0000 7ffb29216640 10 monclient(hunting): handle_auth_done global_id 44127 payload 274 2024-04-26T04:26:54.202 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.193+0000 7ffb29216640 1 -- 172.21.15.63:0/3853672618 >> v1:172.21.15.186:6789/0 conn(0x7ffb24133be0 legacy=0x7ffb240a14a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:26:54.202 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.193+0000 7ffb29216640 1 -- 172.21.15.63:0/3853672618 >> v1:172.21.15.63:6789/0 conn(0x7ffb240a1bb0 legacy=0x7ffb2413efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:26:54.202 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.193+0000 7ffb29216640 10 monclient: _finish_hunting 0 2024-04-26T04:26:54.202 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.193+0000 7ffb29216640 1 monclient: found mon.noname-a 2024-04-26T04:26:54.202 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.193+0000 7ffb29216640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:26:54.202 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.193+0000 7ffb29216640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:26:54.202 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.193+0000 7ffb29216640 1 -- 172.21.15.63:0/3853672618 --> v2:172.21.15.63:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb2406ea30 con 0x7ffb24072490 2024-04-26T04:26:54.205 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.197+0000 7ffb29216640 1 -- 172.21.15.63:0/3853672618 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffb24072490 msgr2=0x7ffb24072870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:26:54.205 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.197+0000 7ffb29216640 1 --2- 172.21.15.63:0/3853672618 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffb24072490 0x7ffb24072870 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ffb200097c0 tx=0x7ffb2001c190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dd9bdd44d155fab server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:26:54.205 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.197+0000 7ffb1bfff640 1 -- 172.21.15.63:0/3853672618 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb200b0070 con 0x7ffb24072490 2024-04-26T04:26:54.205 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.197+0000 7ffb1bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:54.205 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.197+0000 7ffb1bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:26:54.205 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.197+0000 7ffb1bfff640 10 monclient: dump: 2024-04-26T04:26:54.205 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:54.206 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:54.206 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:54.206 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:54.206 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:54.206 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:54.206 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:54.206 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:54.206 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:54.206 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:54.206 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.197+0000 7ffb1bfff640 10 monclient: _finish_auth 0 2024-04-26T04:26:54.206 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.197+0000 7ffb1bfff640 10 monclient: _check_auth_tickets 2024-04-26T04:26:54.206 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.197+0000 7ffb1bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:26:54.206 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.197+0000 7ffb1bfff640 1 -- 172.21.15.63:0/3853672618 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb200ab600 con 0x7ffb24072490 2024-04-26T04:26:54.206 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.197+0000 7ffb2b47a640 5 monclient: authenticate success, global_id 44127 2024-04-26T04:26:54.207 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.197+0000 7ffb2b47a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T04:26:54.207 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.197+0000 7ffb1bfff640 10 monclient: handle_config config(0 keys) 2024-04-26T04:26:54.207 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.197+0000 7ffb2b47a640 10 monclient: get_monmap_and_config success 2024-04-26T04:26:54.207 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.197+0000 7ffb1bfff640 1 -- 172.21.15.63:0/3853672618 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb200ab9d0 con 0x7ffb24072490 2024-04-26T04:26:54.207 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.197+0000 7ffb2b47a640 4 set_mon_vals no callback set 2024-04-26T04:26:54.207 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.197+0000 7ffb1bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:54.207 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.197+0000 7ffb1bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:26:54.207 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.197+0000 7ffb1bfff640 10 monclient: dump: 2024-04-26T04:26:54.207 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:54.207 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:54.207 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:54.207 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:54.207 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:54.207 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:54.208 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:54.208 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:54.208 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:54.208 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:54.208 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.197+0000 7ffb2b47a640 10 monclient: shutdown 2024-04-26T04:26:54.208 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.197+0000 7ffb2b47a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:26:54.208 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.197+0000 7ffb2b47a640 1 -- 172.21.15.63:0/3853672618 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffb24072490 msgr2=0x7ffb24072870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:54.208 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.197+0000 7ffb2b47a640 1 --2- 172.21.15.63:0/3853672618 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffb24072490 0x7ffb24072870 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ffb200097c0 tx=0x7ffb2001c190 comp rx=0 tx=0).stop 2024-04-26T04:26:54.208 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.197+0000 7ffb2b47a640 1 -- 172.21.15.63:0/3853672618 shutdown_connections 2024-04-26T04:26:54.208 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.197+0000 7ffb2b47a640 1 --2- 172.21.15.63:0/3853672618 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffb24072490 0x7ffb24072870 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:54.208 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.197+0000 7ffb2b47a640 1 -- 172.21.15.63:0/3853672618 >> 172.21.15.63:0/3853672618 conn(0x7ffb240a6170 msgr2=0x7ffb240a2590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:26:54.208 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.197+0000 7ffb2b47a640 1 -- 172.21.15.63:0/3853672618 shutdown_connections 2024-04-26T04:26:54.208 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.197+0000 7ffb2b47a640 1 -- 172.21.15.63:0/3853672618 wait complete. 2024-04-26T04:26:54.212 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.201+0000 7ffb2b47a640 10 monclient: build_initial_monmap 2024-04-26T04:26:54.212 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.201+0000 7ffb2b47a640 10 monclient: monmap: 2024-04-26T04:26:54.212 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:26:54.212 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:26:54.212 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:26:54.212 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:26:54.213 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:26:54.213 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:54.213 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:26:54.213 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:26:54.213 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:26:54.213 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:54.213 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.201+0000 7ffb2b47a640 1 Processor -- start 2024-04-26T04:26:54.213 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.201+0000 7ffb2b47a640 1 -- start start 2024-04-26T04:26:54.213 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.201+0000 7ffb2b47a640 10 monclient: init 2024-04-26T04:26:54.213 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.201+0000 7ffb2b47a640 10 monclient: _reopen_session rank -1 2024-04-26T04:26:54.213 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.201+0000 7ffb2b47a640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T04:26:54.213 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.201+0000 7ffb2b47a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffb24072490 0x7ffb241307d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:54.213 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.201+0000 7ffb2b47a640 10 monclient(hunting): picked mon.noname-b con 0x7ffb24072490 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:26:54.213 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.201+0000 7ffb2b47a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffb240a1bb0 0x7ffb24130d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:54.214 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.201+0000 7ffb2b47a640 10 monclient(hunting): picked mon.noname-a con 0x7ffb240a1bb0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:26:54.214 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.201+0000 7ffb2b47a640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ffb24133be0 0x7ffb241397f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:54.214 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.201+0000 7ffb2b47a640 10 monclient(hunting): picked mon.noname-c con 0x7ffb24133be0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:26:54.214 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.201+0000 7ffb2b47a640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:54.214 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.201+0000 7ffb2b47a640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ffb241430a0 con 0x7ffb240a1bb0 2024-04-26T04:26:54.214 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.201+0000 7ffb2b47a640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:54.214 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.201+0000 7ffb2b47a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ffb24142f20 con 0x7ffb24072490 2024-04-26T04:26:54.214 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.201+0000 7ffb2b47a640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:54.214 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.201+0000 7ffb2b47a640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7ffb24143220 con 0x7ffb24133be0 2024-04-26T04:26:54.215 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.201+0000 7ffb2b47a640 10 monclient(hunting): _renew_subs 2024-04-26T04:26:54.215 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.201+0000 7ffb2b47a640 10 monclient(hunting): authenticate will time out at 1445.913373s 2024-04-26T04:26:54.215 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.201+0000 7ffb28a15640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffb240a1bb0 0x7ffb24130d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:54.215 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.201+0000 7ffb29a17640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ffb24133be0 0x7ffb241397f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:54.215 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.201+0000 7ffb28a15640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffb240a1bb0 0x7ffb24130d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:55274/0 (socket says 172.21.15.63:55274) 2024-04-26T04:26:54.215 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.201+0000 7ffb28a15640 1 -- 172.21.15.63:0/3916983066 learned_addr learned my addr 172.21.15.63:0/3916983066 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:26:54.215 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.201+0000 7ffb28a15640 10 monclient(hunting): get_auth_request con 0x7ffb240a1bb0 auth_method 0 2024-04-26T04:26:54.215 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.201+0000 7ffb28a15640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:54.215 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.201+0000 7ffb28a15640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:54.215 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.201+0000 7ffb28a15640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:54.215 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.201+0000 7ffb29a17640 10 monclient(hunting): get_auth_request con 0x7ffb24133be0 auth_method 0 2024-04-26T04:26:54.215 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.201+0000 7ffb29a17640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:54.215 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.201+0000 7ffb29a17640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:54.215 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.201+0000 7ffb29a17640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:54.216 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.201+0000 7ffb28a15640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:26:54.216 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.201+0000 7ffb28a15640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:26:54.216 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.201+0000 7ffb28a15640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:26:54.216 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.201+0000 7ffb29a17640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:26:54.216 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.201+0000 7ffb29a17640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:26:54.216 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.201+0000 7ffb29a17640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:26:54.216 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.201+0000 7ffb29216640 1 --2- 172.21.15.63:0/3916983066 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffb24072490 0x7ffb241307d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:54.216 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.201+0000 7ffb29216640 10 monclient(hunting): get_auth_request con 0x7ffb24072490 auth_method 0 2024-04-26T04:26:54.216 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.201+0000 7ffb29216640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:54.216 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.201+0000 7ffb29216640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:54.216 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.201+0000 7ffb29216640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:54.216 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.201+0000 7ffb28a15640 10 monclient(hunting): handle_auth_done global_id 44130 payload 824 2024-04-26T04:26:54.216 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.201+0000 7ffb28a15640 1 -- 172.21.15.63:0/3916983066 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ffb24133be0 msgr2=0x7ffb241397f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:54.216 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.201+0000 7ffb28a15640 1 --2- 172.21.15.63:0/3916983066 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ffb24133be0 0x7ffb241397f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:54.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.201+0000 7ffb28a15640 1 -- 172.21.15.63:0/3916983066 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffb24072490 msgr2=0x7ffb241307d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:54.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.201+0000 7ffb28a15640 1 --2- 172.21.15.63:0/3916983066 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffb24072490 0x7ffb241307d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:54.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.201+0000 7ffb28a15640 10 monclient: _finish_hunting 0 2024-04-26T04:26:54.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.201+0000 7ffb28a15640 1 monclient: found mon.noname-a 2024-04-26T04:26:54.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.201+0000 7ffb28a15640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:26:54.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.201+0000 7ffb28a15640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:26:54.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.201+0000 7ffb28a15640 1 -- 172.21.15.63:0/3916983066 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb2413d170 con 0x7ffb240a1bb0 2024-04-26T04:26:54.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.201+0000 7ffb29a17640 1 --2- 172.21.15.63:0/3916983066 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ffb24133be0 0x7ffb241397f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T04:26:54.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.205+0000 7ffb28a15640 1 -- 172.21.15.63:0/3916983066 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffb240a1bb0 msgr2=0x7ffb24130d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:26:54.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.205+0000 7ffb28a15640 1 --2- 172.21.15.63:0/3916983066 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffb240a1bb0 0x7ffb24130d10 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ffb1400c450 tx=0x7ffb1400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:26:54.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.205+0000 7ffb19ffb640 1 -- 172.21.15.63:0/3916983066 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb14012a40 con 0x7ffb240a1bb0 2024-04-26T04:26:54.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.205+0000 7ffb19ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:54.217 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.205+0000 7ffb19ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:26:54.218 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.205+0000 7ffb19ffb640 10 monclient: dump: 2024-04-26T04:26:54.218 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:54.218 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:54.218 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:54.218 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:54.218 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:54.218 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:54.218 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:54.218 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:54.218 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:54.218 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:54.218 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.205+0000 7ffb19ffb640 10 monclient: _finish_auth 0 2024-04-26T04:26:54.218 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.205+0000 7ffb19ffb640 10 monclient: _check_auth_tickets 2024-04-26T04:26:54.218 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.205+0000 7ffb19ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:26:54.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.205+0000 7ffb19ffb640 1 -- 172.21.15.63:0/3916983066 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb14017070 con 0x7ffb240a1bb0 2024-04-26T04:26:54.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.205+0000 7ffb19ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T04:26:54.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.205+0000 7ffb19ffb640 1 -- 172.21.15.63:0/3916983066 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb14012e40 con 0x7ffb240a1bb0 2024-04-26T04:26:54.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.205+0000 7ffb2b47a640 5 monclient: authenticate success, global_id 44130 2024-04-26T04:26:54.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.205+0000 7ffb2b47a640 10 monclient: _renew_subs 2024-04-26T04:26:54.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.205+0000 7ffb2b47a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:26:54.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.205+0000 7ffb2b47a640 1 -- 172.21.15.63:0/3916983066 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb2413d450 con 0x7ffb240a1bb0 2024-04-26T04:26:54.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.205+0000 7ffb19ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:54.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.205+0000 7ffb1b7fe640 4 set_mon_vals no callback set 2024-04-26T04:26:54.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.205+0000 7ffb19ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:26:54.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.205+0000 7ffb19ffb640 10 monclient: dump: 2024-04-26T04:26:54.219 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:54.219 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:54.219 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:54.220 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:54.220 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:54.220 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:54.220 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:54.220 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:54.220 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:54.220 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:54.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.205+0000 7ffb2b47a640 10 monclient: _renew_subs 2024-04-26T04:26:54.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.205+0000 7ffb2b47a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:26:54.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.205+0000 7ffb2b47a640 1 -- 172.21.15.63:0/3916983066 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb2413bd20 con 0x7ffb240a1bb0 2024-04-26T04:26:54.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.205+0000 7ffb19ffb640 1 -- 172.21.15.63:0/3916983066 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7ffb1401c460 con 0x7ffb240a1bb0 2024-04-26T04:26:54.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.205+0000 7ffb2b47a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:26:54.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.205+0000 7ffb2b47a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:26:54.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.205+0000 7ffb2b47a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:26:54.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.205+0000 7ffb2b47a640 1 -- 172.21.15.63:0/3916983066 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb24062e10 con 0x7ffb240a1bb0 2024-04-26T04:26:54.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.205+0000 7ffb19ffb640 1 --2- 172.21.15.63:0/3916983066 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7ffb08035ec0 0x7ffb08038380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:54.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.205+0000 7ffb19ffb640 1 -- 172.21.15.63:0/3916983066 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(323..323 src has 1..323) ==== 4346+0+0 (secure 0 0 0) 0x7ffb14048a60 con 0x7ffb240a1bb0 2024-04-26T04:26:54.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.205+0000 7ffb29216640 1 --2- 172.21.15.63:0/3916983066 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7ffb08035ec0 0x7ffb08038380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:54.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.205+0000 7ffb29216640 10 monclient: get_auth_request con 0x7ffb08035ec0 auth_method 0 2024-04-26T04:26:54.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.209+0000 7ffb29216640 1 --2- 172.21.15.63:0/3916983066 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7ffb08035ec0 0x7ffb08038380 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7ffb2406b150 tx=0x7ffb200a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:26:54.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.209+0000 7ffb19ffb640 1 -- 172.21.15.63:0/3916983066 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7ffb1404ae20 con 0x7ffb240a1bb0 2024-04-26T04:26:54.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.209+0000 7ffb19ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:26:54.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.209+0000 7ffb19ffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:26:54.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.437+0000 7ffb2b47a640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-04-26T04:26:54.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.437+0000 7ffb2b47a640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-04-26T04:26:54.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.437+0000 7ffb2b47a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:26:54.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.437+0000 7ffb2b47a640 1 -- 172.21.15.63:0/3916983066 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7ffb24068260 con 0x7ffb240a1bb0 2024-04-26T04:26:54.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.437+0000 7ffb19ffb640 1 -- 172.21.15.63:0/3916983066 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7ffb1401c9e0 con 0x7ffb240a1bb0 2024-04-26T04:26:54.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.437+0000 7ffb19ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-04-26T04:26:54.444 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.437+0000 7ffb19ffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:26:54.445 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.437+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:26:54.445 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.437+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=11,config=2,health=11,kv=12,logm=64} crc {auth=2900978055,config=2586702516,health=3602034646,kv=3642618011,logm=452689929}) 2024-04-26T04:26:54.445 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.437+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=12,config=2,health=12,kv=12,logm=62} crc {auth=2372661172,config=2586702516,health=1361790972,kv=3642618011,logm=297721901}) 2024-04-26T04:26:54.445 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.437+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:26:54.445 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.437+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=11,config=2,health=11,kv=12,logm=64} crc {auth=2900978055,config=2586702516,health=3602034646,kv=3642618011,logm=452689929}) 2024-04-26T04:26:54.445 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.437+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=12,config=2,health=11,kv=11,logm=64} crc {auth=1862920215,config=2586702516,health=3602034646,kv=4042758005,logm=2425642976}) 2024-04-26T04:26:54.446 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.441+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:26:54.446 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.441+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=121174379}) 2024-04-26T04:26:54.446 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.441+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4102033918}) 2024-04-26T04:26:54.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.441+0000 7ffb2b47a640 1 -- 172.21.15.63:0/3916983066 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7ffb08035ec0 msgr2=0x7ffb08038380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:54.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.441+0000 7ffb2b47a640 1 --2- 172.21.15.63:0/3916983066 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7ffb08035ec0 0x7ffb08038380 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7ffb2406b150 tx=0x7ffb200a6000 comp rx=0 tx=0).stop 2024-04-26T04:26:54.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.441+0000 7ffb2b47a640 10 monclient: shutdown 2024-04-26T04:26:54.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.441+0000 7ffb2b47a640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:26:54.447 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.441+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:26:54.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.441+0000 7ffb2b47a640 1 -- 172.21.15.63:0/3916983066 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffb240a1bb0 msgr2=0x7ffb24130d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:54.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.441+0000 7ffb2b47a640 1 --2- 172.21.15.63:0/3916983066 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffb240a1bb0 0x7ffb24130d10 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ffb1400c450 tx=0x7ffb1400c910 comp rx=0 tx=0).stop 2024-04-26T04:26:54.448 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.441+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=121174379}) 2024-04-26T04:26:54.448 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.441+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=755149363}) 2024-04-26T04:26:54.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.441+0000 7ffb2b47a640 1 -- 172.21.15.63:0/3916983066 shutdown_connections 2024-04-26T04:26:54.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.441+0000 7ffb2b47a640 1 --2- 172.21.15.63:0/3916983066 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7ffb08035ec0 0x7ffb08038380 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:54.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.441+0000 7ffb2b47a640 1 --2- 172.21.15.63:0/3916983066 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ffb24133be0 0x7ffb241397f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:54.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.441+0000 7ffb2b47a640 1 --2- 172.21.15.63:0/3916983066 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffb240a1bb0 0x7ffb24130d10 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:54.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.441+0000 7ffb2b47a640 1 --2- 172.21.15.63:0/3916983066 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffb24072490 0x7ffb241307d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:54.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.441+0000 7ffb2b47a640 1 -- 172.21.15.63:0/3916983066 >> 172.21.15.63:0/3916983066 conn(0x7ffb240a6170 msgr2=0x7ffb240711d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:26:54.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.441+0000 7ffb2b47a640 1 -- 172.21.15.63:0/3916983066 shutdown_connections 2024-04-26T04:26:54.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:54.441+0000 7ffb2b47a640 1 -- 172.21.15.63:0/3916983066 wait complete. 2024-04-26T04:26:54.450 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.441+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:26:54.451 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.441+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1072496634}) 2024-04-26T04:26:54.451 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.441+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1818731739}) 2024-04-26T04:26:54.451 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.441+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:26:54.451 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.441+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1072496634}) 2024-04-26T04:26:54.451 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.441+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1788920044}) 2024-04-26T04:26:54.453 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.445+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:26:54.453 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.445+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2188758666}) 2024-04-26T04:26:54.453 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.445+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=119640628}) 2024-04-26T04:26:54.453 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.445+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:26:54.454 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.445+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2188758666}) 2024-04-26T04:26:54.454 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.445+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2426535977}) 2024-04-26T04:26:54.457 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.449+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:26:54.457 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.449+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2122882224}) 2024-04-26T04:26:54.458 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.449+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=740222015}) 2024-04-26T04:26:54.458 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.449+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:26:54.458 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.449+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2122882224}) 2024-04-26T04:26:54.458 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.449+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=716878388}) 2024-04-26T04:26:54.460 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.453+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:26:54.460 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.453+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4143061146}) 2024-04-26T04:26:54.460 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.453+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2962545241}) 2024-04-26T04:26:54.460 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.453+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:26:54.460 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.453+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4143061146}) 2024-04-26T04:26:54.461 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.453+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2632595178}) 2024-04-26T04:26:54.462 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-04-26T04:26:54.468 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.461+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:26:54.469 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.461+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=110301892}) 2024-04-26T04:26:54.469 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.461+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=137222793}) 2024-04-26T04:26:54.469 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.461+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:26:54.469 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.461+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=110301892}) 2024-04-26T04:26:54.469 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.461+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1777156861}) 2024-04-26T04:26:54.472 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.465+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:26:54.472 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.465+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2239985164}) 2024-04-26T04:26:54.472 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.465+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=286351534}) 2024-04-26T04:26:54.472 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.465+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:26:54.472 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.465+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2239985164}) 2024-04-26T04:26:54.472 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.465+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1051532153}) 2024-04-26T04:26:54.475 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.469+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:26:54.475 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.469+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1058434478}) 2024-04-26T04:26:54.475 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.469+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=969579579}) 2024-04-26T04:26:54.475 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.469+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:26:54.475 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.469+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1058434478}) 2024-04-26T04:26:54.475 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.469+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1231582589}) 2024-04-26T04:26:54.478 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.473+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:26:54.478 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.473+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=430264306}) 2024-04-26T04:26:54.478 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.473+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2965080509}) 2024-04-26T04:26:54.478 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.473+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:26:54.478 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.473+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=430264306}) 2024-04-26T04:26:54.478 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.473+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1311001123}) 2024-04-26T04:26:54.481 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.473+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:26:54.481 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.473+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4267507889}) 2024-04-26T04:26:54.481 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.473+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2802410891}) 2024-04-26T04:26:54.481 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.473+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:26:54.481 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.473+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4267507889}) 2024-04-26T04:26:54.481 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.473+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3641009533}) 2024-04-26T04:26:54.484 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.477+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:26:54.484 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.477+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1296798835}) 2024-04-26T04:26:54.485 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.477+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1187443970}) 2024-04-26T04:26:54.485 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.477+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:26:54.485 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.477+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1296798835}) 2024-04-26T04:26:54.485 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.477+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1574951001}) 2024-04-26T04:26:54.487 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.481+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:26:54.488 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.481+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2106923550}) 2024-04-26T04:26:54.488 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.481+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3760513853}) 2024-04-26T04:26:54.488 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.481+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:26:54.488 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.481+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2106923550}) 2024-04-26T04:26:54.488 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.481+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1323177796}) 2024-04-26T04:26:54.491 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.485+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:26:54.491 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.485+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2346216079}) 2024-04-26T04:26:54.491 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.485+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2690191801}) 2024-04-26T04:26:54.491 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.485+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:26:54.491 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.485+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2346216079}) 2024-04-26T04:26:54.491 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.485+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1984580954}) 2024-04-26T04:26:54.495 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.489+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:26:54.495 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.489+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=300400686}) 2024-04-26T04:26:54.495 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.489+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=484567067}) 2024-04-26T04:26:54.495 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.489+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:26:54.495 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.489+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=300400686}) 2024-04-26T04:26:54.495 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.489+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=354315196}) 2024-04-26T04:26:54.496 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.489+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:26:54.496 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.489+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1948496152}) 2024-04-26T04:26:54.496 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.489+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=539588555}) 2024-04-26T04:26:54.497 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.489+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:26:54.497 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.489+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1948496152}) 2024-04-26T04:26:54.497 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.489+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3659046221}) 2024-04-26T04:26:54.499 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.493+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:26:54.499 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.493+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=81,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=6} crc {logm=261630435,mdsmap=2840952586,mgr=4110609431,mgr_command_descs=3057339438,mgr_metadata=3709105003,mgrstat=3294465371}) 2024-04-26T04:26:54.499 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.493+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=86,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=1} crc {logm=416253005,mdsmap=2840952586,mgr=4110609431,mgr_command_descs=3057339438,mgr_metadata=3709105003,mgrstat=3384167079}) 2024-04-26T04:26:54.499 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.493+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:26:54.499 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.493+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=81,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=6} crc {logm=261630435,mdsmap=2840952586,mgr=4110609431,mgr_command_descs=3057339438,mgr_metadata=3709105003,mgrstat=3294465371}) 2024-04-26T04:26:54.499 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.493+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=83,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=4} crc {logm=1105413762,mdsmap=2840952586,mgr=4110609431,mgr_command_descs=3057339438,mgr_metadata=3709105003,mgrstat=40953660}) 2024-04-26T04:26:54.503 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.493+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:26:54.503 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.493+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=4,mon_config_key=16,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=70} crc {mgrstat=2656142496,mon_config_key=620695095,monmap=1192322635,osd_metadata=1397308133,osd_pg_creating=2200595312,osdmap=1499003702}) 2024-04-26T04:26:54.503 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.493+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgrstat=4,mon_config_key=16,monmap=3,osd_metadata=4,osd_pg_creating=1,osdmap=72} crc {mgrstat=2656142496,mon_config_key=1316903946,monmap=1192322635,osd_metadata=162918814,osd_pg_creating=2200595312,osdmap=79333716}) 2024-04-26T04:26:54.503 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.493+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:26:54.503 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.493+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=4,mon_config_key=16,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=70} crc {mgrstat=2656142496,mon_config_key=620695095,monmap=1192322635,osd_metadata=1397308133,osd_pg_creating=2200595312,osdmap=1499003702}) 2024-04-26T04:26:54.503 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.493+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgrstat=4,mon_config_key=17,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=69} crc {mgrstat=2656142496,mon_config_key=3978598243,monmap=1192322635,osd_metadata=1082025948,osd_pg_creating=2200595312,osdmap=2354858992}) 2024-04-26T04:26:54.504 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.497+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:26:54.504 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.497+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2084308974}) 2024-04-26T04:26:54.504 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.497+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=750634316}) 2024-04-26T04:26:54.504 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.497+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:26:54.504 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.497+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2084308974}) 2024-04-26T04:26:54.504 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.497+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3996337856}) 2024-04-26T04:26:54.507 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.501+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:26:54.507 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.501+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1272770118}) 2024-04-26T04:26:54.507 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.501+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=355669731}) 2024-04-26T04:26:54.507 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.501+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:26:54.507 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.501+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1272770118}) 2024-04-26T04:26:54.507 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.501+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=4285009627}) 2024-04-26T04:26:54.509 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.501+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:26:54.509 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.501+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2422218037}) 2024-04-26T04:26:54.509 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.501+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3607255695}) 2024-04-26T04:26:54.509 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.501+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:26:54.509 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.501+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2422218037}) 2024-04-26T04:26:54.509 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.501+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=567159997}) 2024-04-26T04:26:54.513 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.505+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:26:54.513 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.505+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1006408859}) 2024-04-26T04:26:54.513 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.505+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=97} crc {osdmap=2904883521}) 2024-04-26T04:26:54.513 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.505+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:26:54.513 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.505+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1006408859}) 2024-04-26T04:26:54.513 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.505+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1161450391}) 2024-04-26T04:26:54.514 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.509+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:26:54.514 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.509+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=8} crc {osdmap=1397884071}) 2024-04-26T04:26:54.514 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.509+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=8} crc {osdmap=2190537778}) 2024-04-26T04:26:54.514 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.509+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-04-26T04:26:54.514 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.509+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=8} crc {osdmap=1397884071}) 2024-04-26T04:26:54.514 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-04-26T04:26:54.509+0000 7f3b42474640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=7} crc {osdmap=1647959016}) 2024-04-26T04:26:55.463 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-04-26T04:26:55.464 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-04-26T04:26:55.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.589+0000 7f6854867640 10 monclient: get_monmap_and_config 2024-04-26T04:26:55.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.589+0000 7f6854867640 10 monclient: build_initial_monmap 2024-04-26T04:26:55.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.589+0000 7f6854867640 10 monclient: monmap: 2024-04-26T04:26:55.597 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:26:55.597 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:55.598 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:26:55.597288+0000 2024-04-26T04:26:55.598 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:26:55.597288+0000 2024-04-26T04:26:55.598 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:26:55.598 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:55.598 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:26:55.598 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:26:55.598 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:26:55.598 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:26:55.598 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:26:55.598 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:55.600 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.593+0000 7f6854867640 1 Processor -- start 2024-04-26T04:26:55.600 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.593+0000 7f6854867640 1 -- start start 2024-04-26T04:26:55.600 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.593+0000 7f6854867640 10 monclient: init 2024-04-26T04:26:55.601 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.593+0000 7f6854867640 10 monclient: _reopen_session rank -1 2024-04-26T04:26:55.601 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.593+0000 7f6854867640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-04-26T04:26:55.601 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.593+0000 7f6854867640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f68500a2cd0 0x7f68500a30b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:55.601 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.593+0000 7f6854867640 10 monclient(hunting): picked mon.noname-b con 0x7f68500a2cd0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:26:55.601 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.593+0000 7f6854867640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f68500a47d0 0x7f6850136ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:55.601 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.593+0000 7f6854867640 10 monclient(hunting): picked mon.noname-a con 0x7f68500a47d0 addr v2:172.21.15.63:3300/0 2024-04-26T04:26:55.601 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.593+0000 7f6854867640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f68500a4bd0 0x7f6850137220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:55.601 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.593+0000 7f6854867640 10 monclient(hunting): picked mon.noname-c con 0x7f68500a4bd0 addr v2:172.21.15.186:3300/0 2024-04-26T04:26:55.601 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.593+0000 7f6854867640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:55.601 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.593+0000 7f6854867640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7f68500a35f0 con 0x7f68500a47d0 2024-04-26T04:26:55.602 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.593+0000 7f6854867640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:55.602 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.593+0000 7f6854867640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f68500a0fb0 con 0x7f68500a4bd0 2024-04-26T04:26:55.602 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.593+0000 7f6854867640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:55.602 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.593+0000 7f6854867640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f68500a5380 con 0x7f68500a2cd0 2024-04-26T04:26:55.602 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.593+0000 7f6854867640 10 monclient(hunting): _renew_subs 2024-04-26T04:26:55.602 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.593+0000 7f6854867640 10 monclient(hunting): authenticate will time out at 1447.305069s 2024-04-26T04:26:55.602 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.593+0000 7f684e59c640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f68500a2cd0 0x7f68500a30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:55.602 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.593+0000 7f684e59c640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f68500a2cd0 0x7f68500a30b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:41054/0 (socket says 172.21.15.63:41054) 2024-04-26T04:26:55.602 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.593+0000 7f684e59c640 1 -- 172.21.15.63:0/3035823192 learned_addr learned my addr 172.21.15.63:0/3035823192 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:26:55.602 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.593+0000 7f684e59c640 10 monclient(hunting): get_auth_request con 0x7f68500a2cd0 auth_method 0 2024-04-26T04:26:55.602 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.593+0000 7f684e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:55.602 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.593+0000 7f684e59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:55.602 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.593+0000 7f684e59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:55.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.593+0000 7f6847fff640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f68500a47d0 0x7f6850136ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:55.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.593+0000 7f6847fff640 10 monclient(hunting): get_auth_request con 0x7f68500a47d0 auth_method 0 2024-04-26T04:26:55.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.593+0000 7f6847fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:55.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.593+0000 7f6847fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:55.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.593+0000 7f6847fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:55.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.593+0000 7f684e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:26:55.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.593+0000 7f684e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:26:55.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.593+0000 7f684e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:26:55.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.593+0000 7f6847fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:26:55.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.593+0000 7f6847fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:26:55.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.593+0000 7f6847fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:26:55.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.593+0000 7f684ed9d640 1 --2- 172.21.15.63:0/3035823192 >> v2:172.21.15.186:3300/0 conn(0x7f68500a4bd0 0x7f6850137220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:55.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.593+0000 7f684e59c640 10 monclient(hunting): handle_auth_done global_id 24695 payload 274 2024-04-26T04:26:55.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.593+0000 7f684e59c640 1 -- 172.21.15.63:0/3035823192 >> v2:172.21.15.186:3300/0 conn(0x7f68500a4bd0 msgr2=0x7f6850137220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:26:55.603 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.593+0000 7f684e59c640 1 --2- 172.21.15.63:0/3035823192 >> v2:172.21.15.186:3300/0 conn(0x7f68500a4bd0 0x7f6850137220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:55.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.593+0000 7f684e59c640 1 -- 172.21.15.63:0/3035823192 >> v2:172.21.15.63:3300/0 conn(0x7f68500a47d0 msgr2=0x7f6850136ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:26:55.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.593+0000 7f684e59c640 1 --2- 172.21.15.63:0/3035823192 >> v2:172.21.15.63:3300/0 conn(0x7f68500a47d0 0x7f6850136ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:55.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.593+0000 7f684e59c640 10 monclient: _finish_hunting 0 2024-04-26T04:26:55.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.593+0000 7f684e59c640 1 monclient: found mon.noname-b 2024-04-26T04:26:55.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.593+0000 7f684e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:26:55.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.593+0000 7f684e59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.63:3301/0 2024-04-26T04:26:55.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.593+0000 7f684e59c640 1 -- 172.21.15.63:0/3035823192 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68500a1290 con 0x7f68500a2cd0 2024-04-26T04:26:55.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.597+0000 7f684e59c640 1 -- 172.21.15.63:0/3035823192 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f68500a2cd0 msgr2=0x7f68500a30b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:26:55.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.597+0000 7f684e59c640 1 --2- 172.21.15.63:0/3035823192 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f68500a2cd0 0x7f68500a30b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f6850137d80 tx=0x7f684001c470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=deb922329e0770b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:26:55.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.597+0000 7f684dd9b640 1 -- 172.21.15.63:0/3035823192 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68400b0070 con 0x7f68500a2cd0 2024-04-26T04:26:55.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.597+0000 7f684dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:55.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.597+0000 7f684dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T04:26:55.609 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.597+0000 7f684dd9b640 10 monclient: dump: 2024-04-26T04:26:55.609 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:55.609 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:55.609 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:55.609 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:55.609 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:55.609 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:55.609 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:55.609 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:55.609 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:55.609 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:55.609 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.597+0000 7f684dd9b640 10 monclient: _finish_auth 0 2024-04-26T04:26:55.609 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.597+0000 7f684dd9b640 10 monclient: _check_auth_tickets 2024-04-26T04:26:55.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.597+0000 7f684dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:26:55.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.597+0000 7f6854867640 5 monclient: authenticate success, global_id 24695 2024-04-26T04:26:55.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.597+0000 7f6854867640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T04:26:55.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.597+0000 7f684dd9b640 1 -- 172.21.15.63:0/3035823192 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68400ab600 con 0x7f68500a2cd0 2024-04-26T04:26:55.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.597+0000 7f684dd9b640 10 monclient: handle_config config(0 keys) 2024-04-26T04:26:55.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.597+0000 7f684dd9b640 1 -- 172.21.15.63:0/3035823192 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68400ab9d0 con 0x7f68500a2cd0 2024-04-26T04:26:55.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.597+0000 7f6854867640 10 monclient: get_monmap_and_config success 2024-04-26T04:26:55.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.597+0000 7f6854867640 4 set_mon_vals no callback set 2024-04-26T04:26:55.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.597+0000 7f684dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:55.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.597+0000 7f684dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:26:55.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.597+0000 7f684dd9b640 10 monclient: dump: 2024-04-26T04:26:55.610 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:55.610 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:55.611 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:55.611 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:55.611 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:55.611 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:55.611 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:55.611 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:55.611 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:55.611 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:55.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.597+0000 7f6854867640 10 monclient: shutdown 2024-04-26T04:26:55.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.597+0000 7f6854867640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:26:55.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.597+0000 7f6854867640 1 -- 172.21.15.63:0/3035823192 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f68500a2cd0 msgr2=0x7f68500a30b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:55.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.597+0000 7f6854867640 1 --2- 172.21.15.63:0/3035823192 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f68500a2cd0 0x7f68500a30b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f6850137d80 tx=0x7f684001c470 comp rx=0 tx=0).stop 2024-04-26T04:26:55.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.597+0000 7f6854867640 1 -- 172.21.15.63:0/3035823192 shutdown_connections 2024-04-26T04:26:55.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.597+0000 7f6854867640 1 --2- 172.21.15.63:0/3035823192 >> v2:172.21.15.186:3300/0 conn(0x7f68500a4bd0 0x7f6850137220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:55.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.597+0000 7f6854867640 1 --2- 172.21.15.63:0/3035823192 >> v2:172.21.15.63:3300/0 conn(0x7f68500a47d0 0x7f6850136ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:55.612 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.597+0000 7f6854867640 1 --2- 172.21.15.63:0/3035823192 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f68500a2cd0 0x7f68500a30b0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:55.612 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.597+0000 7f6854867640 1 -- 172.21.15.63:0/3035823192 >> 172.21.15.63:0/3035823192 conn(0x7f68500a5f40 msgr2=0x7f68500a6320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:26:55.612 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.597+0000 7f6854867640 1 -- 172.21.15.63:0/3035823192 shutdown_connections 2024-04-26T04:26:55.612 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.597+0000 7f6854867640 1 -- 172.21.15.63:0/3035823192 wait complete. 2024-04-26T04:26:55.612 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.597+0000 7f6854867640 10 monclient: build_initial_monmap 2024-04-26T04:26:55.612 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.597+0000 7f6854867640 10 monclient: monmap: 2024-04-26T04:26:55.612 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:26:55.612 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:26:55.612 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:26:55.613 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:26:55.613 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:26:55.613 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:55.613 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:26:55.613 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:26:55.613 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:26:55.613 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:55.613 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.597+0000 7f6854867640 1 Processor -- start 2024-04-26T04:26:55.613 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.597+0000 7f6854867640 1 -- start start 2024-04-26T04:26:55.613 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.597+0000 7f6854867640 10 monclient: init 2024-04-26T04:26:55.613 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.601+0000 7f6854867640 10 monclient: _reopen_session rank -1 2024-04-26T04:26:55.613 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.601+0000 7f6854867640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T04:26:55.613 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.601+0000 7f6854867640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f68500a2cd0 0x7f68501cee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:55.613 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.601+0000 7f6854867640 10 monclient(hunting): picked mon.noname-b con 0x7f68500a2cd0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:26:55.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.601+0000 7f6854867640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f68500a47d0 0x7f6850135be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:55.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.601+0000 7f6854867640 10 monclient(hunting): picked mon.noname-c con 0x7f68500a47d0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:26:55.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.601+0000 7f6854867640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f68500a4bd0 0x7f685013bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:55.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.601+0000 7f6854867640 10 monclient(hunting): picked mon.noname-a con 0x7f68500a4bd0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:26:55.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.601+0000 7f6854867640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:55.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.601+0000 7f6854867640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f6850140a40 con 0x7f68500a4bd0 2024-04-26T04:26:55.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.601+0000 7f6854867640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:55.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.601+0000 7f6854867640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6850137b70 con 0x7f68500a2cd0 2024-04-26T04:26:55.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.601+0000 7f6854867640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:55.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.601+0000 7f6854867640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f6850140bc0 con 0x7f68500a47d0 2024-04-26T04:26:55.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.601+0000 7f6854867640 10 monclient(hunting): _renew_subs 2024-04-26T04:26:55.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.601+0000 7f6854867640 10 monclient(hunting): authenticate will time out at 1447.312445s 2024-04-26T04:26:55.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.601+0000 7f6847fff640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f68500a47d0 0x7f6850135be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:55.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.601+0000 7f6847fff640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f68500a47d0 0x7f6850135be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:41056/0 (socket says 172.21.15.63:41056) 2024-04-26T04:26:55.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.601+0000 7f6847fff640 1 -- 172.21.15.63:0/1255595660 learned_addr learned my addr 172.21.15.63:0/1255595660 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:26:55.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.601+0000 7f6847fff640 10 monclient(hunting): get_auth_request con 0x7f68500a47d0 auth_method 0 2024-04-26T04:26:55.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.601+0000 7f6847fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:55.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.601+0000 7f6847fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:55.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.601+0000 7f6847fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:55.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.601+0000 7f684ed9d640 1 --2- 172.21.15.63:0/1255595660 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f68500a4bd0 0x7f685013bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:55.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.601+0000 7f684ed9d640 10 monclient(hunting): get_auth_request con 0x7f68500a4bd0 auth_method 0 2024-04-26T04:26:55.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.601+0000 7f684ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:55.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.601+0000 7f684ed9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:55.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.601+0000 7f684ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:55.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.601+0000 7f6847fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:26:55.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.601+0000 7f6847fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:26:55.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.601+0000 7f6847fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:26:55.616 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.601+0000 7f684ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:26:55.616 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.601+0000 7f684ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:26:55.616 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.601+0000 7f684ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:26:55.616 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.601+0000 7f684e59c640 1 --2- 172.21.15.63:0/1255595660 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f68500a2cd0 0x7f68501cee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:55.616 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.601+0000 7f6847fff640 10 monclient(hunting): handle_auth_done global_id 24698 payload 824 2024-04-26T04:26:55.616 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.601+0000 7f6847fff640 1 -- 172.21.15.63:0/1255595660 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f68500a2cd0 msgr2=0x7f68501cee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:55.616 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.601+0000 7f6847fff640 1 --2- 172.21.15.63:0/1255595660 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f68500a2cd0 0x7f68501cee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:55.616 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.601+0000 7f6847fff640 1 -- 172.21.15.63:0/1255595660 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f68500a4bd0 msgr2=0x7f685013bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:55.616 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.601+0000 7f6847fff640 1 --2- 172.21.15.63:0/1255595660 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f68500a4bd0 0x7f685013bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:55.616 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.601+0000 7f6847fff640 10 monclient: _finish_hunting 0 2024-04-26T04:26:55.616 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.601+0000 7f6847fff640 1 monclient: found mon.noname-c 2024-04-26T04:26:55.616 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.601+0000 7f6847fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:26:55.616 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.601+0000 7f6847fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.63:3301/0 2024-04-26T04:26:55.616 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.601+0000 7f6847fff640 1 -- 172.21.15.63:0/1255595660 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6850072e10 con 0x7f68500a47d0 2024-04-26T04:26:55.617 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.601+0000 7f6847fff640 1 -- 172.21.15.63:0/1255595660 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f68500a47d0 msgr2=0x7f6850135be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:26:55.617 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.601+0000 7f6847fff640 1 --2- 172.21.15.63:0/1255595660 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f68500a47d0 0x7f6850135be0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f683800e930 tx=0x7f683800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:26:55.618 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.601+0000 7f6846ffd640 1 -- 172.21.15.63:0/1255595660 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6838017070 con 0x7f68500a47d0 2024-04-26T04:26:55.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.601+0000 7f6846ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:55.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.601+0000 7f6846ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:26:55.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.601+0000 7f6846ffd640 10 monclient: dump: 2024-04-26T04:26:55.619 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:55.619 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:55.619 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:55.619 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:55.619 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:55.619 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:55.619 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:55.619 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:55.619 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:55.619 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:55.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.601+0000 7f6846ffd640 10 monclient: _finish_auth 0 2024-04-26T04:26:55.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.601+0000 7f6846ffd640 10 monclient: _check_auth_tickets 2024-04-26T04:26:55.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.601+0000 7f6846ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:26:55.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.601+0000 7f6846ffd640 1 -- 172.21.15.63:0/1255595660 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f683800cc10 con 0x7f68500a47d0 2024-04-26T04:26:55.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.601+0000 7f6846ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T04:26:55.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.601+0000 7f6846ffd640 1 -- 172.21.15.63:0/1255595660 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6838013470 con 0x7f68500a47d0 2024-04-26T04:26:55.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.601+0000 7f6846ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:55.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.601+0000 7f6846ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:26:55.620 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.601+0000 7f6846ffd640 10 monclient: dump: 2024-04-26T04:26:55.620 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:55.620 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:55.620 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:55.620 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:55.620 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:55.620 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:55.621 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:55.621 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:55.621 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:55.621 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:55.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.601+0000 7f6854867640 5 monclient: authenticate success, global_id 24698 2024-04-26T04:26:55.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.601+0000 7f6854867640 10 monclient: _renew_subs 2024-04-26T04:26:55.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.601+0000 7f684d59a640 4 set_mon_vals no callback set 2024-04-26T04:26:55.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.601+0000 7f6854867640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:26:55.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.601+0000 7f6854867640 1 -- 172.21.15.63:0/1255595660 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68501369f0 con 0x7f68500a47d0 2024-04-26T04:26:55.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.601+0000 7f6854867640 10 monclient: _renew_subs 2024-04-26T04:26:55.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.601+0000 7f6854867640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:26:55.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.601+0000 7f6854867640 1 -- 172.21.15.63:0/1255595660 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f68501cfc90 con 0x7f68500a47d0 2024-04-26T04:26:55.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.605+0000 7f6846ffd640 1 -- 172.21.15.63:0/1255595660 <== mon.2 v2:172.21.15.63:3301/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f683801a070 con 0x7f68500a47d0 2024-04-26T04:26:55.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.605+0000 7f6854867640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:26:55.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.605+0000 7f6854867640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:26:55.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.605+0000 7f6854867640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:26:55.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.605+0000 7f6854867640 1 -- 172.21.15.63:0/1255595660 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68501212f0 con 0x7f68500a47d0 2024-04-26T04:26:55.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.605+0000 7f6846ffd640 1 --2- 172.21.15.63:0/1255595660 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f6830035f10 0x7f68300383d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:55.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.605+0000 7f6846ffd640 1 -- 172.21.15.63:0/1255595660 <== mon.2 v2:172.21.15.63:3301/0 5 ==== osd_map(323..323 src has 1..323) ==== 4346+0+0 (secure 0 0 0) 0x7f6838048df0 con 0x7f68500a47d0 2024-04-26T04:26:55.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.605+0000 7f684e59c640 1 --2- 172.21.15.63:0/1255595660 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f6830035f10 0x7f68300383d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:55.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.605+0000 7f684e59c640 10 monclient: get_auth_request con 0x7f6830035f10 auth_method 0 2024-04-26T04:26:55.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.605+0000 7f684e59c640 1 --2- 172.21.15.63:0/1255595660 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f6830035f10 0x7f68300383d0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f685006d7c0 tx=0x7f6840008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:26:55.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.609+0000 7f6846ffd640 1 -- 172.21.15.63:0/1255595660 <== mon.2 v2:172.21.15.63:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f683804c030 con 0x7f68500a47d0 2024-04-26T04:26:55.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.609+0000 7f6846ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:26:55.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.609+0000 7f6846ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:26:55.836 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.829+0000 7f6854867640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-04-26T04:26:55.836 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.829+0000 7f6854867640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-04-26T04:26:55.836 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.829+0000 7f6854867640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:26:55.836 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.829+0000 7f6854867640 1 -- 172.21.15.63:0/1255595660 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f685013d7c0 con 0x7f68500a47d0 2024-04-26T04:26:55.837 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.829+0000 7f6846ffd640 1 -- 172.21.15.63:0/1255595660 <== mon.2 v2:172.21.15.63:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f6838039630 con 0x7f68500a47d0 2024-04-26T04:26:55.837 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.829+0000 7f6846ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-04-26T04:26:55.837 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.829+0000 7f6846ffd640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:26:55.837 INFO:teuthology.orchestra.run.smithi063.stdout:{"election_epoch":42,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"cd1c6a0c-4e4b-4014-89c0-2e535479ef39","modified":"2024-04-26T04:18:03.702339Z","created":"2024-04-26T04:18:03.702339Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3300","nonce":0},{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:3300","nonce":0},{"type":"v1","addr":"172.21.15.186:6789","nonce":0}]},"addr":"172.21.15.186:6789/0","public_addr":"172.21.15.186:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3301","nonce":0},{"type":"v1","addr":"172.21.15.63:6790","nonce":0}]},"addr":"172.21.15.63:6790/0","public_addr":"172.21.15.63:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-04-26T04:26:55.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.833+0000 7f6854867640 1 -- 172.21.15.63:0/1255595660 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f6830035f10 msgr2=0x7f68300383d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:55.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.833+0000 7f6854867640 1 --2- 172.21.15.63:0/1255595660 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f6830035f10 0x7f68300383d0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f685006d7c0 tx=0x7f6840008040 comp rx=0 tx=0).stop 2024-04-26T04:26:55.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.833+0000 7f6854867640 10 monclient: shutdown 2024-04-26T04:26:55.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.833+0000 7f6854867640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:26:55.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.833+0000 7f6854867640 1 -- 172.21.15.63:0/1255595660 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f68500a47d0 msgr2=0x7f6850135be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:55.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.833+0000 7f6854867640 1 --2- 172.21.15.63:0/1255595660 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f68500a47d0 0x7f6850135be0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f683800e930 tx=0x7f683800edf0 comp rx=0 tx=0).stop 2024-04-26T04:26:55.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.833+0000 7f6854867640 1 -- 172.21.15.63:0/1255595660 shutdown_connections 2024-04-26T04:26:55.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.833+0000 7f6854867640 1 --2- 172.21.15.63:0/1255595660 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f6830035f10 0x7f68300383d0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:55.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.833+0000 7f6854867640 1 --2- 172.21.15.63:0/1255595660 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f68500a4bd0 0x7f685013bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:55.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.833+0000 7f6854867640 1 --2- 172.21.15.63:0/1255595660 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f68500a47d0 0x7f6850135be0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:55.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.833+0000 7f6854867640 1 --2- 172.21.15.63:0/1255595660 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f68500a2cd0 0x7f68501cee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:55.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.833+0000 7f6854867640 1 -- 172.21.15.63:0/1255595660 >> 172.21.15.63:0/1255595660 conn(0x7f68500a5f40 msgr2=0x7f68500a1a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:26:55.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.833+0000 7f6854867640 1 -- 172.21.15.63:0/1255595660 shutdown_connections 2024-04-26T04:26:55.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.833+0000 7f6854867640 1 -- 172.21.15.63:0/1255595660 wait complete. 2024-04-26T04:26:55.856 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-04-26T04:26:55.856 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-26T04:26:55.866 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-26T04:26:55.866 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T04:26:55.989 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.981+0000 7f12727ff640 10 monclient: get_monmap_and_config 2024-04-26T04:26:55.989 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.981+0000 7f12727ff640 10 monclient: build_initial_monmap 2024-04-26T04:26:55.989 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.981+0000 7f12727ff640 10 monclient: monmap: 2024-04-26T04:26:55.989 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:26:55.989 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:55.989 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:26:55.989244+0000 2024-04-26T04:26:55.989 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:26:55.989244+0000 2024-04-26T04:26:55.989 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:26:55.989 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:55.989 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:26:55.989 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:26:55.990 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:26:55.990 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:26:55.990 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:26:55.990 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:55.992 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.985+0000 7f12727ff640 1 Processor -- start 2024-04-26T04:26:55.992 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.985+0000 7f12727ff640 1 -- start start 2024-04-26T04:26:55.992 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.985+0000 7f12727ff640 10 monclient: init 2024-04-26T04:26:55.992 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.985+0000 7f12727ff640 10 monclient: _reopen_session rank -1 2024-04-26T04:26:55.992 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.985+0000 7f12727ff640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-04-26T04:26:55.992 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.985+0000 7f12727ff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f126c09d110 0x7f126c09aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:55.993 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.985+0000 7f12727ff640 10 monclient(hunting): picked mon.noname-c con 0x7f126c09d110 addr v2:172.21.15.186:3300/0 2024-04-26T04:26:55.993 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.985+0000 7f12727ff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f126c09d4f0 addr v1:172.21.15.63:6789/0 2024-04-26T04:26:55.993 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.985+0000 7f12727ff640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f126c09d900 0x7f126c0a33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:55.993 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.985+0000 7f12727ff640 10 monclient(hunting): picked mon.noname-a con 0x7f126c09d900 addr v2:172.21.15.63:3300/0 2024-04-26T04:26:55.993 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.985+0000 7f12727ff640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f126c09ed60 con 0x7f126c09d4f0 2024-04-26T04:26:55.993 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.985+0000 7f12727ff640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:55.993 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.985+0000 7f12727ff640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7f126c0a0550 con 0x7f126c09d900 2024-04-26T04:26:55.993 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.985+0000 7f12727ff640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:55.993 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.985+0000 7f12727ff640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f126c09dd30 con 0x7f126c09d110 2024-04-26T04:26:55.993 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.985+0000 7f12727ff640 10 monclient(hunting): _renew_subs 2024-04-26T04:26:55.993 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.985+0000 7f12727ff640 10 monclient(hunting): authenticate will time out at 1447.696950s 2024-04-26T04:26:55.993 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.985+0000 7f1270d9c640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f126c09d900 0x7f126c0a33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:55.993 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.985+0000 7f126b7fe640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f126c09d4f0 0x7f126c0a2ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:36750/0 (socket says 172.21.15.63:36750) 2024-04-26T04:26:55.993 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.985+0000 7f126b7fe640 1 -- 172.21.15.63:0/2416073688 learned_addr learned my addr 172.21.15.63:0/2416073688 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-04-26T04:26:55.994 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.985+0000 7f1270d9c640 10 monclient(hunting): get_auth_request con 0x7f126c09d900 auth_method 0 2024-04-26T04:26:55.994 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.985+0000 7f1270d9c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:55.994 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.985+0000 7f1270d9c640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:55.994 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.985+0000 7f1270d9c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:55.994 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.985+0000 7f126bfff640 1 --2- 172.21.15.63:0/2416073688 >> v2:172.21.15.186:3300/0 conn(0x7f126c09d110 0x7f126c09aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:55.994 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.985+0000 7f126bfff640 10 monclient(hunting): get_auth_request con 0x7f126c09d110 auth_method 0 2024-04-26T04:26:55.994 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.985+0000 7f126bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:55.994 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.985+0000 7f126bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:55.994 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.985+0000 7f126bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:55.994 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.985+0000 7f1270d9c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:26:55.994 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.985+0000 7f1270d9c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:26:55.994 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.985+0000 7f1270d9c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:26:55.994 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.985+0000 7f126bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:26:55.994 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.985+0000 7f126bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:26:55.994 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.985+0000 7f126bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:26:55.995 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.985+0000 7f1270d9c640 10 monclient(hunting): handle_auth_done global_id 44139 payload 274 2024-04-26T04:26:55.995 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.985+0000 7f1270d9c640 1 -- 172.21.15.63:0/2416073688 >> v2:172.21.15.186:3300/0 conn(0x7f126c09d110 msgr2=0x7f126c09aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:26:55.995 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.985+0000 7f1270d9c640 1 --2- 172.21.15.63:0/2416073688 >> v2:172.21.15.186:3300/0 conn(0x7f126c09d110 0x7f126c09aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:55.995 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.985+0000 7f1270d9c640 1 -- 172.21.15.63:0/2416073688 >> v1:172.21.15.63:6789/0 conn(0x7f126c09d4f0 legacy=0x7f126c0a2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:26:55.995 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.985+0000 7f1270d9c640 10 monclient: _finish_hunting 0 2024-04-26T04:26:55.995 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.985+0000 7f1270d9c640 1 monclient: found mon.noname-a 2024-04-26T04:26:55.995 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.985+0000 7f1270d9c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:26:55.995 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.985+0000 7f1270d9c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:26:55.995 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.985+0000 7f1270d9c640 1 -- 172.21.15.63:0/2416073688 --> v2:172.21.15.63:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f126c0a0a80 con 0x7f126c09d900 2024-04-26T04:26:55.996 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.989+0000 7f1270d9c640 1 -- 172.21.15.63:0/2416073688 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f126c09d900 msgr2=0x7f126c0a33f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:26:55.996 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.989+0000 7f1270d9c640 1 --2- 172.21.15.63:0/2416073688 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f126c09d900 0x7f126c0a33f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f126c0a06d0 tx=0x7f126401c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dc04d2e6116714b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:26:55.997 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.989+0000 7f126affd640 1 -- 172.21.15.63:0/2416073688 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12640b2070 con 0x7f126c09d900 2024-04-26T04:26:55.997 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.989+0000 7f126affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:55.997 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.989+0000 7f126affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:26:55.997 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.989+0000 7f126affd640 10 monclient: dump: 2024-04-26T04:26:55.997 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:55.997 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:55.997 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:55.997 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:55.997 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:55.997 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:55.997 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:55.997 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:55.997 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:55.997 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:55.998 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.989+0000 7f126affd640 10 monclient: _finish_auth 0 2024-04-26T04:26:55.998 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.989+0000 7f126affd640 10 monclient: _check_auth_tickets 2024-04-26T04:26:55.998 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.989+0000 7f126affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:26:55.998 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.989+0000 7f126affd640 1 -- 172.21.15.63:0/2416073688 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12640a5760 con 0x7f126c09d900 2024-04-26T04:26:55.998 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.989+0000 7f126affd640 10 monclient: handle_config config(0 keys) 2024-04-26T04:26:55.998 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.989+0000 7f126affd640 1 -- 172.21.15.63:0/2416073688 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12640a5b30 con 0x7f126c09d900 2024-04-26T04:26:55.998 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.989+0000 7f12727ff640 5 monclient: authenticate success, global_id 44139 2024-04-26T04:26:55.998 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.989+0000 7f12727ff640 10 monclient: get_monmap_and_config success 2024-04-26T04:26:55.998 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.989+0000 7f12727ff640 4 set_mon_vals no callback set 2024-04-26T04:26:55.998 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.989+0000 7f126affd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:55.998 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.989+0000 7f126affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:26:55.998 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.989+0000 7f126affd640 10 monclient: dump: 2024-04-26T04:26:55.998 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:55.999 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:55.999 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:55.999 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:55.999 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:55.999 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:55.999 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:55.999 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:55.999 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:55.999 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:55.999 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.989+0000 7f12727ff640 10 monclient: shutdown 2024-04-26T04:26:55.999 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.989+0000 7f12727ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:26:56.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.989+0000 7f12727ff640 1 -- 172.21.15.63:0/2416073688 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f126c09d900 msgr2=0x7f126c0a33f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:56.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.989+0000 7f12727ff640 1 --2- 172.21.15.63:0/2416073688 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f126c09d900 0x7f126c0a33f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f126c0a06d0 tx=0x7f126401c7b0 comp rx=0 tx=0).stop 2024-04-26T04:26:56.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.989+0000 7f12727ff640 1 -- 172.21.15.63:0/2416073688 shutdown_connections 2024-04-26T04:26:56.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.989+0000 7f12727ff640 1 --2- 172.21.15.63:0/2416073688 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f126c09d900 0x7f126c0a33f0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:56.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.989+0000 7f12727ff640 1 --2- 172.21.15.63:0/2416073688 >> v2:172.21.15.186:3300/0 conn(0x7f126c09d110 0x7f126c09aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:56.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.989+0000 7f12727ff640 1 -- 172.21.15.63:0/2416073688 >> 172.21.15.63:0/2416073688 conn(0x7f126c0a7ab0 msgr2=0x7f126c0a3dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:26:56.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.989+0000 7f12727ff640 1 -- 172.21.15.63:0/2416073688 shutdown_connections 2024-04-26T04:26:56.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.989+0000 7f12727ff640 1 -- 172.21.15.63:0/2416073688 wait complete. 2024-04-26T04:26:56.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.989+0000 7f12727ff640 10 monclient: build_initial_monmap 2024-04-26T04:26:56.000 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.989+0000 7f12727ff640 10 monclient: monmap: 2024-04-26T04:26:56.000 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:26:56.000 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:26:56.000 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:26:56.000 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:26:56.000 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:26:56.001 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:56.001 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:26:56.001 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:26:56.001 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:26:56.001 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:56.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.989+0000 7f12727ff640 1 Processor -- start 2024-04-26T04:26:56.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.989+0000 7f12727ff640 1 -- start start 2024-04-26T04:26:56.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.989+0000 7f12727ff640 10 monclient: init 2024-04-26T04:26:56.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f12727ff640 10 monclient: _reopen_session rank -1 2024-04-26T04:26:56.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f12727ff640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T04:26:56.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f12727ff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f126c09d110 0x7f126c13dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:56.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f12727ff640 10 monclient(hunting): picked mon.noname-b con 0x7f126c09d110 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:26:56.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f12727ff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f126c09d4f0 0x7f126c13e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:56.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f12727ff640 10 monclient(hunting): picked mon.noname-a con 0x7f126c09d4f0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:26:56.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f12727ff640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f126c09d900 0x7f126c124bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:56.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f12727ff640 10 monclient(hunting): picked mon.noname-c con 0x7f126c09d900 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:26:56.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f12727ff640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:56.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f12727ff640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f126c0a3ab0 con 0x7f126c09d4f0 2024-04-26T04:26:56.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f12727ff640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:56.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f12727ff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f126c0a3930 con 0x7f126c09d110 2024-04-26T04:26:56.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f12727ff640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:56.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f12727ff640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f126c141900 con 0x7f126c09d900 2024-04-26T04:26:56.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f12727ff640 10 monclient(hunting): _renew_subs 2024-04-26T04:26:56.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f12727ff640 10 monclient(hunting): authenticate will time out at 1447.704304s 2024-04-26T04:26:56.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f1270d9c640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f126c09d900 0x7f126c124bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:56.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f1270d9c640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f126c09d900 0x7f126c124bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:41064/0 (socket says 172.21.15.63:41064) 2024-04-26T04:26:56.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f1270d9c640 1 -- 172.21.15.63:0/2531764963 learned_addr learned my addr 172.21.15.63:0/2531764963 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:26:56.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f1270d9c640 10 monclient(hunting): get_auth_request con 0x7f126c09d900 auth_method 0 2024-04-26T04:26:56.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f126b7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f126c09d4f0 0x7f126c13e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:56.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f1270d9c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:56.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f1270d9c640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:56.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f1270d9c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:56.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f126b7fe640 10 monclient(hunting): get_auth_request con 0x7f126c09d4f0 auth_method 0 2024-04-26T04:26:56.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f126b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:56.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f126b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:56.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f126b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:56.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f1270d9c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:26:56.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f1270d9c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:26:56.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f1270d9c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:26:56.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f126bfff640 1 --2- 172.21.15.63:0/2531764963 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f126c09d110 0x7f126c13dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:56.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f126b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:26:56.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f126b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:26:56.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f126b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:26:56.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f126bfff640 10 monclient(hunting): get_auth_request con 0x7f126c09d110 auth_method 0 2024-04-26T04:26:56.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f126bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:56.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f126bfff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:56.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f126bfff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:56.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f1270d9c640 10 monclient(hunting): handle_auth_done global_id 24701 payload 824 2024-04-26T04:26:56.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f1270d9c640 1 -- 172.21.15.63:0/2531764963 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f126c09d110 msgr2=0x7f126c13dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:56.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f1270d9c640 1 --2- 172.21.15.63:0/2531764963 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f126c09d110 0x7f126c13dcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:56.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f1270d9c640 1 -- 172.21.15.63:0/2531764963 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f126c09d4f0 msgr2=0x7f126c13e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:56.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f1270d9c640 1 --2- 172.21.15.63:0/2531764963 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f126c09d4f0 0x7f126c13e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:56.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f1270d9c640 10 monclient: _finish_hunting 0 2024-04-26T04:26:56.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f1270d9c640 1 monclient: found mon.noname-c 2024-04-26T04:26:56.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f1270d9c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:26:56.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f1270d9c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.63:3301/0 2024-04-26T04:26:56.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f1270d9c640 1 -- 172.21.15.63:0/2531764963 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f126c0a6aa0 con 0x7f126c09d900 2024-04-26T04:26:56.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f126bfff640 1 --2- 172.21.15.63:0/2531764963 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f126c09d110 0x7f126c13dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T04:26:56.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f126b7fe640 1 --2- 172.21.15.63:0/2531764963 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f126c09d4f0 0x7f126c13e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T04:26:56.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f1270d9c640 1 -- 172.21.15.63:0/2531764963 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f126c09d900 msgr2=0x7f126c124bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:26:56.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f1270d9c640 1 --2- 172.21.15.63:0/2531764963 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f126c09d900 0x7f126c124bb0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f12640b0040 tx=0x7f1264019550 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:26:56.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f1268ff9640 1 -- 172.21.15.63:0/2531764963 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12640b2050 con 0x7f126c09d900 2024-04-26T04:26:56.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f1268ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:56.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f1268ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:26:56.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f1268ff9640 10 monclient: dump: 2024-04-26T04:26:56.008 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:56.008 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:56.008 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:56.008 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:56.008 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:56.008 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:56.008 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:56.008 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:56.008 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:56.009 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:56.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f1268ff9640 10 monclient: _finish_auth 0 2024-04-26T04:26:56.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f1268ff9640 10 monclient: _check_auth_tickets 2024-04-26T04:26:56.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f1268ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:26:56.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f1268ff9640 1 -- 172.21.15.63:0/2531764963 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12640be070 con 0x7f126c09d900 2024-04-26T04:26:56.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f12727ff640 5 monclient: authenticate success, global_id 24701 2024-04-26T04:26:56.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f12727ff640 10 monclient: _renew_subs 2024-04-26T04:26:56.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f12727ff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:26:56.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f12727ff640 1 -- 172.21.15.63:0/2531764963 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f126c09df50 con 0x7f126c09d900 2024-04-26T04:26:56.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f1268ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T04:26:56.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f1268ff9640 1 -- 172.21.15.63:0/2531764963 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12640adaa0 con 0x7f126c09d900 2024-04-26T04:26:56.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f1268ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:56.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f1268ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:26:56.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f1268ff9640 10 monclient: dump: 2024-04-26T04:26:56.010 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:56.010 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:56.010 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:56.010 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:56.010 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:56.010 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:56.010 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:56.010 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:56.010 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:56.010 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:56.010 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f12727ff640 10 monclient: _renew_subs 2024-04-26T04:26:56.010 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f12727ff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:26:56.010 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f12727ff640 1 -- 172.21.15.63:0/2531764963 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f126c128af0 con 0x7f126c09d900 2024-04-26T04:26:56.010 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.993+0000 7f126a7fc640 4 set_mon_vals no callback set 2024-04-26T04:26:56.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.997+0000 7f12727ff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:26:56.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.997+0000 7f12727ff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:26:56.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.997+0000 7f12727ff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:26:56.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.997+0000 7f12727ff640 1 -- 172.21.15.63:0/2531764963 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f126c127140 con 0x7f126c09d900 2024-04-26T04:26:56.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.997+0000 7f1268ff9640 1 -- 172.21.15.63:0/2531764963 <== mon.2 v2:172.21.15.63:3301/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f12640adcc0 con 0x7f126c09d900 2024-04-26T04:26:56.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.997+0000 7f1268ff9640 1 --2- 172.21.15.63:0/2531764963 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f1258035ef0 0x7f12580383b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:56.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.997+0000 7f1268ff9640 1 -- 172.21.15.63:0/2531764963 <== mon.2 v2:172.21.15.63:3301/0 5 ==== osd_map(323..323 src has 1..323) ==== 4346+0+0 (secure 0 0 0) 0x7f12640e3640 con 0x7f126c09d900 2024-04-26T04:26:56.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.997+0000 7f126bfff640 1 --2- 172.21.15.63:0/2531764963 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f1258035ef0 0x7f12580383b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:56.011 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.997+0000 7f126bfff640 10 monclient: get_auth_request con 0x7f1258035ef0 auth_method 0 2024-04-26T04:26:56.012 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:55.997+0000 7f126bfff640 1 --2- 172.21.15.63:0/2531764963 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f1258035ef0 0x7f12580383b0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f125c009bb0 tx=0x7f125c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:26:56.012 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.001+0000 7f1268ff9640 1 -- 172.21.15.63:0/2531764963 <== mon.2 v2:172.21.15.63:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f126c127140 con 0x7f126c09d900 2024-04-26T04:26:56.012 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.001+0000 7f1268ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:26:56.012 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.001+0000 7f1268ff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:26:56.163 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.157+0000 7f12727ff640 1 -- 172.21.15.63:0/2531764963 --> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f126c062e10 con 0x7f1258035ef0 2024-04-26T04:26:56.166 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.165+0000 7f1268ff9640 1 -- 172.21.15.63:0/2531764963 <== mgr.4105 v2:172.21.15.186:6824/2583706639 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33778 (secure 0 0 0) 0x7f126c062e10 con 0x7f1258035ef0 2024-04-26T04:26:56.167 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-26T04:26:56.167 INFO:teuthology.orchestra.run.smithi063.stderr:dumped all 2024-04-26T04:26:56.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.165+0000 7f12727ff640 1 -- 172.21.15.63:0/2531764963 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f1258035ef0 msgr2=0x7f12580383b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:56.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.165+0000 7f12727ff640 1 --2- 172.21.15.63:0/2531764963 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f1258035ef0 0x7f12580383b0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f125c009bb0 tx=0x7f125c008040 comp rx=0 tx=0).stop 2024-04-26T04:26:56.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.165+0000 7f12727ff640 10 monclient: shutdown 2024-04-26T04:26:56.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.165+0000 7f12727ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:26:56.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.165+0000 7f12727ff640 1 -- 172.21.15.63:0/2531764963 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f126c09d900 msgr2=0x7f126c124bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:56.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.165+0000 7f12727ff640 1 --2- 172.21.15.63:0/2531764963 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f126c09d900 0x7f126c124bb0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f12640b0040 tx=0x7f1264019550 comp rx=0 tx=0).stop 2024-04-26T04:26:56.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.165+0000 7f12727ff640 1 -- 172.21.15.63:0/2531764963 shutdown_connections 2024-04-26T04:26:56.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.165+0000 7f12727ff640 1 --2- 172.21.15.63:0/2531764963 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f1258035ef0 0x7f12580383b0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:56.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.165+0000 7f12727ff640 1 --2- 172.21.15.63:0/2531764963 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f126c09d900 0x7f126c124bb0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:56.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.165+0000 7f12727ff640 1 --2- 172.21.15.63:0/2531764963 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f126c09d4f0 0x7f126c13e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:56.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.165+0000 7f12727ff640 1 --2- 172.21.15.63:0/2531764963 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f126c09d110 0x7f126c13dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:56.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.165+0000 7f12727ff640 1 -- 172.21.15.63:0/2531764963 >> 172.21.15.63:0/2531764963 conn(0x7f126c0a7ab0 msgr2=0x7f126c0a2b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:26:56.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.165+0000 7f12727ff640 1 -- 172.21.15.63:0/2531764963 shutdown_connections 2024-04-26T04:26:56.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.165+0000 7f12727ff640 1 -- 172.21.15.63:0/2531764963 wait complete. 2024-04-26T04:26:56.185 INFO:teuthology.orchestra.run.smithi063.stdout:{"pg_ready":true,"pg_map":{"version":564,"stamp":"2024-04-26T04:26:55.592036+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":205168,"kb_used_data":26408,"kb_used_omap":9,"kb_used_meta":178678,"kb_avail":566025872,"statfs":{"total":579820584960,"available":579610492928,"internally_reserved":0,"allocated":27041792,"data_stored":10728854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9567,"internal_metadata":182966945},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003047"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":617,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-04-26T04:26:38.082572+0000","last_change":"2024-04-26T04:18:47.131857+0000","last_active":"2024-04-26T04:26:38.082572+0000","last_peered":"2024-04-26T04:26:38.082572+0000","last_clean":"2024-04-26T04:26:38.082572+0000","last_became_active":"2024-04-26T04:18:47.131175+0000","last_became_peered":"2024-04-26T04:18:47.131175+0000","last_unstale":"2024-04-26T04:26:38.082572+0000","last_undegraded":"2024-04-26T04:26:38.082572+0000","last_fullsized":"2024-04-26T04:26:38.082572+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_clean_scrub_stamp":"2024-04-26T04:18:45.482773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:35:03.860574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":610,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-04-26T04:26:37.279334+0000","last_change":"2024-04-26T04:18:46.499700+0000","last_active":"2024-04-26T04:26:37.279334+0000","last_peered":"2024-04-26T04:26:37.279334+0000","last_clean":"2024-04-26T04:26:37.279334+0000","last_became_active":"2024-04-26T04:18:46.498983+0000","last_became_peered":"2024-04-26T04:18:46.498983+0000","last_unstale":"2024-04-26T04:26:37.279334+0000","last_undegraded":"2024-04-26T04:26:37.279334+0000","last_fullsized":"2024-04-26T04:26:37.279334+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_clean_scrub_stamp":"2024-04-26T04:18:45.482773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:51:56.740670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":617,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-04-26T04:26:38.082586+0000","last_change":"2024-04-26T04:18:46.507904+0000","last_active":"2024-04-26T04:26:38.082586+0000","last_peered":"2024-04-26T04:26:38.082586+0000","last_clean":"2024-04-26T04:26:38.082586+0000","last_became_active":"2024-04-26T04:18:46.507335+0000","last_became_peered":"2024-04-26T04:18:46.507335+0000","last_unstale":"2024-04-26T04:26:38.082586+0000","last_undegraded":"2024-04-26T04:26:38.082586+0000","last_fullsized":"2024-04-26T04:26:38.082586+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_clean_scrub_stamp":"2024-04-26T04:18:45.482773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:52:50.029369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":611,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-04-26T04:26:37.276307+0000","last_change":"2024-04-26T04:18:46.506682+0000","last_active":"2024-04-26T04:26:37.276307+0000","last_peered":"2024-04-26T04:26:37.276307+0000","last_clean":"2024-04-26T04:26:37.276307+0000","last_became_active":"2024-04-26T04:18:46.506209+0000","last_became_peered":"2024-04-26T04:18:46.506209+0000","last_unstale":"2024-04-26T04:26:37.276307+0000","last_undegraded":"2024-04-26T04:26:37.276307+0000","last_fullsized":"2024-04-26T04:26:37.276307+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_clean_scrub_stamp":"2024-04-26T04:18:45.482773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:20:00.475307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":613,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-04-26T04:26:37.275301+0000","last_change":"2024-04-26T04:18:46.504685+0000","last_active":"2024-04-26T04:26:37.275301+0000","last_peered":"2024-04-26T04:26:37.275301+0000","last_clean":"2024-04-26T04:26:37.275301+0000","last_became_active":"2024-04-26T04:18:46.504292+0000","last_became_peered":"2024-04-26T04:18:46.504292+0000","last_unstale":"2024-04-26T04:26:37.275301+0000","last_undegraded":"2024-04-26T04:26:37.275301+0000","last_fullsized":"2024-04-26T04:26:37.275301+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_clean_scrub_stamp":"2024-04-26T04:18:45.482773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:42:29.110482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":605,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-04-26T04:26:38.261871+0000","last_change":"2024-04-26T04:18:46.504010+0000","last_active":"2024-04-26T04:26:38.261871+0000","last_peered":"2024-04-26T04:26:38.261871+0000","last_clean":"2024-04-26T04:26:38.261871+0000","last_became_active":"2024-04-26T04:18:46.503343+0000","last_became_peered":"2024-04-26T04:18:46.503343+0000","last_unstale":"2024-04-26T04:26:38.261871+0000","last_undegraded":"2024-04-26T04:26:38.261871+0000","last_fullsized":"2024-04-26T04:26:38.261871+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_clean_scrub_stamp":"2024-04-26T04:18:45.482773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:19:38.324198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":618,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-04-26T04:26:38.082607+0000","last_change":"2024-04-26T04:18:46.503233+0000","last_active":"2024-04-26T04:26:38.082607+0000","last_peered":"2024-04-26T04:26:38.082607+0000","last_clean":"2024-04-26T04:26:38.082607+0000","last_became_active":"2024-04-26T04:18:46.502682+0000","last_became_peered":"2024-04-26T04:18:46.502682+0000","last_unstale":"2024-04-26T04:26:38.082607+0000","last_undegraded":"2024-04-26T04:26:38.082607+0000","last_fullsized":"2024-04-26T04:26:38.082607+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_clean_scrub_stamp":"2024-04-26T04:18:45.482773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:26:15.760818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":613,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-04-26T04:26:37.275301+0000","last_change":"2024-04-26T04:18:46.504844+0000","last_active":"2024-04-26T04:26:37.275301+0000","last_peered":"2024-04-26T04:26:37.275301+0000","last_clean":"2024-04-26T04:26:37.275301+0000","last_became_active":"2024-04-26T04:18:46.504332+0000","last_became_peered":"2024-04-26T04:18:46.504332+0000","last_unstale":"2024-04-26T04:26:37.275301+0000","last_undegraded":"2024-04-26T04:26:37.275301+0000","last_fullsized":"2024-04-26T04:26:37.275301+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_clean_scrub_stamp":"2024-04-26T04:18:45.482773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:13:26.538440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":770,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-04-26T04:26:38.082546+0000","last_change":"2024-04-26T04:18:45.505818+0000","last_active":"2024-04-26T04:26:38.082546+0000","last_peered":"2024-04-26T04:26:38.082546+0000","last_clean":"2024-04-26T04:26:38.082546+0000","last_became_active":"2024-04-26T04:18:45.504887+0000","last_became_peered":"2024-04-26T04:18:45.504887+0000","last_unstale":"2024-04-26T04:26:38.082546+0000","last_undegraded":"2024-04-26T04:26:38.082546+0000","last_fullsized":"2024-04-26T04:26:38.082546+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:44.470061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:44.470061+0000","last_clean_scrub_stamp":"2024-04-26T04:18:44.470061+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:42:56.629937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":12,"seq":51539607654,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35928,"kb_used_data":4748,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335912,"statfs":{"total":96636764160,"available":96599973888,"internally_reserved":0,"allocated":4861952,"data_stored":2140761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":12,"seq":51539607654,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35408,"kb_used_data":4228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94336432,"statfs":{"total":96636764160,"available":96600506368,"internally_reserved":0,"allocated":4329472,"data_stored":1611833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607654,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35408,"kb_used_data":4228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94336432,"statfs":{"total":96636764160,"available":96600506368,"internally_reserved":0,"allocated":4329472,"data_stored":1611833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640358,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35408,"kb_used_data":4228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94336432,"statfs":{"total":96636764160,"available":96600506368,"internally_reserved":0,"allocated":4329472,"data_stored":1611833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640357,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31248,"kb_used_data":4228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340592,"statfs":{"total":96636764160,"available":96604766208,"internally_reserved":0,"allocated":4329472,"data_stored":1611833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640357,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31768,"kb_used_data":4748,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340072,"statfs":{"total":96636764160,"available":96604233728,"internally_reserved":0,"allocated":4861952,"data_stored":2140761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T04:26:56.186 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T04:26:56.312 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.305+0000 7f60f8380640 10 monclient: get_monmap_and_config 2024-04-26T04:26:56.312 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.305+0000 7f60f8380640 10 monclient: build_initial_monmap 2024-04-26T04:26:56.312 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.305+0000 7f60f8380640 10 monclient: monmap: 2024-04-26T04:26:56.312 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:26:56.312 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:56.312 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:26:56.312476+0000 2024-04-26T04:26:56.312 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:26:56.312476+0000 2024-04-26T04:26:56.313 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:26:56.313 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:56.313 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:26:56.313 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:26:56.313 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:26:56.313 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:26:56.313 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:26:56.313 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:56.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.305+0000 7f60f8380640 1 Processor -- start 2024-04-26T04:26:56.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.305+0000 7f60f8380640 1 -- start start 2024-04-26T04:26:56.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.305+0000 7f60f8380640 10 monclient: init 2024-04-26T04:26:56.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.309+0000 7f60f8380640 10 monclient: _reopen_session rank -1 2024-04-26T04:26:56.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.309+0000 7f60f8380640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-04-26T04:26:56.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.309+0000 7f60f8380640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f60f009fe60 addr v1:172.21.15.63:6789/0 2024-04-26T04:26:56.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.309+0000 7f60f8380640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f60f00a0240 0x7f60f00a2a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:56.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.309+0000 7f60f8380640 10 monclient(hunting): picked mon.noname-b con 0x7f60f00a0240 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:26:56.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.309+0000 7f60f8380640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f60f00a2f70 addr v1:172.21.15.186:6789/0 2024-04-26T04:26:56.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.309+0000 7f60f8380640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60f00a0c60 con 0x7f60f009fe60 2024-04-26T04:26:56.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.309+0000 7f60f8380640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60f00a0f40 con 0x7f60f00a2f70 2024-04-26T04:26:56.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.309+0000 7f60f8380640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:56.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.309+0000 7f60f8380640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f60f00a3730 con 0x7f60f00a0240 2024-04-26T04:26:56.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.309+0000 7f60f8380640 10 monclient(hunting): _renew_subs 2024-04-26T04:26:56.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.309+0000 7f60f8380640 10 monclient(hunting): authenticate will time out at 1448.020328s 2024-04-26T04:26:56.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.309+0000 7f60f611c640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f60f009fe60 0x7f60f01372f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:36762/0 (socket says 172.21.15.63:36762) 2024-04-26T04:26:56.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.309+0000 7f60f591b640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f60f00a0240 0x7f60f00a2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:56.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.309+0000 7f60f611c640 1 -- 172.21.15.63:0/952319573 learned_addr learned my addr 172.21.15.63:0/952319573 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-04-26T04:26:56.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.309+0000 7f60f591b640 10 monclient(hunting): get_auth_request con 0x7f60f00a0240 auth_method 0 2024-04-26T04:26:56.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.309+0000 7f60f591b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:56.318 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.309+0000 7f60f591b640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:56.318 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.309+0000 7f60f591b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:56.318 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.309+0000 7f60f591b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:26:56.318 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.309+0000 7f60f591b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:26:56.318 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.309+0000 7f60f591b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:26:56.318 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.309+0000 7f60f591b640 10 monclient(hunting): handle_auth_done global_id 24704 payload 274 2024-04-26T04:26:56.318 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.309+0000 7f60f591b640 1 -- 172.21.15.63:0/952319573 >> v1:172.21.15.186:6789/0 conn(0x7f60f00a2f70 legacy=0x7f60f013e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:26:56.318 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.309+0000 7f60f591b640 1 -- 172.21.15.63:0/952319573 >> v1:172.21.15.63:6789/0 conn(0x7f60f009fe60 legacy=0x7f60f01372f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:26:56.318 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.309+0000 7f60f591b640 10 monclient: _finish_hunting 0 2024-04-26T04:26:56.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.309+0000 7f60f591b640 1 monclient: found mon.noname-b 2024-04-26T04:26:56.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.309+0000 7f60f591b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:26:56.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.309+0000 7f60f591b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.63:3301/0 2024-04-26T04:26:56.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.309+0000 7f60f591b640 1 -- 172.21.15.63:0/952319573 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60f00a1220 con 0x7f60f00a0240 2024-04-26T04:26:56.328 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.313+0000 7f60f591b640 1 -- 172.21.15.63:0/952319573 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f60f00a0240 msgr2=0x7f60f00a2a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:26:56.329 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.313+0000 7f60f591b640 1 --2- 172.21.15.63:0/952319573 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f60f00a0240 0x7f60f00a2a30 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f60e00097f0 tx=0x7f60e001c320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20288f8a0d2dc02d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:26:56.329 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.313+0000 7f60f511a640 1 -- 172.21.15.63:0/952319573 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60e00b0070 con 0x7f60f00a0240 2024-04-26T04:26:56.329 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.313+0000 7f60f511a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:56.330 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.313+0000 7f60f511a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T04:26:56.330 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.313+0000 7f60f511a640 10 monclient: dump: 2024-04-26T04:26:56.330 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:56.330 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:56.330 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:56.330 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:56.331 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:56.331 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:56.331 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:56.331 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:56.331 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:56.331 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:56.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.313+0000 7f60f511a640 10 monclient: _finish_auth 0 2024-04-26T04:26:56.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.313+0000 7f60f511a640 10 monclient: _check_auth_tickets 2024-04-26T04:26:56.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.313+0000 7f60f511a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:26:56.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.313+0000 7f60f511a640 1 -- 172.21.15.63:0/952319573 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60e00a97f0 con 0x7f60f00a0240 2024-04-26T04:26:56.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.313+0000 7f60f8380640 5 monclient: authenticate success, global_id 24704 2024-04-26T04:26:56.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.313+0000 7f60f8380640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T04:26:56.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.313+0000 7f60f511a640 10 monclient: handle_config config(0 keys) 2024-04-26T04:26:56.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.313+0000 7f60f8380640 10 monclient: get_monmap_and_config success 2024-04-26T04:26:56.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.313+0000 7f60f511a640 1 -- 172.21.15.63:0/952319573 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60e00a9bc0 con 0x7f60f00a0240 2024-04-26T04:26:56.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.313+0000 7f60f8380640 4 set_mon_vals no callback set 2024-04-26T04:26:56.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.313+0000 7f60f511a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:56.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.313+0000 7f60f511a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:26:56.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.313+0000 7f60f511a640 10 monclient: dump: 2024-04-26T04:26:56.333 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:56.334 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:56.334 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:56.334 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:56.334 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:56.334 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:56.334 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:56.335 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:56.335 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:56.335 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:56.335 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.313+0000 7f60f8380640 10 monclient: shutdown 2024-04-26T04:26:56.335 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.313+0000 7f60f8380640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:26:56.335 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.313+0000 7f60f8380640 1 -- 172.21.15.63:0/952319573 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f60f00a0240 msgr2=0x7f60f00a2a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:56.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.313+0000 7f60f8380640 1 --2- 172.21.15.63:0/952319573 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f60f00a0240 0x7f60f00a2a30 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f60e00097f0 tx=0x7f60e001c320 comp rx=0 tx=0).stop 2024-04-26T04:26:56.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.313+0000 7f60f8380640 1 -- 172.21.15.63:0/952319573 shutdown_connections 2024-04-26T04:26:56.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.313+0000 7f60f8380640 1 --2- 172.21.15.63:0/952319573 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f60f00a0240 0x7f60f00a2a30 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:56.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.313+0000 7f60f8380640 1 -- 172.21.15.63:0/952319573 >> 172.21.15.63:0/952319573 conn(0x7f60f00a5fb0 msgr2=0x7f60f00a63c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:26:56.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.313+0000 7f60f8380640 1 -- 172.21.15.63:0/952319573 shutdown_connections 2024-04-26T04:26:56.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.313+0000 7f60f8380640 1 -- 172.21.15.63:0/952319573 wait complete. 2024-04-26T04:26:56.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.313+0000 7f60f8380640 10 monclient: build_initial_monmap 2024-04-26T04:26:56.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.313+0000 7f60f8380640 10 monclient: monmap: 2024-04-26T04:26:56.337 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:26:56.337 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:26:56.337 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:26:56.337 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:26:56.338 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:26:56.338 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:56.338 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:26:56.338 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:26:56.338 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:26:56.338 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:56.339 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.313+0000 7f60f8380640 1 Processor -- start 2024-04-26T04:26:56.339 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.313+0000 7f60f8380640 1 -- start start 2024-04-26T04:26:56.339 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.313+0000 7f60f8380640 10 monclient: init 2024-04-26T04:26:56.339 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.313+0000 7f60f8380640 10 monclient: _reopen_session rank -1 2024-04-26T04:26:56.339 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.313+0000 7f60f8380640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T04:26:56.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.313+0000 7f60f8380640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f60f009fe60 0x7f60f0137df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:56.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.313+0000 7f60f8380640 10 monclient(hunting): picked mon.noname-c con 0x7f60f009fe60 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:26:56.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.313+0000 7f60f8380640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60f00a0240 0x7f60f0138330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:56.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.313+0000 7f60f8380640 10 monclient(hunting): picked mon.noname-b con 0x7f60f00a0240 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:26:56.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.313+0000 7f60f8380640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f60f00a2f70 0x7f60f0138870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:56.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.313+0000 7f60f8380640 10 monclient(hunting): picked mon.noname-a con 0x7f60f00a2f70 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:26:56.341 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.313+0000 7f60f8380640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:56.341 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.313+0000 7f60f8380640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f60f01428f0 con 0x7f60f00a2f70 2024-04-26T04:26:56.341 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.313+0000 7f60f8380640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:56.341 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.313+0000 7f60f8380640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f60f0142770 con 0x7f60f00a0240 2024-04-26T04:26:56.341 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.313+0000 7f60f8380640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:56.341 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.313+0000 7f60f8380640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f60f0142a70 con 0x7f60f009fe60 2024-04-26T04:26:56.342 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.313+0000 7f60f8380640 10 monclient(hunting): _renew_subs 2024-04-26T04:26:56.342 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.313+0000 7f60f8380640 10 monclient(hunting): authenticate will time out at 1448.027675s 2024-04-26T04:26:56.342 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.313+0000 7f60f611c640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f60f009fe60 0x7f60f0137df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:56.342 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.313+0000 7f60f691d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f60f00a2f70 0x7f60f0138870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:56.342 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.313+0000 7f60f611c640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f60f009fe60 0x7f60f0137df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:41084/0 (socket says 172.21.15.63:41084) 2024-04-26T04:26:56.342 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.313+0000 7f60f611c640 1 -- 172.21.15.63:0/2219845566 learned_addr learned my addr 172.21.15.63:0/2219845566 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:26:56.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.313+0000 7f60f611c640 10 monclient(hunting): get_auth_request con 0x7f60f009fe60 auth_method 0 2024-04-26T04:26:56.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.313+0000 7f60f611c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:56.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.313+0000 7f60f611c640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:56.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.313+0000 7f60f611c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:56.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.313+0000 7f60f691d640 10 monclient(hunting): get_auth_request con 0x7f60f00a2f70 auth_method 0 2024-04-26T04:26:56.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.313+0000 7f60f691d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:56.344 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.313+0000 7f60f691d640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:56.344 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.313+0000 7f60f691d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:56.344 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.313+0000 7f60f611c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:26:56.344 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.313+0000 7f60f611c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:26:56.344 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.313+0000 7f60f611c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:26:56.344 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.313+0000 7f60f691d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:26:56.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.313+0000 7f60f691d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:26:56.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.313+0000 7f60f691d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:26:56.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.317+0000 7f60f591b640 1 --2- 172.21.15.63:0/2219845566 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60f00a0240 0x7f60f0138330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:56.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.317+0000 7f60f591b640 10 monclient(hunting): get_auth_request con 0x7f60f00a0240 auth_method 0 2024-04-26T04:26:56.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.317+0000 7f60f591b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:56.345 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.317+0000 7f60f591b640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:56.346 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.317+0000 7f60f591b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:56.346 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.317+0000 7f60f611c640 10 monclient(hunting): handle_auth_done global_id 24707 payload 824 2024-04-26T04:26:56.346 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.317+0000 7f60f611c640 1 -- 172.21.15.63:0/2219845566 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60f00a0240 msgr2=0x7f60f0138330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:56.346 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.317+0000 7f60f611c640 1 --2- 172.21.15.63:0/2219845566 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60f00a0240 0x7f60f0138330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:56.346 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.317+0000 7f60f611c640 1 -- 172.21.15.63:0/2219845566 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f60f00a2f70 msgr2=0x7f60f0138870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:56.346 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.317+0000 7f60f611c640 1 --2- 172.21.15.63:0/2219845566 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f60f00a2f70 0x7f60f0138870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:56.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.317+0000 7f60f611c640 10 monclient: _finish_hunting 0 2024-04-26T04:26:56.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.317+0000 7f60f611c640 1 monclient: found mon.noname-c 2024-04-26T04:26:56.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.317+0000 7f60f611c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:26:56.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.317+0000 7f60f611c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.63:3301/0 2024-04-26T04:26:56.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.317+0000 7f60f611c640 1 -- 172.21.15.63:0/2219845566 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60f0139cc0 con 0x7f60f009fe60 2024-04-26T04:26:56.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.317+0000 7f60f691d640 1 --2- 172.21.15.63:0/2219845566 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f60f00a2f70 0x7f60f0138870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T04:26:56.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.317+0000 7f60f611c640 1 -- 172.21.15.63:0/2219845566 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f60f009fe60 msgr2=0x7f60f0137df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:26:56.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.317+0000 7f60f611c640 1 --2- 172.21.15.63:0/2219845566 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f60f009fe60 0x7f60f0137df0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f60ec00c470 tx=0x7f60ec00c930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:26:56.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.317+0000 7f60e6ffd640 1 -- 172.21.15.63:0/2219845566 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60ec007070 con 0x7f60f009fe60 2024-04-26T04:26:56.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.317+0000 7f60e6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:56.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.317+0000 7f60e6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:26:56.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.317+0000 7f60e6ffd640 10 monclient: dump: 2024-04-26T04:26:56.348 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:56.348 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:56.348 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:56.348 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:56.348 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:56.348 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:56.349 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:56.349 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:56.349 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:56.349 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:56.349 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.317+0000 7f60e6ffd640 10 monclient: _finish_auth 0 2024-04-26T04:26:56.349 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.317+0000 7f60e6ffd640 10 monclient: _check_auth_tickets 2024-04-26T04:26:56.349 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.317+0000 7f60e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:26:56.349 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.317+0000 7f60e6ffd640 1 -- 172.21.15.63:0/2219845566 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60ec007210 con 0x7f60f009fe60 2024-04-26T04:26:56.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.317+0000 7f60f8380640 5 monclient: authenticate success, global_id 24707 2024-04-26T04:26:56.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.317+0000 7f60f8380640 10 monclient: _renew_subs 2024-04-26T04:26:56.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.317+0000 7f60f8380640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:26:56.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.317+0000 7f60f8380640 1 -- 172.21.15.63:0/2219845566 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60f00a38f0 con 0x7f60f009fe60 2024-04-26T04:26:56.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.317+0000 7f60e6ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T04:26:56.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.317+0000 7f60e6ffd640 1 -- 172.21.15.63:0/2219845566 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60ec007600 con 0x7f60f009fe60 2024-04-26T04:26:56.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.317+0000 7f60e6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:56.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.317+0000 7f60e6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:26:56.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.317+0000 7f60e6ffd640 10 monclient: dump: 2024-04-26T04:26:56.351 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:56.351 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:56.351 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:56.351 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:56.351 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:56.352 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:56.352 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:56.352 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:56.352 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:56.352 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:56.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.317+0000 7f60f4919640 4 set_mon_vals no callback set 2024-04-26T04:26:56.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.317+0000 7f60f8380640 10 monclient: _renew_subs 2024-04-26T04:26:56.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.317+0000 7f60f8380640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:26:56.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.317+0000 7f60f8380640 1 -- 172.21.15.63:0/2219845566 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f60f013a520 con 0x7f60f009fe60 2024-04-26T04:26:56.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.317+0000 7f60e6ffd640 1 -- 172.21.15.63:0/2219845566 <== mon.2 v2:172.21.15.63:3301/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f60ec01b400 con 0x7f60f009fe60 2024-04-26T04:26:56.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.317+0000 7f60e6ffd640 1 --2- 172.21.15.63:0/2219845566 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f60d4035ec0 0x7f60d4038380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:56.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.317+0000 7f60e6ffd640 1 -- 172.21.15.63:0/2219845566 <== mon.2 v2:172.21.15.63:3301/0 5 ==== osd_map(323..323 src has 1..323) ==== 4346+0+0 (secure 0 0 0) 0x7f60ec048a10 con 0x7f60f009fe60 2024-04-26T04:26:56.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.317+0000 7f60f8380640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:26:56.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.317+0000 7f60f8380640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:26:56.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.317+0000 7f60f8380640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:26:56.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.317+0000 7f60f8380640 1 -- 172.21.15.63:0/2219845566 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60f0062e10 con 0x7f60f009fe60 2024-04-26T04:26:56.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.321+0000 7f60f591b640 1 --2- 172.21.15.63:0/2219845566 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f60d4035ec0 0x7f60d4038380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:56.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.321+0000 7f60f591b640 10 monclient: get_auth_request con 0x7f60d4035ec0 auth_method 0 2024-04-26T04:26:56.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.321+0000 7f60f591b640 1 --2- 172.21.15.63:0/2219845566 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f60d4035ec0 0x7f60d4038380 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f60f00a7010 tx=0x7f60e00a4f90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:26:56.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.325+0000 7f60e6ffd640 1 -- 172.21.15.63:0/2219845566 <== mon.2 v2:172.21.15.63:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f60ec01b8d0 con 0x7f60f009fe60 2024-04-26T04:26:56.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.325+0000 7f60e6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:26:56.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.325+0000 7f60e6ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:26:56.488 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.481+0000 7f60f8380640 1 -- 172.21.15.63:0/2219845566 --> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f60f013b230 con 0x7f60d4035ec0 2024-04-26T04:26:56.492 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.485+0000 7f60e6ffd640 1 -- 172.21.15.63:0/2219845566 <== mgr.4105 v2:172.21.15.186:6824/2583706639 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33778 (secure 0 0 0) 0x7f60f013b230 con 0x7f60d4035ec0 2024-04-26T04:26:56.492 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-26T04:26:56.493 INFO:teuthology.orchestra.run.smithi063.stderr:dumped all 2024-04-26T04:26:56.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.489+0000 7f60f8380640 1 -- 172.21.15.63:0/2219845566 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f60d4035ec0 msgr2=0x7f60d4038380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:56.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.489+0000 7f60f8380640 1 --2- 172.21.15.63:0/2219845566 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f60d4035ec0 0x7f60d4038380 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f60f00a7010 tx=0x7f60e00a4f90 comp rx=0 tx=0).stop 2024-04-26T04:26:56.494 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.489+0000 7f60f8380640 10 monclient: shutdown 2024-04-26T04:26:56.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.489+0000 7f60f8380640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:26:56.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.489+0000 7f60f8380640 1 -- 172.21.15.63:0/2219845566 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f60f009fe60 msgr2=0x7f60f0137df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:56.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.489+0000 7f60f8380640 1 --2- 172.21.15.63:0/2219845566 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f60f009fe60 0x7f60f0137df0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f60ec00c470 tx=0x7f60ec00c930 comp rx=0 tx=0).stop 2024-04-26T04:26:56.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.489+0000 7f60f8380640 1 -- 172.21.15.63:0/2219845566 shutdown_connections 2024-04-26T04:26:56.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.489+0000 7f60f8380640 1 --2- 172.21.15.63:0/2219845566 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f60d4035ec0 0x7f60d4038380 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:56.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.489+0000 7f60f8380640 1 --2- 172.21.15.63:0/2219845566 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f60f00a2f70 0x7f60f0138870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:56.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.489+0000 7f60f8380640 1 --2- 172.21.15.63:0/2219845566 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60f00a0240 0x7f60f0138330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:56.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.489+0000 7f60f8380640 1 --2- 172.21.15.63:0/2219845566 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f60f009fe60 0x7f60f0137df0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:56.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.489+0000 7f60f8380640 1 -- 172.21.15.63:0/2219845566 >> 172.21.15.63:0/2219845566 conn(0x7f60f00a5fb0 msgr2=0x7f60f00a19e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:26:56.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.489+0000 7f60f8380640 1 -- 172.21.15.63:0/2219845566 shutdown_connections 2024-04-26T04:26:56.495 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.489+0000 7f60f8380640 1 -- 172.21.15.63:0/2219845566 wait complete. 2024-04-26T04:26:56.511 INFO:teuthology.orchestra.run.smithi063.stdout:{"pg_ready":true,"pg_map":{"version":564,"stamp":"2024-04-26T04:26:55.592036+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":205168,"kb_used_data":26408,"kb_used_omap":9,"kb_used_meta":178678,"kb_avail":566025872,"statfs":{"total":579820584960,"available":579610492928,"internally_reserved":0,"allocated":27041792,"data_stored":10728854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9567,"internal_metadata":182966945},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003047"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":617,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-04-26T04:26:38.082572+0000","last_change":"2024-04-26T04:18:47.131857+0000","last_active":"2024-04-26T04:26:38.082572+0000","last_peered":"2024-04-26T04:26:38.082572+0000","last_clean":"2024-04-26T04:26:38.082572+0000","last_became_active":"2024-04-26T04:18:47.131175+0000","last_became_peered":"2024-04-26T04:18:47.131175+0000","last_unstale":"2024-04-26T04:26:38.082572+0000","last_undegraded":"2024-04-26T04:26:38.082572+0000","last_fullsized":"2024-04-26T04:26:38.082572+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_clean_scrub_stamp":"2024-04-26T04:18:45.482773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:35:03.860574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":610,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-04-26T04:26:37.279334+0000","last_change":"2024-04-26T04:18:46.499700+0000","last_active":"2024-04-26T04:26:37.279334+0000","last_peered":"2024-04-26T04:26:37.279334+0000","last_clean":"2024-04-26T04:26:37.279334+0000","last_became_active":"2024-04-26T04:18:46.498983+0000","last_became_peered":"2024-04-26T04:18:46.498983+0000","last_unstale":"2024-04-26T04:26:37.279334+0000","last_undegraded":"2024-04-26T04:26:37.279334+0000","last_fullsized":"2024-04-26T04:26:37.279334+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_clean_scrub_stamp":"2024-04-26T04:18:45.482773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:51:56.740670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":617,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-04-26T04:26:38.082586+0000","last_change":"2024-04-26T04:18:46.507904+0000","last_active":"2024-04-26T04:26:38.082586+0000","last_peered":"2024-04-26T04:26:38.082586+0000","last_clean":"2024-04-26T04:26:38.082586+0000","last_became_active":"2024-04-26T04:18:46.507335+0000","last_became_peered":"2024-04-26T04:18:46.507335+0000","last_unstale":"2024-04-26T04:26:38.082586+0000","last_undegraded":"2024-04-26T04:26:38.082586+0000","last_fullsized":"2024-04-26T04:26:38.082586+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_clean_scrub_stamp":"2024-04-26T04:18:45.482773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:52:50.029369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":611,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-04-26T04:26:37.276307+0000","last_change":"2024-04-26T04:18:46.506682+0000","last_active":"2024-04-26T04:26:37.276307+0000","last_peered":"2024-04-26T04:26:37.276307+0000","last_clean":"2024-04-26T04:26:37.276307+0000","last_became_active":"2024-04-26T04:18:46.506209+0000","last_became_peered":"2024-04-26T04:18:46.506209+0000","last_unstale":"2024-04-26T04:26:37.276307+0000","last_undegraded":"2024-04-26T04:26:37.276307+0000","last_fullsized":"2024-04-26T04:26:37.276307+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_clean_scrub_stamp":"2024-04-26T04:18:45.482773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:20:00.475307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":613,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-04-26T04:26:37.275301+0000","last_change":"2024-04-26T04:18:46.504685+0000","last_active":"2024-04-26T04:26:37.275301+0000","last_peered":"2024-04-26T04:26:37.275301+0000","last_clean":"2024-04-26T04:26:37.275301+0000","last_became_active":"2024-04-26T04:18:46.504292+0000","last_became_peered":"2024-04-26T04:18:46.504292+0000","last_unstale":"2024-04-26T04:26:37.275301+0000","last_undegraded":"2024-04-26T04:26:37.275301+0000","last_fullsized":"2024-04-26T04:26:37.275301+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_clean_scrub_stamp":"2024-04-26T04:18:45.482773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:42:29.110482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":605,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-04-26T04:26:38.261871+0000","last_change":"2024-04-26T04:18:46.504010+0000","last_active":"2024-04-26T04:26:38.261871+0000","last_peered":"2024-04-26T04:26:38.261871+0000","last_clean":"2024-04-26T04:26:38.261871+0000","last_became_active":"2024-04-26T04:18:46.503343+0000","last_became_peered":"2024-04-26T04:18:46.503343+0000","last_unstale":"2024-04-26T04:26:38.261871+0000","last_undegraded":"2024-04-26T04:26:38.261871+0000","last_fullsized":"2024-04-26T04:26:38.261871+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_clean_scrub_stamp":"2024-04-26T04:18:45.482773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:19:38.324198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":618,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-04-26T04:26:38.082607+0000","last_change":"2024-04-26T04:18:46.503233+0000","last_active":"2024-04-26T04:26:38.082607+0000","last_peered":"2024-04-26T04:26:38.082607+0000","last_clean":"2024-04-26T04:26:38.082607+0000","last_became_active":"2024-04-26T04:18:46.502682+0000","last_became_peered":"2024-04-26T04:18:46.502682+0000","last_unstale":"2024-04-26T04:26:38.082607+0000","last_undegraded":"2024-04-26T04:26:38.082607+0000","last_fullsized":"2024-04-26T04:26:38.082607+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_clean_scrub_stamp":"2024-04-26T04:18:45.482773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:26:15.760818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":613,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-04-26T04:26:37.275301+0000","last_change":"2024-04-26T04:18:46.504844+0000","last_active":"2024-04-26T04:26:37.275301+0000","last_peered":"2024-04-26T04:26:37.275301+0000","last_clean":"2024-04-26T04:26:37.275301+0000","last_became_active":"2024-04-26T04:18:46.504332+0000","last_became_peered":"2024-04-26T04:18:46.504332+0000","last_unstale":"2024-04-26T04:26:37.275301+0000","last_undegraded":"2024-04-26T04:26:37.275301+0000","last_fullsized":"2024-04-26T04:26:37.275301+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_clean_scrub_stamp":"2024-04-26T04:18:45.482773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:13:26.538440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":770,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-04-26T04:26:38.082546+0000","last_change":"2024-04-26T04:18:45.505818+0000","last_active":"2024-04-26T04:26:38.082546+0000","last_peered":"2024-04-26T04:26:38.082546+0000","last_clean":"2024-04-26T04:26:38.082546+0000","last_became_active":"2024-04-26T04:18:45.504887+0000","last_became_peered":"2024-04-26T04:18:45.504887+0000","last_unstale":"2024-04-26T04:26:38.082546+0000","last_undegraded":"2024-04-26T04:26:38.082546+0000","last_fullsized":"2024-04-26T04:26:38.082546+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:44.470061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:44.470061+0000","last_clean_scrub_stamp":"2024-04-26T04:18:44.470061+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:42:56.629937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":12,"seq":51539607654,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35928,"kb_used_data":4748,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335912,"statfs":{"total":96636764160,"available":96599973888,"internally_reserved":0,"allocated":4861952,"data_stored":2140761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":12,"seq":51539607654,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35408,"kb_used_data":4228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94336432,"statfs":{"total":96636764160,"available":96600506368,"internally_reserved":0,"allocated":4329472,"data_stored":1611833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607654,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35408,"kb_used_data":4228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94336432,"statfs":{"total":96636764160,"available":96600506368,"internally_reserved":0,"allocated":4329472,"data_stored":1611833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640358,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35408,"kb_used_data":4228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94336432,"statfs":{"total":96636764160,"available":96600506368,"internally_reserved":0,"allocated":4329472,"data_stored":1611833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640357,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31248,"kb_used_data":4228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340592,"statfs":{"total":96636764160,"available":96604766208,"internally_reserved":0,"allocated":4329472,"data_stored":1611833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640357,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31768,"kb_used_data":4748,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340072,"statfs":{"total":96636764160,"available":96604233728,"internally_reserved":0,"allocated":4861952,"data_stored":2140761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T04:26:56.512 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-26T04:26:56.512 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T04:26:56.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.629+0000 7f986107f640 10 monclient: get_monmap_and_config 2024-04-26T04:26:56.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.629+0000 7f986107f640 10 monclient: build_initial_monmap 2024-04-26T04:26:56.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.629+0000 7f986107f640 10 monclient: monmap: 2024-04-26T04:26:56.633 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:26:56.633 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:56.634 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:26:56.633007+0000 2024-04-26T04:26:56.634 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:26:56.633007+0000 2024-04-26T04:26:56.634 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:26:56.634 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:56.634 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:26:56.635 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:26:56.635 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:26:56.635 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:26:56.635 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:26:56.635 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:56.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.629+0000 7f986107f640 1 Processor -- start 2024-04-26T04:26:56.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.629+0000 7f986107f640 1 -- start start 2024-04-26T04:26:56.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.629+0000 7f986107f640 10 monclient: init 2024-04-26T04:26:56.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.629+0000 7f986107f640 10 monclient: _reopen_session rank -1 2024-04-26T04:26:56.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.629+0000 7f986107f640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-04-26T04:26:56.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.629+0000 7f986107f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f985c0a0bb0 addr v1:172.21.15.63:6789/0 2024-04-26T04:26:56.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.629+0000 7f986107f640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f985c136310 0x7f985c138700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:56.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.629+0000 7f986107f640 10 monclient(hunting): picked mon.noname-c con 0x7f985c136310 addr v2:172.21.15.186:3300/0 2024-04-26T04:26:56.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.629+0000 7f986107f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f985c138c40 addr v1:172.21.15.186:6789/0 2024-04-26T04:26:56.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.629+0000 7f986107f640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f985c09f270 con 0x7f985c0a0bb0 2024-04-26T04:26:56.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.629+0000 7f986107f640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f985c0a0f90 con 0x7f985c138c40 2024-04-26T04:26:56.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.629+0000 7f986107f640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:56.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.629+0000 7f986107f640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f985c099240 con 0x7f985c136310 2024-04-26T04:26:56.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.629+0000 7f986107f640 10 monclient(hunting): _renew_subs 2024-04-26T04:26:56.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.629+0000 7f986107f640 10 monclient(hunting): authenticate will time out at 1448.341074s 2024-04-26T04:26:56.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.629+0000 7f985ad9d640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f985c0a0bb0 0x7f985c09eb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:36772/0 (socket says 172.21.15.63:36772) 2024-04-26T04:26:56.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.629+0000 7f985ad9d640 1 -- 172.21.15.63:0/4110736125 learned_addr learned my addr 172.21.15.63:0/4110736125 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-04-26T04:26:56.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.629+0000 7f985a59c640 1 --2- 172.21.15.63:0/4110736125 >> v2:172.21.15.186:3300/0 conn(0x7f985c136310 0x7f985c138700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:56.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.629+0000 7f985a59c640 10 monclient(hunting): get_auth_request con 0x7f985c136310 auth_method 0 2024-04-26T04:26:56.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.629+0000 7f985a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:56.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.629+0000 7f985a59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:56.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.629+0000 7f985a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:56.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.629+0000 7f985a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:26:56.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.629+0000 7f985a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:26:56.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.629+0000 7f985a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:26:56.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.629+0000 7f985a59c640 10 monclient(hunting): handle_auth_done global_id 34573 payload 274 2024-04-26T04:26:56.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.629+0000 7f985a59c640 1 -- 172.21.15.63:0/4110736125 >> v1:172.21.15.186:6789/0 conn(0x7f985c138c40 legacy=0x7f985c13b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:26:56.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.629+0000 7f985a59c640 1 -- 172.21.15.63:0/4110736125 >> v1:172.21.15.63:6789/0 conn(0x7f985c0a0bb0 legacy=0x7f985c09eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:26:56.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.629+0000 7f985a59c640 10 monclient: _finish_hunting 0 2024-04-26T04:26:56.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.629+0000 7f985a59c640 1 monclient: found mon.noname-c 2024-04-26T04:26:56.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.629+0000 7f985a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:26:56.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.629+0000 7f985a59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.186:3300/0 2024-04-26T04:26:56.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.629+0000 7f985a59c640 1 -- 172.21.15.63:0/4110736125 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f985c09b820 con 0x7f985c136310 2024-04-26T04:26:56.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.633+0000 7f985a59c640 1 -- 172.21.15.63:0/4110736125 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f985c136310 msgr2=0x7f985c138700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:26:56.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.633+0000 7f985a59c640 1 --2- 172.21.15.63:0/4110736125 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f985c136310 0x7f985c138700 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f98500097f0 tx=0x7f985001c0d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6670b017c6b09e9d server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:26:56.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.633+0000 7f9859d9b640 1 -- 172.21.15.63:0/4110736125 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98500b0070 con 0x7f985c136310 2024-04-26T04:26:56.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.633+0000 7f9859d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:56.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.633+0000 7f9859d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:26:56.646 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.633+0000 7f9859d9b640 10 monclient: dump: 2024-04-26T04:26:56.646 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:56.646 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:56.646 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:56.646 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:56.646 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:56.646 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:56.646 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:56.646 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:56.647 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:56.647 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:56.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.633+0000 7f9859d9b640 10 monclient: _finish_auth 0 2024-04-26T04:26:56.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.633+0000 7f9859d9b640 10 monclient: _check_auth_tickets 2024-04-26T04:26:56.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.633+0000 7f9859d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:26:56.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.633+0000 7f9859d9b640 1 -- 172.21.15.63:0/4110736125 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98500ab600 con 0x7f985c136310 2024-04-26T04:26:56.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.633+0000 7f9859d9b640 10 monclient: handle_config config(0 keys) 2024-04-26T04:26:56.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.633+0000 7f986107f640 5 monclient: authenticate success, global_id 34573 2024-04-26T04:26:56.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.633+0000 7f986107f640 10 monclient: get_monmap_and_config success 2024-04-26T04:26:56.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.633+0000 7f986107f640 4 set_mon_vals no callback set 2024-04-26T04:26:56.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.633+0000 7f9859d9b640 1 -- 172.21.15.63:0/4110736125 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98500ab9d0 con 0x7f985c136310 2024-04-26T04:26:56.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.633+0000 7f9859d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:56.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.633+0000 7f9859d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-04-26T04:26:56.647 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.633+0000 7f9859d9b640 10 monclient: dump: 2024-04-26T04:26:56.647 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:56.648 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:56.648 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:56.648 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:56.648 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:56.648 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:56.648 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:56.648 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:56.648 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:56.648 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:56.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.637+0000 7f986107f640 10 monclient: shutdown 2024-04-26T04:26:56.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.637+0000 7f986107f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:26:56.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.637+0000 7f986107f640 1 -- 172.21.15.63:0/4110736125 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f985c136310 msgr2=0x7f985c138700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:56.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.637+0000 7f986107f640 1 --2- 172.21.15.63:0/4110736125 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f985c136310 0x7f985c138700 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f98500097f0 tx=0x7f985001c0d0 comp rx=0 tx=0).stop 2024-04-26T04:26:56.648 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.637+0000 7f986107f640 1 -- 172.21.15.63:0/4110736125 shutdown_connections 2024-04-26T04:26:56.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.637+0000 7f986107f640 1 --2- 172.21.15.63:0/4110736125 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f985c136310 0x7f985c138700 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:56.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.637+0000 7f986107f640 1 -- 172.21.15.63:0/4110736125 >> 172.21.15.63:0/4110736125 conn(0x7f985c0a1e70 msgr2=0x7f985c0a2280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:26:56.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.637+0000 7f986107f640 1 -- 172.21.15.63:0/4110736125 shutdown_connections 2024-04-26T04:26:56.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.637+0000 7f986107f640 1 -- 172.21.15.63:0/4110736125 wait complete. 2024-04-26T04:26:56.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.637+0000 7f986107f640 10 monclient: build_initial_monmap 2024-04-26T04:26:56.649 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.637+0000 7f986107f640 10 monclient: monmap: 2024-04-26T04:26:56.649 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:26:56.649 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:26:56.649 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:26:56.649 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:26:56.649 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:26:56.649 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:56.649 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:26:56.649 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:26:56.649 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:26:56.650 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:56.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.637+0000 7f986107f640 1 Processor -- start 2024-04-26T04:26:56.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.637+0000 7f986107f640 1 -- start start 2024-04-26T04:26:56.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.637+0000 7f986107f640 10 monclient: init 2024-04-26T04:26:56.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.637+0000 7f986107f640 10 monclient: _reopen_session rank -1 2024-04-26T04:26:56.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.637+0000 7f986107f640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T04:26:56.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.637+0000 7f986107f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f985c0a0bb0 0x7f985c12a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:56.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.637+0000 7f986107f640 10 monclient(hunting): picked mon.noname-b con 0x7f985c0a0bb0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:26:56.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.637+0000 7f986107f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f985c136310 0x7f985c1356f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:56.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.637+0000 7f986107f640 10 monclient(hunting): picked mon.noname-a con 0x7f985c136310 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:26:56.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.637+0000 7f986107f640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f985c138c40 0x7f985c135c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:56.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.637+0000 7f986107f640 10 monclient(hunting): picked mon.noname-c con 0x7f985c138c40 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:26:56.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.637+0000 7f986107f640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:56.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.637+0000 7f986107f640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f985c13f450 con 0x7f985c136310 2024-04-26T04:26:56.650 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.637+0000 7f986107f640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:56.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.637+0000 7f986107f640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f985c13f2d0 con 0x7f985c0a0bb0 2024-04-26T04:26:56.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.637+0000 7f986107f640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:56.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.637+0000 7f986107f640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f985c13f5d0 con 0x7f985c138c40 2024-04-26T04:26:56.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.637+0000 7f986107f640 10 monclient(hunting): _renew_subs 2024-04-26T04:26:56.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.637+0000 7f986107f640 10 monclient(hunting): authenticate will time out at 1448.350564s 2024-04-26T04:26:56.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.637+0000 7f985a59c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f985c136310 0x7f985c1356f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:56.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.637+0000 7f985b59e640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f985c138c40 0x7f985c135c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:56.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.637+0000 7f985a59c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f985c136310 0x7f985c1356f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:55356/0 (socket says 172.21.15.63:55356) 2024-04-26T04:26:56.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.637+0000 7f985a59c640 1 -- 172.21.15.63:0/4128690679 learned_addr learned my addr 172.21.15.63:0/4128690679 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:26:56.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.637+0000 7f985a59c640 10 monclient(hunting): get_auth_request con 0x7f985c136310 auth_method 0 2024-04-26T04:26:56.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.637+0000 7f985a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:56.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.637+0000 7f985a59c640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:56.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.637+0000 7f985a59c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:56.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.637+0000 7f985b59e640 10 monclient(hunting): get_auth_request con 0x7f985c138c40 auth_method 0 2024-04-26T04:26:56.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.637+0000 7f985b59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:56.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.637+0000 7f985b59e640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:56.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.637+0000 7f985b59e640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:56.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.637+0000 7f985a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:26:56.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.637+0000 7f985a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:26:56.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.637+0000 7f985a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:26:56.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.637+0000 7f985b59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:26:56.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.637+0000 7f985b59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:26:56.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.637+0000 7f985b59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:26:56.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.637+0000 7f985ad9d640 1 --2- 172.21.15.63:0/4128690679 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f985c0a0bb0 0x7f985c12a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:56.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.637+0000 7f985ad9d640 10 monclient(hunting): get_auth_request con 0x7f985c0a0bb0 auth_method 0 2024-04-26T04:26:56.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.637+0000 7f985ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:56.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.637+0000 7f985ad9d640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:56.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.637+0000 7f985ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:56.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.637+0000 7f985a59c640 10 monclient(hunting): handle_auth_done global_id 44148 payload 824 2024-04-26T04:26:56.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.637+0000 7f985a59c640 1 -- 172.21.15.63:0/4128690679 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f985c138c40 msgr2=0x7f985c135c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:56.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.637+0000 7f985a59c640 1 --2- 172.21.15.63:0/4128690679 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f985c138c40 0x7f985c135c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:56.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.637+0000 7f985a59c640 1 -- 172.21.15.63:0/4128690679 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f985c0a0bb0 msgr2=0x7f985c12a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:56.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.637+0000 7f985a59c640 1 --2- 172.21.15.63:0/4128690679 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f985c0a0bb0 0x7f985c12a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:56.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.637+0000 7f985a59c640 10 monclient: _finish_hunting 0 2024-04-26T04:26:56.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.637+0000 7f985a59c640 1 monclient: found mon.noname-a 2024-04-26T04:26:56.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.637+0000 7f985a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:26:56.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.637+0000 7f985a59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:26:56.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.637+0000 7f985a59c640 1 -- 172.21.15.63:0/4128690679 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f985c12c610 con 0x7f985c136310 2024-04-26T04:26:56.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.637+0000 7f985b59e640 1 --2- 172.21.15.63:0/4128690679 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f985c138c40 0x7f985c135c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T04:26:56.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.641+0000 7f985a59c640 1 -- 172.21.15.63:0/4128690679 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f985c136310 msgr2=0x7f985c1356f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:26:56.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.641+0000 7f985a59c640 1 --2- 172.21.15.63:0/4128690679 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f985c136310 0x7f985c1356f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9850019d00 tx=0x7f985001a910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:26:56.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.641+0000 7f98437fe640 1 -- 172.21.15.63:0/4128690679 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98500b0050 con 0x7f985c136310 2024-04-26T04:26:56.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.641+0000 7f98437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:56.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.641+0000 7f98437fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:26:56.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.641+0000 7f98437fe640 10 monclient: dump: 2024-04-26T04:26:56.655 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:56.656 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:56.656 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:56.656 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:56.656 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:56.656 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:56.656 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:56.656 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:56.656 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:56.656 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:56.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.641+0000 7f98437fe640 10 monclient: _finish_auth 0 2024-04-26T04:26:56.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.641+0000 7f98437fe640 10 monclient: _check_auth_tickets 2024-04-26T04:26:56.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.641+0000 7f98437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:26:56.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.641+0000 7f98437fe640 1 -- 172.21.15.63:0/4128690679 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98500bc070 con 0x7f985c136310 2024-04-26T04:26:56.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.641+0000 7f98437fe640 10 monclient: handle_config config(0 keys) 2024-04-26T04:26:56.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.641+0000 7f98437fe640 1 -- 172.21.15.63:0/4128690679 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98500b5a60 con 0x7f985c136310 2024-04-26T04:26:56.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.641+0000 7f986107f640 5 monclient: authenticate success, global_id 44148 2024-04-26T04:26:56.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.641+0000 7f986107f640 10 monclient: _renew_subs 2024-04-26T04:26:56.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.641+0000 7f986107f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:26:56.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.641+0000 7f986107f640 1 -- 172.21.15.63:0/4128690679 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f985c12c8f0 con 0x7f985c136310 2024-04-26T04:26:56.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.641+0000 7f985959a640 4 set_mon_vals no callback set 2024-04-26T04:26:56.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.641+0000 7f98437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:56.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.641+0000 7f98437fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:26:56.657 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.641+0000 7f98437fe640 10 monclient: dump: 2024-04-26T04:26:56.657 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:56.657 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:56.657 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:56.657 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:56.657 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:56.657 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:56.657 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:56.658 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:56.658 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:56.658 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:56.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.641+0000 7f986107f640 10 monclient: _renew_subs 2024-04-26T04:26:56.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.641+0000 7f986107f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:26:56.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.641+0000 7f986107f640 1 -- 172.21.15.63:0/4128690679 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f985c12ac60 con 0x7f985c136310 2024-04-26T04:26:56.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.641+0000 7f986107f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:26:56.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.641+0000 7f986107f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:26:56.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.641+0000 7f986107f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:26:56.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.641+0000 7f98437fe640 1 -- 172.21.15.63:0/4128690679 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f98500b5c00 con 0x7f985c136310 2024-04-26T04:26:56.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.641+0000 7f986107f640 1 -- 172.21.15.63:0/4128690679 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f985c062e10 con 0x7f985c136310 2024-04-26T04:26:56.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.641+0000 7f98437fe640 1 --2- 172.21.15.63:0/4128690679 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f9848035ef0 0x7f98480383b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:56.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.641+0000 7f98437fe640 1 -- 172.21.15.63:0/4128690679 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(323..323 src has 1..323) ==== 4346+0+0 (secure 0 0 0) 0x7f98500e38c0 con 0x7f985c136310 2024-04-26T04:26:56.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.641+0000 7f985ad9d640 1 --2- 172.21.15.63:0/4128690679 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f9848035ef0 0x7f98480383b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:56.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.641+0000 7f985ad9d640 10 monclient: get_auth_request con 0x7f9848035ef0 auth_method 0 2024-04-26T04:26:56.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.645+0000 7f985ad9d640 1 --2- 172.21.15.63:0/4128690679 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f9848035ef0 0x7f98480383b0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f984c007fd0 tx=0x7f984c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:26:56.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.645+0000 7f98437fe640 1 -- 172.21.15.63:0/4128690679 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f985c0a0f90 con 0x7f985c136310 2024-04-26T04:26:56.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.645+0000 7f98437fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:26:56.659 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.645+0000 7f98437fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:26:56.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.797+0000 7f986107f640 1 -- 172.21.15.63:0/4128690679 --> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f985c068430 con 0x7f9848035ef0 2024-04-26T04:26:56.807 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.805+0000 7f98437fe640 1 -- 172.21.15.63:0/4128690679 <== mgr.4105 v2:172.21.15.186:6824/2583706639 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33778 (secure 0 0 0) 0x7f984c020410 con 0x7f9848035ef0 2024-04-26T04:26:56.808 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-26T04:26:56.808 INFO:teuthology.orchestra.run.smithi063.stderr:dumped all 2024-04-26T04:26:56.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.805+0000 7f986107f640 1 -- 172.21.15.63:0/4128690679 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f9848035ef0 msgr2=0x7f98480383b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:56.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.805+0000 7f986107f640 1 --2- 172.21.15.63:0/4128690679 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f9848035ef0 0x7f98480383b0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f984c007fd0 tx=0x7f984c009040 comp rx=0 tx=0).stop 2024-04-26T04:26:56.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.805+0000 7f986107f640 10 monclient: shutdown 2024-04-26T04:26:56.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.805+0000 7f986107f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:26:56.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.805+0000 7f986107f640 1 -- 172.21.15.63:0/4128690679 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f985c136310 msgr2=0x7f985c1356f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:56.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.805+0000 7f986107f640 1 --2- 172.21.15.63:0/4128690679 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f985c136310 0x7f985c1356f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9850019d00 tx=0x7f985001a910 comp rx=0 tx=0).stop 2024-04-26T04:26:56.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.805+0000 7f986107f640 1 -- 172.21.15.63:0/4128690679 shutdown_connections 2024-04-26T04:26:56.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.805+0000 7f986107f640 1 --2- 172.21.15.63:0/4128690679 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f9848035ef0 0x7f98480383b0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:56.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.805+0000 7f986107f640 1 --2- 172.21.15.63:0/4128690679 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f985c138c40 0x7f985c135c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:56.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.805+0000 7f986107f640 1 --2- 172.21.15.63:0/4128690679 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f985c136310 0x7f985c1356f0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:56.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.805+0000 7f986107f640 1 --2- 172.21.15.63:0/4128690679 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f985c0a0bb0 0x7f985c12a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:56.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.805+0000 7f986107f640 1 -- 172.21.15.63:0/4128690679 >> 172.21.15.63:0/4128690679 conn(0x7f985c0a1e70 msgr2=0x7f985c099fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:26:56.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.805+0000 7f986107f640 1 -- 172.21.15.63:0/4128690679 shutdown_connections 2024-04-26T04:26:56.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.805+0000 7f986107f640 1 -- 172.21.15.63:0/4128690679 wait complete. 2024-04-26T04:26:56.825 INFO:teuthology.orchestra.run.smithi063.stdout:{"pg_ready":true,"pg_map":{"version":564,"stamp":"2024-04-26T04:26:55.592036+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":205168,"kb_used_data":26408,"kb_used_omap":9,"kb_used_meta":178678,"kb_avail":566025872,"statfs":{"total":579820584960,"available":579610492928,"internally_reserved":0,"allocated":27041792,"data_stored":10728854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9567,"internal_metadata":182966945},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003047"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":617,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-04-26T04:26:38.082572+0000","last_change":"2024-04-26T04:18:47.131857+0000","last_active":"2024-04-26T04:26:38.082572+0000","last_peered":"2024-04-26T04:26:38.082572+0000","last_clean":"2024-04-26T04:26:38.082572+0000","last_became_active":"2024-04-26T04:18:47.131175+0000","last_became_peered":"2024-04-26T04:18:47.131175+0000","last_unstale":"2024-04-26T04:26:38.082572+0000","last_undegraded":"2024-04-26T04:26:38.082572+0000","last_fullsized":"2024-04-26T04:26:38.082572+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_clean_scrub_stamp":"2024-04-26T04:18:45.482773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:35:03.860574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":610,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-04-26T04:26:37.279334+0000","last_change":"2024-04-26T04:18:46.499700+0000","last_active":"2024-04-26T04:26:37.279334+0000","last_peered":"2024-04-26T04:26:37.279334+0000","last_clean":"2024-04-26T04:26:37.279334+0000","last_became_active":"2024-04-26T04:18:46.498983+0000","last_became_peered":"2024-04-26T04:18:46.498983+0000","last_unstale":"2024-04-26T04:26:37.279334+0000","last_undegraded":"2024-04-26T04:26:37.279334+0000","last_fullsized":"2024-04-26T04:26:37.279334+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_clean_scrub_stamp":"2024-04-26T04:18:45.482773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:51:56.740670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":617,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-04-26T04:26:38.082586+0000","last_change":"2024-04-26T04:18:46.507904+0000","last_active":"2024-04-26T04:26:38.082586+0000","last_peered":"2024-04-26T04:26:38.082586+0000","last_clean":"2024-04-26T04:26:38.082586+0000","last_became_active":"2024-04-26T04:18:46.507335+0000","last_became_peered":"2024-04-26T04:18:46.507335+0000","last_unstale":"2024-04-26T04:26:38.082586+0000","last_undegraded":"2024-04-26T04:26:38.082586+0000","last_fullsized":"2024-04-26T04:26:38.082586+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_clean_scrub_stamp":"2024-04-26T04:18:45.482773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:52:50.029369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":611,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-04-26T04:26:37.276307+0000","last_change":"2024-04-26T04:18:46.506682+0000","last_active":"2024-04-26T04:26:37.276307+0000","last_peered":"2024-04-26T04:26:37.276307+0000","last_clean":"2024-04-26T04:26:37.276307+0000","last_became_active":"2024-04-26T04:18:46.506209+0000","last_became_peered":"2024-04-26T04:18:46.506209+0000","last_unstale":"2024-04-26T04:26:37.276307+0000","last_undegraded":"2024-04-26T04:26:37.276307+0000","last_fullsized":"2024-04-26T04:26:37.276307+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_clean_scrub_stamp":"2024-04-26T04:18:45.482773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:20:00.475307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":613,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-04-26T04:26:37.275301+0000","last_change":"2024-04-26T04:18:46.504685+0000","last_active":"2024-04-26T04:26:37.275301+0000","last_peered":"2024-04-26T04:26:37.275301+0000","last_clean":"2024-04-26T04:26:37.275301+0000","last_became_active":"2024-04-26T04:18:46.504292+0000","last_became_peered":"2024-04-26T04:18:46.504292+0000","last_unstale":"2024-04-26T04:26:37.275301+0000","last_undegraded":"2024-04-26T04:26:37.275301+0000","last_fullsized":"2024-04-26T04:26:37.275301+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_clean_scrub_stamp":"2024-04-26T04:18:45.482773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:42:29.110482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":605,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-04-26T04:26:38.261871+0000","last_change":"2024-04-26T04:18:46.504010+0000","last_active":"2024-04-26T04:26:38.261871+0000","last_peered":"2024-04-26T04:26:38.261871+0000","last_clean":"2024-04-26T04:26:38.261871+0000","last_became_active":"2024-04-26T04:18:46.503343+0000","last_became_peered":"2024-04-26T04:18:46.503343+0000","last_unstale":"2024-04-26T04:26:38.261871+0000","last_undegraded":"2024-04-26T04:26:38.261871+0000","last_fullsized":"2024-04-26T04:26:38.261871+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_clean_scrub_stamp":"2024-04-26T04:18:45.482773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:19:38.324198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":618,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-04-26T04:26:38.082607+0000","last_change":"2024-04-26T04:18:46.503233+0000","last_active":"2024-04-26T04:26:38.082607+0000","last_peered":"2024-04-26T04:26:38.082607+0000","last_clean":"2024-04-26T04:26:38.082607+0000","last_became_active":"2024-04-26T04:18:46.502682+0000","last_became_peered":"2024-04-26T04:18:46.502682+0000","last_unstale":"2024-04-26T04:26:38.082607+0000","last_undegraded":"2024-04-26T04:26:38.082607+0000","last_fullsized":"2024-04-26T04:26:38.082607+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_clean_scrub_stamp":"2024-04-26T04:18:45.482773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:26:15.760818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":613,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-04-26T04:26:37.275301+0000","last_change":"2024-04-26T04:18:46.504844+0000","last_active":"2024-04-26T04:26:37.275301+0000","last_peered":"2024-04-26T04:26:37.275301+0000","last_clean":"2024-04-26T04:26:37.275301+0000","last_became_active":"2024-04-26T04:18:46.504332+0000","last_became_peered":"2024-04-26T04:18:46.504332+0000","last_unstale":"2024-04-26T04:26:37.275301+0000","last_undegraded":"2024-04-26T04:26:37.275301+0000","last_fullsized":"2024-04-26T04:26:37.275301+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_clean_scrub_stamp":"2024-04-26T04:18:45.482773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:13:26.538440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":770,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-04-26T04:26:38.082546+0000","last_change":"2024-04-26T04:18:45.505818+0000","last_active":"2024-04-26T04:26:38.082546+0000","last_peered":"2024-04-26T04:26:38.082546+0000","last_clean":"2024-04-26T04:26:38.082546+0000","last_became_active":"2024-04-26T04:18:45.504887+0000","last_became_peered":"2024-04-26T04:18:45.504887+0000","last_unstale":"2024-04-26T04:26:38.082546+0000","last_undegraded":"2024-04-26T04:26:38.082546+0000","last_fullsized":"2024-04-26T04:26:38.082546+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:44.470061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:44.470061+0000","last_clean_scrub_stamp":"2024-04-26T04:18:44.470061+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:42:56.629937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":12,"seq":51539607654,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35928,"kb_used_data":4748,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335912,"statfs":{"total":96636764160,"available":96599973888,"internally_reserved":0,"allocated":4861952,"data_stored":2140761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":12,"seq":51539607654,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35408,"kb_used_data":4228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94336432,"statfs":{"total":96636764160,"available":96600506368,"internally_reserved":0,"allocated":4329472,"data_stored":1611833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607654,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35408,"kb_used_data":4228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94336432,"statfs":{"total":96636764160,"available":96600506368,"internally_reserved":0,"allocated":4329472,"data_stored":1611833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640358,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35408,"kb_used_data":4228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94336432,"statfs":{"total":96636764160,"available":96600506368,"internally_reserved":0,"allocated":4329472,"data_stored":1611833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640357,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31248,"kb_used_data":4228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340592,"statfs":{"total":96636764160,"available":96604766208,"internally_reserved":0,"allocated":4329472,"data_stored":1611833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640357,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31768,"kb_used_data":4748,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340072,"statfs":{"total":96636764160,"available":96604233728,"internally_reserved":0,"allocated":4861952,"data_stored":2140761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T04:26:56.825 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-26T04:26:56.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.933+0000 7ff9572cc640 10 monclient: get_monmap_and_config 2024-04-26T04:26:56.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.933+0000 7ff9572cc640 10 monclient: build_initial_monmap 2024-04-26T04:26:56.936 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.933+0000 7ff9572cc640 10 monclient: monmap: 2024-04-26T04:26:56.936 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:26:56.936 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:56.936 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:26:56.935973+0000 2024-04-26T04:26:56.936 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:26:56.935973+0000 2024-04-26T04:26:56.936 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:26:56.936 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:56.936 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:26:56.936 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:26:56.937 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:26:56.937 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:26:56.937 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:26:56.937 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:56.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.933+0000 7ff9572cc640 1 Processor -- start 2024-04-26T04:26:56.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.933+0000 7ff9572cc640 1 -- start start 2024-04-26T04:26:56.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.933+0000 7ff9572cc640 10 monclient: init 2024-04-26T04:26:56.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.933+0000 7ff9572cc640 10 monclient: _reopen_session rank -1 2024-04-26T04:26:56.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.933+0000 7ff9572cc640 10 monclient: _add_conns ranks=[3,2,0,1,4] 2024-04-26T04:26:56.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.933+0000 7ff9572cc640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7ff95009c6a0 0x7ff9500996b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:56.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.933+0000 7ff9572cc640 10 monclient(hunting): picked mon.noname-a con 0x7ff95009c6a0 addr v2:172.21.15.63:3300/0 2024-04-26T04:26:56.939 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.933+0000 7ff9572cc640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff95009ca80 0x7ff95009ce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:56.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.933+0000 7ff9572cc640 10 monclient(hunting): picked mon.noname-b con 0x7ff95009ca80 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:26:56.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.933+0000 7ff9572cc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff950098090 addr v1:172.21.15.63:6789/0 2024-04-26T04:26:56.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.933+0000 7ff9572cc640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff95009d9f0 con 0x7ff950098090 2024-04-26T04:26:56.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.933+0000 7ff9572cc640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:56.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.933+0000 7ff9572cc640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7ff95009c1d0 con 0x7ff95009c6a0 2024-04-26T04:26:56.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.933+0000 7ff9572cc640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:56.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.933+0000 7ff9572cc640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7ff9500992d0 con 0x7ff95009ca80 2024-04-26T04:26:56.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.933+0000 7ff9572cc640 10 monclient(hunting): _renew_subs 2024-04-26T04:26:56.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.933+0000 7ff9572cc640 10 monclient(hunting): authenticate will time out at 1448.644132s 2024-04-26T04:26:56.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.933+0000 7ff955068640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7ff95009c6a0 0x7ff9500996b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:56.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.933+0000 7ff954867640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff95009ca80 0x7ff95009ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:56.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.933+0000 7ff955869640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7ff950098090 0x7ff9500984a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:36776/0 (socket says 172.21.15.63:36776) 2024-04-26T04:26:56.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.933+0000 7ff955068640 10 monclient(hunting): get_auth_request con 0x7ff95009c6a0 auth_method 0 2024-04-26T04:26:56.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.933+0000 7ff955869640 1 -- 172.21.15.63:0/1528782773 learned_addr learned my addr 172.21.15.63:0/1528782773 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-04-26T04:26:56.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.933+0000 7ff955068640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:56.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.933+0000 7ff955068640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:56.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.933+0000 7ff955068640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:56.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.933+0000 7ff954867640 10 monclient(hunting): get_auth_request con 0x7ff95009ca80 auth_method 0 2024-04-26T04:26:56.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.933+0000 7ff954867640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:56.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.933+0000 7ff954867640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:56.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.933+0000 7ff954867640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:56.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.933+0000 7ff954867640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:26:56.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.933+0000 7ff954867640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:26:56.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.933+0000 7ff954867640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:26:56.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.933+0000 7ff955068640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:26:56.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.933+0000 7ff955068640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:26:56.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.933+0000 7ff955068640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:26:56.941 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.933+0000 7ff954867640 10 monclient(hunting): handle_auth_done global_id 24713 payload 274 2024-04-26T04:26:56.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.933+0000 7ff954867640 1 -- 172.21.15.63:0/1528782773 >> v2:172.21.15.63:3300/0 conn(0x7ff95009c6a0 msgr2=0x7ff9500996b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:26:56.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.933+0000 7ff954867640 1 --2- 172.21.15.63:0/1528782773 >> v2:172.21.15.63:3300/0 conn(0x7ff95009c6a0 0x7ff9500996b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:56.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.933+0000 7ff954867640 1 -- 172.21.15.63:0/1528782773 >> v1:172.21.15.63:6789/0 conn(0x7ff950098090 legacy=0x7ff9500984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:26:56.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.933+0000 7ff954867640 10 monclient: _finish_hunting 0 2024-04-26T04:26:56.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.933+0000 7ff954867640 1 monclient: found mon.noname-b 2024-04-26T04:26:56.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.933+0000 7ff954867640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:26:56.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.933+0000 7ff954867640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.63:3301/0 2024-04-26T04:26:56.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.933+0000 7ff954867640 1 -- 172.21.15.63:0/1528782773 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff95009dc30 con 0x7ff95009ca80 2024-04-26T04:26:56.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.933+0000 7ff955068640 1 --2- 172.21.15.63:0/1528782773 >> v2:172.21.15.63:3300/0 conn(0x7ff95009c6a0 0x7ff9500996b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T04:26:56.944 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff954867640 1 -- 172.21.15.63:0/1528782773 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff95009ca80 msgr2=0x7ff95009ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:26:56.944 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff954867640 1 --2- 172.21.15.63:0/1528782773 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff95009ca80 0x7ff95009ce90 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ff9500a1a00 tx=0x7ff94001c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5bf665e82b60d3a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:26:56.944 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff947fff640 1 -- 172.21.15.63:0/1528782773 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9400b0070 con 0x7ff95009ca80 2024-04-26T04:26:56.944 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff947fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:56.944 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff947fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T04:26:56.944 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff947fff640 10 monclient: dump: 2024-04-26T04:26:56.944 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:56.944 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:56.944 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:56.944 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:56.944 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:56.944 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:56.944 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:56.944 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:56.945 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:56.945 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:56.945 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff947fff640 10 monclient: _finish_auth 0 2024-04-26T04:26:56.945 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff947fff640 10 monclient: _check_auth_tickets 2024-04-26T04:26:56.945 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff947fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:26:56.945 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff947fff640 1 -- 172.21.15.63:0/1528782773 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9400ab600 con 0x7ff95009ca80 2024-04-26T04:26:56.945 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff947fff640 10 monclient: handle_config config(0 keys) 2024-04-26T04:26:56.945 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff947fff640 1 -- 172.21.15.63:0/1528782773 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9400ab9d0 con 0x7ff95009ca80 2024-04-26T04:26:56.945 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff947fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:56.945 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff947fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:26:56.945 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff947fff640 10 monclient: dump: 2024-04-26T04:26:56.945 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:56.945 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:56.945 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:56.945 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:56.946 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:56.946 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:56.946 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:56.946 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:56.946 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:56.946 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:56.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff9572cc640 5 monclient: authenticate success, global_id 24713 2024-04-26T04:26:56.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff9572cc640 10 monclient: get_monmap_and_config success 2024-04-26T04:26:56.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff9572cc640 4 set_mon_vals no callback set 2024-04-26T04:26:56.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff9572cc640 10 monclient: shutdown 2024-04-26T04:26:56.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff9572cc640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:26:56.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff9572cc640 1 -- 172.21.15.63:0/1528782773 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff95009ca80 msgr2=0x7ff95009ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:56.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff9572cc640 1 --2- 172.21.15.63:0/1528782773 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff95009ca80 0x7ff95009ce90 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ff9500a1a00 tx=0x7ff94001c2d0 comp rx=0 tx=0).stop 2024-04-26T04:26:56.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff9572cc640 1 -- 172.21.15.63:0/1528782773 shutdown_connections 2024-04-26T04:26:56.946 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff9572cc640 1 --2- 172.21.15.63:0/1528782773 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff95009ca80 0x7ff95009ce90 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:56.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff9572cc640 1 --2- 172.21.15.63:0/1528782773 >> v2:172.21.15.63:3300/0 conn(0x7ff95009c6a0 0x7ff9500996b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:56.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff9572cc640 1 -- 172.21.15.63:0/1528782773 >> 172.21.15.63:0/1528782773 conn(0x7ff9500a1e70 msgr2=0x7ff9500a2280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:26:56.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff9572cc640 1 -- 172.21.15.63:0/1528782773 shutdown_connections 2024-04-26T04:26:56.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff9572cc640 1 -- 172.21.15.63:0/1528782773 wait complete. 2024-04-26T04:26:56.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff9572cc640 10 monclient: build_initial_monmap 2024-04-26T04:26:56.947 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff9572cc640 10 monclient: monmap: 2024-04-26T04:26:56.947 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:26:56.947 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:26:56.947 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:26:56.947 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:26:56.947 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:26:56.947 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:56.947 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:26:56.947 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:26:56.947 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:26:56.948 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:56.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff9572cc640 1 Processor -- start 2024-04-26T04:26:56.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff9572cc640 1 -- start start 2024-04-26T04:26:56.948 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff9572cc640 10 monclient: init 2024-04-26T04:26:56.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff9572cc640 10 monclient: _reopen_session rank -1 2024-04-26T04:26:56.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff9572cc640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T04:26:56.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff9572cc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff950098090 0x7ff9501ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:56.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff9572cc640 10 monclient(hunting): picked mon.noname-a con 0x7ff950098090 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:26:56.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff9572cc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff95009c6a0 0x7ff950097040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:56.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff9572cc640 10 monclient(hunting): picked mon.noname-b con 0x7ff95009c6a0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:26:56.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff9572cc640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff95009ca80 0x7ff950097580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:56.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff9572cc640 10 monclient(hunting): picked mon.noname-c con 0x7ff95009ca80 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:26:56.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff9572cc640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:56.952 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff9572cc640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff9500a1820 con 0x7ff950098090 2024-04-26T04:26:56.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff9572cc640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:56.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff9572cc640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff9500a16a0 con 0x7ff95009c6a0 2024-04-26T04:26:56.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff9572cc640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:56.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff9572cc640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7ff95013fa90 con 0x7ff95009ca80 2024-04-26T04:26:56.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff9572cc640 10 monclient(hunting): _renew_subs 2024-04-26T04:26:56.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff9572cc640 10 monclient(hunting): authenticate will time out at 1448.650955s 2024-04-26T04:26:56.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff955068640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff950098090 0x7ff9501ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:56.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff955068640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff950098090 0x7ff9501ccf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:55368/0 (socket says 172.21.15.63:55368) 2024-04-26T04:26:56.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff955068640 1 -- 172.21.15.63:0/531868784 learned_addr learned my addr 172.21.15.63:0/531868784 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:26:56.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff955068640 10 monclient(hunting): get_auth_request con 0x7ff950098090 auth_method 0 2024-04-26T04:26:56.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff955068640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:56.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff955068640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:56.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff955068640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:56.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff955068640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:26:56.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff955068640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:26:56.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff955068640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:26:56.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff955869640 1 --2- 172.21.15.63:0/531868784 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff95009ca80 0x7ff950097580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:56.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff955869640 10 monclient(hunting): get_auth_request con 0x7ff95009ca80 auth_method 0 2024-04-26T04:26:56.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff955869640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:56.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff955869640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:56.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff955869640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:56.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff955869640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:26:56.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff955869640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:26:56.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff955869640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:26:56.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff954867640 1 --2- 172.21.15.63:0/531868784 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff95009c6a0 0x7ff950097040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:56.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff955068640 10 monclient(hunting): handle_auth_done global_id 44154 payload 824 2024-04-26T04:26:56.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff955068640 1 -- 172.21.15.63:0/531868784 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff95009ca80 msgr2=0x7ff950097580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:56.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff955068640 1 --2- 172.21.15.63:0/531868784 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff95009ca80 0x7ff950097580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:56.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff955068640 1 -- 172.21.15.63:0/531868784 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff95009c6a0 msgr2=0x7ff950097040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:56.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff955068640 1 --2- 172.21.15.63:0/531868784 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff95009c6a0 0x7ff950097040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:56.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff955068640 10 monclient: _finish_hunting 0 2024-04-26T04:26:56.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff955068640 1 monclient: found mon.noname-a 2024-04-26T04:26:56.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff955068640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:26:56.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff955068640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:26:56.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff955068640 1 -- 172.21.15.63:0/531868784 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff95009bb00 con 0x7ff950098090 2024-04-26T04:26:56.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff954867640 10 monclient: get_auth_request con 0x7ff95009c6a0 auth_method 0 2024-04-26T04:26:56.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.937+0000 7ff954867640 1 --2- 172.21.15.63:0/531868784 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff95009c6a0 0x7ff950097040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T04:26:56.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.941+0000 7ff955068640 1 -- 172.21.15.63:0/531868784 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff950098090 msgr2=0x7ff9501ccf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:26:56.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.941+0000 7ff955068640 1 --2- 172.21.15.63:0/531868784 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff950098090 0x7ff9501ccf30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff94c00ebe0 tx=0x7ff94c00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:26:56.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.941+0000 7ff945ffb640 1 -- 172.21.15.63:0/531868784 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff94c017070 con 0x7ff950098090 2024-04-26T04:26:56.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.941+0000 7ff945ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:56.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.941+0000 7ff945ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:26:56.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.941+0000 7ff945ffb640 10 monclient: dump: 2024-04-26T04:26:56.956 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:56.956 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:56.956 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:56.956 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:56.956 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:56.956 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:56.956 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:56.956 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:56.956 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:56.956 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:56.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.941+0000 7ff945ffb640 10 monclient: _finish_auth 0 2024-04-26T04:26:56.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.941+0000 7ff945ffb640 10 monclient: _check_auth_tickets 2024-04-26T04:26:56.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.941+0000 7ff945ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:26:56.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.941+0000 7ff945ffb640 1 -- 172.21.15.63:0/531868784 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff94c00ce40 con 0x7ff950098090 2024-04-26T04:26:56.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.941+0000 7ff9572cc640 5 monclient: authenticate success, global_id 44154 2024-04-26T04:26:56.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.941+0000 7ff9572cc640 10 monclient: _renew_subs 2024-04-26T04:26:56.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.941+0000 7ff9572cc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:26:56.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.941+0000 7ff9572cc640 1 -- 172.21.15.63:0/531868784 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff95008f270 con 0x7ff950098090 2024-04-26T04:26:56.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.941+0000 7ff945ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T04:26:56.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.941+0000 7ff945ffb640 1 -- 172.21.15.63:0/531868784 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff94c013690 con 0x7ff950098090 2024-04-26T04:26:56.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.941+0000 7ff945ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:56.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.941+0000 7ff945ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:26:56.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.941+0000 7ff945ffb640 10 monclient: dump: 2024-04-26T04:26:56.957 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:56.957 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:56.958 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:56.958 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:56.958 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:56.958 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:56.958 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:56.958 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:56.958 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:56.958 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:56.958 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.941+0000 7ff9477fe640 4 set_mon_vals no callback set 2024-04-26T04:26:56.958 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.941+0000 7ff9572cc640 10 monclient: _renew_subs 2024-04-26T04:26:56.958 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.941+0000 7ff9572cc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:26:56.958 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.941+0000 7ff9572cc640 1 -- 172.21.15.63:0/531868784 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff95008e540 con 0x7ff950098090 2024-04-26T04:26:56.958 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.941+0000 7ff9572cc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:26:56.958 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.941+0000 7ff9572cc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:26:56.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.941+0000 7ff9572cc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:26:56.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.941+0000 7ff9572cc640 1 -- 172.21.15.63:0/531868784 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff950062e10 con 0x7ff950098090 2024-04-26T04:26:56.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.941+0000 7ff945ffb640 1 -- 172.21.15.63:0/531868784 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7ff94c01a070 con 0x7ff950098090 2024-04-26T04:26:56.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.941+0000 7ff945ffb640 1 --2- 172.21.15.63:0/531868784 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7ff934035ec0 0x7ff934038380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:56.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.941+0000 7ff945ffb640 1 -- 172.21.15.63:0/531868784 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(323..323 src has 1..323) ==== 4346+0+0 (secure 0 0 0) 0x7ff94c048cf0 con 0x7ff950098090 2024-04-26T04:26:56.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.941+0000 7ff954867640 1 --2- 172.21.15.63:0/531868784 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7ff934035ec0 0x7ff934038380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:56.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.941+0000 7ff954867640 10 monclient: get_auth_request con 0x7ff934035ec0 auth_method 0 2024-04-26T04:26:56.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.945+0000 7ff954867640 1 --2- 172.21.15.63:0/531868784 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7ff934035ec0 0x7ff934038380 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7ff9400ab000 tx=0x7ff9400a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:26:56.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.945+0000 7ff945ffb640 1 -- 172.21.15.63:0/531868784 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7ff94c04e020 con 0x7ff950098090 2024-04-26T04:26:56.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.945+0000 7ff945ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:26:56.960 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:56.945+0000 7ff945ffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:26:57.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:57.105+0000 7ff9572cc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-04-26T04:26:57.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:57.105+0000 7ff9572cc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-26T04:26:57.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:57.105+0000 7ff9572cc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:26:57.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:57.105+0000 7ff9572cc640 1 -- 172.21.15.63:0/531868784 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff950097ac0 con 0x7ff950098090 2024-04-26T04:26:57.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:57.105+0000 7ff945ffb640 1 -- 172.21.15.63:0/531868784 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v323) ==== 74+0+12256 (secure 0 0 0) 0x7ff95008f270 con 0x7ff950098090 2024-04-26T04:26:57.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:57.105+0000 7ff945ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-04-26T04:26:57.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:57.105+0000 7ff945ffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:26:57.113 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-26T04:26:57.113 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":323,"fsid":"cd1c6a0c-4e4b-4014-89c0-2e535479ef39","created":"2024-04-26T04:18:34.126231+0000","modified":"2024-04-26T04:26:37.269795+0000","last_up_change":"2024-04-26T04:18:44.470061+0000","last_in_change":"2024-04-26T04:18:36.862750+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":100,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-26T04:18:43.521296+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-26T04:18:45.211226+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2d37141b-8282-4c62-864a-84a75ebf8f6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":320,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":2758152700},{"type":"v1","addr":"172.21.15.63:6801","nonce":2758152700}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":2758152700},{"type":"v1","addr":"172.21.15.63:6803","nonce":2758152700}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":2758152700},{"type":"v1","addr":"172.21.15.63:6807","nonce":2758152700}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":2758152700},{"type":"v1","addr":"172.21.15.63:6805","nonce":2758152700}]},"public_addr":"172.21.15.63:6801/2758152700","cluster_addr":"172.21.15.63:6803/2758152700","heartbeat_back_addr":"172.21.15.63:6807/2758152700","heartbeat_front_addr":"172.21.15.63:6805/2758152700","state":["exists","up"]},{"osd":1,"uuid":"7220e45a-72bb-4d8e-93c2-32005fa2c85e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":320,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":3596874741},{"type":"v1","addr":"172.21.15.63:6817","nonce":3596874741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":3596874741},{"type":"v1","addr":"172.21.15.63:6819","nonce":3596874741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":3596874741},{"type":"v1","addr":"172.21.15.63:6823","nonce":3596874741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":3596874741},{"type":"v1","addr":"172.21.15.63:6821","nonce":3596874741}]},"public_addr":"172.21.15.63:6817/3596874741","cluster_addr":"172.21.15.63:6819/3596874741","heartbeat_back_addr":"172.21.15.63:6823/3596874741","heartbeat_front_addr":"172.21.15.63:6821/3596874741","state":["exists","up"]},{"osd":2,"uuid":"913de4d8-31af-4394-a0ee-194cc31f822f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":320,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":413829082},{"type":"v1","addr":"172.21.15.63:6809","nonce":413829082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":413829082},{"type":"v1","addr":"172.21.15.63:6811","nonce":413829082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":413829082},{"type":"v1","addr":"172.21.15.63:6815","nonce":413829082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":413829082},{"type":"v1","addr":"172.21.15.63:6813","nonce":413829082}]},"public_addr":"172.21.15.63:6809/413829082","cluster_addr":"172.21.15.63:6811/413829082","heartbeat_back_addr":"172.21.15.63:6815/413829082","heartbeat_front_addr":"172.21.15.63:6813/413829082","state":["exists","up"]},{"osd":3,"uuid":"c492772c-ccf5-4b85-9056-6a5ee9686c0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":320,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6807","nonce":4227496981},{"type":"v1","addr":"172.21.15.186:6808","nonce":4227496981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4227496981},{"type":"v1","addr":"172.21.15.186:6811","nonce":4227496981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4227496981},{"type":"v1","addr":"172.21.15.186:6815","nonce":4227496981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4227496981},{"type":"v1","addr":"172.21.15.186:6813","nonce":4227496981}]},"public_addr":"172.21.15.186:6808/4227496981","cluster_addr":"172.21.15.186:6811/4227496981","heartbeat_back_addr":"172.21.15.186:6815/4227496981","heartbeat_front_addr":"172.21.15.186:6813/4227496981","state":["exists","up"]},{"osd":4,"uuid":"24f067f0-554a-4dd2-a26c-7bda3d0978f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":320,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":3746743567},{"type":"v1","addr":"172.21.15.186:6801","nonce":3746743567}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":3746743567},{"type":"v1","addr":"172.21.15.186:6803","nonce":3746743567}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":3746743567},{"type":"v1","addr":"172.21.15.186:6809","nonce":3746743567}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":3746743567},{"type":"v1","addr":"172.21.15.186:6805","nonce":3746743567}]},"public_addr":"172.21.15.186:6801/3746743567","cluster_addr":"172.21.15.186:6803/3746743567","heartbeat_back_addr":"172.21.15.186:6809/3746743567","heartbeat_front_addr":"172.21.15.186:6805/3746743567","state":["exists","up"]},{"osd":5,"uuid":"70454d09-e50c-4f65-83c5-0f9bb5c0f8f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":320,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":3109931567},{"type":"v1","addr":"172.21.15.186:6817","nonce":3109931567}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":3109931567},{"type":"v1","addr":"172.21.15.186:6819","nonce":3109931567}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":3109931567},{"type":"v1","addr":"172.21.15.186:6823","nonce":3109931567}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":3109931567},{"type":"v1","addr":"172.21.15.186:6821","nonce":3109931567}]},"public_addr":"172.21.15.186:6817/3109931567","cluster_addr":"172.21.15.186:6819/3109931567","heartbeat_back_addr":"172.21.15.186:6823/3109931567","heartbeat_front_addr":"172.21.15.186:6821/3109931567","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T04:18:42.278922+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T04:18:42.387279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T04:18:42.394303+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T04:18:42.367922+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T04:18:42.357914+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-26T04:18:42.415215+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-26T04:26:57.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:57.109+0000 7ff9572cc640 1 -- 172.21.15.63:0/531868784 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7ff934035ec0 msgr2=0x7ff934038380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:57.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:57.109+0000 7ff9572cc640 1 --2- 172.21.15.63:0/531868784 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7ff934035ec0 0x7ff934038380 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7ff9400ab000 tx=0x7ff9400a6000 comp rx=0 tx=0).stop 2024-04-26T04:26:57.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:57.109+0000 7ff9572cc640 10 monclient: shutdown 2024-04-26T04:26:57.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:57.109+0000 7ff9572cc640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:26:57.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:57.109+0000 7ff9572cc640 1 -- 172.21.15.63:0/531868784 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff950098090 msgr2=0x7ff9501ccf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:57.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:57.109+0000 7ff9572cc640 1 --2- 172.21.15.63:0/531868784 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff950098090 0x7ff9501ccf30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff94c00ebe0 tx=0x7ff94c00c520 comp rx=0 tx=0).stop 2024-04-26T04:26:57.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:57.109+0000 7ff9572cc640 1 -- 172.21.15.63:0/531868784 shutdown_connections 2024-04-26T04:26:57.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:57.109+0000 7ff9572cc640 1 --2- 172.21.15.63:0/531868784 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7ff934035ec0 0x7ff934038380 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:57.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:57.109+0000 7ff9572cc640 1 --2- 172.21.15.63:0/531868784 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff95009ca80 0x7ff950097580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:57.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:57.109+0000 7ff9572cc640 1 --2- 172.21.15.63:0/531868784 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff95009c6a0 0x7ff950097040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:57.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:57.109+0000 7ff9572cc640 1 --2- 172.21.15.63:0/531868784 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff950098090 0x7ff9501ccf30 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:57.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:57.109+0000 7ff9572cc640 1 -- 172.21.15.63:0/531868784 >> 172.21.15.63:0/531868784 conn(0x7ff9500a1e70 msgr2=0x7ff9500a3e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:26:57.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:57.109+0000 7ff9572cc640 1 -- 172.21.15.63:0/531868784 shutdown_connections 2024-04-26T04:26:57.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:57.109+0000 7ff9572cc640 1 -- 172.21.15.63:0/531868784 wait complete. 2024-04-26T04:26:58.133 INFO:tasks.ceph:Scrubbing osd.0 2024-04-26T04:26:58.134 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-04-26T04:26:58.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.269+0000 7fadda106640 10 monclient: get_monmap_and_config 2024-04-26T04:26:58.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.269+0000 7fadda106640 10 monclient: build_initial_monmap 2024-04-26T04:26:58.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.269+0000 7fadda106640 10 monclient: monmap: 2024-04-26T04:26:58.271 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:26:58.271 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:58.271 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:26:58.271309+0000 2024-04-26T04:26:58.271 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:26:58.271309+0000 2024-04-26T04:26:58.271 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:26:58.271 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:58.272 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:26:58.272 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:26:58.272 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:26:58.272 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:26:58.272 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:26:58.272 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:58.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.269+0000 7fadda106640 1 Processor -- start 2024-04-26T04:26:58.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.269+0000 7fadda106640 1 -- start start 2024-04-26T04:26:58.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.269+0000 7fadda106640 10 monclient: init 2024-04-26T04:26:58.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.269+0000 7fadda106640 10 monclient: _reopen_session rank -1 2024-04-26T04:26:58.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.269+0000 7fadda106640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-04-26T04:26:58.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.269+0000 7fadda106640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fadd409c6a0 0x7fadd409ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:58.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.269+0000 7fadda106640 10 monclient(hunting): picked mon.noname-c con 0x7fadd409c6a0 addr v2:172.21.15.186:3300/0 2024-04-26T04:26:58.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.269+0000 7fadda106640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fadd409e0a0 addr v1:172.21.15.63:6789/0 2024-04-26T04:26:58.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.269+0000 7fadda106640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fadd409cfc0 addr v1:172.21.15.186:6789/0 2024-04-26T04:26:58.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.269+0000 7fadda106640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fadd4098eb0 con 0x7fadd409e0a0 2024-04-26T04:26:58.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.269+0000 7fadda106640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fadd409d7d0 con 0x7fadd409cfc0 2024-04-26T04:26:58.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.269+0000 7fadda106640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:58.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.269+0000 7fadda106640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fadd4099b40 con 0x7fadd409c6a0 2024-04-26T04:26:58.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.269+0000 7fadda106640 10 monclient(hunting): _renew_subs 2024-04-26T04:26:58.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.269+0000 7fadda106640 10 monclient(hunting): authenticate will time out at 1449.979112s 2024-04-26T04:26:58.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.269+0000 7fadd2ffd640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7fadd409e0a0 0x7fadd4098090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:36792/0 (socket says 172.21.15.63:36792) 2024-04-26T04:26:58.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.269+0000 7fadd2ffd640 1 -- 172.21.15.63:0/2162273291 learned_addr learned my addr 172.21.15.63:0/2162273291 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-04-26T04:26:58.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.269+0000 7fadd37fe640 1 --2- 172.21.15.63:0/2162273291 >> v2:172.21.15.186:3300/0 conn(0x7fadd409c6a0 0x7fadd409ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:58.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.269+0000 7fadd37fe640 10 monclient(hunting): get_auth_request con 0x7fadd409c6a0 auth_method 0 2024-04-26T04:26:58.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.269+0000 7fadd37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:58.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.269+0000 7fadd37fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:58.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.269+0000 7fadd37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:58.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.269+0000 7fadd37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:26:58.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.269+0000 7fadd37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:26:58.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.269+0000 7fadd37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:26:58.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd2ffd640 1 -- 172.21.15.63:0/2162273291 >> v1:172.21.15.63:6789/0 conn(0x7fadd409e0a0 legacy=0x7fadd4098090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:26:58.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd27fc640 1 -- 172.21.15.63:0/2162273291 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2993068592 0 0) 0x7fadd4098eb0 con 0x7fadd409e0a0 2024-04-26T04:26:58.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd27fc640 10 monclient(hunting): _negotiate 2024-04-26T04:26:58.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd27fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:58.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd27fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:58.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd27fc640 10 monclient(hunting): my global_id is 44157 2024-04-26T04:26:58.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd27fc640 1 -- 172.21.15.63:0/2162273291 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fadc0003450 con 0x7fadd409e0a0 2024-04-26T04:26:58.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd3fff640 1 -- 172.21.15.63:0/2162273291 >> v1:172.21.15.186:6789/0 conn(0x7fadd409cfc0 legacy=0x7fadd40987a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:26:58.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd27fc640 1 -- 172.21.15.63:0/2162273291 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2064133018 0 0) 0x7fadc0003450 con 0x7fadd409e0a0 2024-04-26T04:26:58.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd27fc640 1 -- 172.21.15.63:0/2162273291 >> v2:172.21.15.186:3300/0 conn(0x7fadd409c6a0 msgr2=0x7fadd409ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:26:58.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd27fc640 1 --2- 172.21.15.63:0/2162273291 >> v2:172.21.15.186:3300/0 conn(0x7fadd409c6a0 0x7fadd409ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:58.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd27fc640 1 -- 172.21.15.63:0/2162273291 >> v1:172.21.15.186:6789/0 conn(0x7fadd409cfc0 legacy=0x7fadd40987a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:58.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd27fc640 10 monclient: _finish_hunting 0 2024-04-26T04:26:58.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd27fc640 1 monclient: found mon.noname-a-legacy 2024-04-26T04:26:58.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:26:58.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd27fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.63:6789/0 2024-04-26T04:26:58.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd27fc640 1 -- 172.21.15.63:0/2162273291 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadd409f210 con 0x7fadd409e0a0 2024-04-26T04:26:58.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd27fc640 10 monclient: _finish_auth 0 2024-04-26T04:26:58.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd27fc640 10 monclient: _check_auth_tickets 2024-04-26T04:26:58.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:26:58.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd27fc640 1 -- 172.21.15.63:0/2162273291 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2612289768 0 0) 0x7fadbc001930 con 0x7fadd409e0a0 2024-04-26T04:26:58.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadda106640 5 monclient: authenticate success, global_id 44157 2024-04-26T04:26:58.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadda106640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T04:26:58.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:58.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd27fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-26T04:26:58.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd27fc640 10 monclient: dump: 2024-04-26T04:26:58.280 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:58.280 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:58.280 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:58.281 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:58.281 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:58.281 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:58.281 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:58.281 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:58.281 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:58.281 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:58.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd27fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] but i'm connected to v1:172.21.15.63:6789/0, reconnecting 2024-04-26T04:26:58.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd27fc640 10 monclient: _reopen_session rank -1 2024-04-26T04:26:58.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd27fc640 1 -- 172.21.15.63:0/2162273291 >> v1:172.21.15.63:6789/0 conn(0x7fadd409e0a0 legacy=0x7fadd4098090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:58.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd27fc640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T04:26:58.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd27fc640 1 --2- 172.21.15.63:0/2162273291 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fadc0003be0 0x7fadc0004000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:58.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd27fc640 10 monclient(hunting): picked mon.a con 0x7fadc0003be0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:26:58.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd27fc640 1 --2- 172.21.15.63:0/2162273291 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fadc0008a10 0x7fadc000aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:58.281 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd27fc640 10 monclient(hunting): picked mon.b con 0x7fadc0008a10 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:26:58.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd27fc640 1 --2- 172.21.15.63:0/2162273291 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fadc000b740 0x7fadc000dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:58.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd27fc640 10 monclient(hunting): picked mon.c con 0x7fadc000b740 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:26:58.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd27fc640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:58.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd27fc640 1 -- 172.21.15.63:0/2162273291 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fadc000e4b0 con 0x7fadc0003be0 2024-04-26T04:26:58.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd27fc640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:58.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd27fc640 1 -- 172.21.15.63:0/2162273291 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fadc000e660 con 0x7fadc0008a10 2024-04-26T04:26:58.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd27fc640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:58.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd27fc640 1 -- 172.21.15.63:0/2162273291 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7fadc000e860 con 0x7fadc000b740 2024-04-26T04:26:58.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd27fc640 10 monclient(hunting): _renew_subs 2024-04-26T04:26:58.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd27fc640 10 monclient(hunting): _finish_auth 0 2024-04-26T04:26:58.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadda106640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-04-26T04:26:58.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd37fe640 1 --2- 172.21.15.63:0/2162273291 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fadc0003be0 0x7fadc0004000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:58.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd37fe640 10 monclient(hunting): get_auth_request con 0x7fadc0003be0 auth_method 0 2024-04-26T04:26:58.282 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:58.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd37fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:58.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:26:58.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd3fff640 1 --2- 172.21.15.63:0/2162273291 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fadc000b740 0x7fadc000dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:58.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd3fff640 10 monclient(hunting): get_auth_request con 0x7fadc000b740 auth_method 0 2024-04-26T04:26:58.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:58.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd3fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:58.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:26:58.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:26:58.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:26:58.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd37fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T04:26:58.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:26:58.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:26:58.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd2ffd640 1 --2- 172.21.15.63:0/2162273291 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fadc0008a10 0x7fadc000aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:58.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd3fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T04:26:58.283 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd2ffd640 10 monclient(hunting): get_auth_request con 0x7fadc0008a10 auth_method 0 2024-04-26T04:26:58.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:58.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:58.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd2ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:26:58.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd37fe640 10 monclient(hunting): handle_auth_done global_id 44157 payload 293 2024-04-26T04:26:58.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd37fe640 1 -- 172.21.15.63:0/2162273291 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fadc000b740 msgr2=0x7fadc000dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:26:58.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd37fe640 1 --2- 172.21.15.63:0/2162273291 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fadc000b740 0x7fadc000dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:58.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd37fe640 1 -- 172.21.15.63:0/2162273291 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fadc0008a10 msgr2=0x7fadc000aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:26:58.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd37fe640 1 --2- 172.21.15.63:0/2162273291 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fadc0008a10 0x7fadc000aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:58.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd37fe640 10 monclient: _finish_hunting 0 2024-04-26T04:26:58.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd37fe640 1 monclient: found mon.a 2024-04-26T04:26:58.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:26:58.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd37fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:26:58.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd37fe640 1 -- 172.21.15.63:0/2162273291 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadc0003450 con 0x7fadc0003be0 2024-04-26T04:26:58.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd37fe640 10 monclient: _finish_auth 0 2024-04-26T04:26:58.284 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd37fe640 10 monclient: _check_auth_tickets 2024-04-26T04:26:58.285 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:26:58.285 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.273+0000 7fadd3fff640 1 -- 172.21.15.63:0/2162273291 reap_dead start 2024-04-26T04:26:58.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.277+0000 7fadd37fe640 1 -- 172.21.15.63:0/2162273291 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fadc0003be0 msgr2=0x7fadc0004000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:26:58.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.277+0000 7fadd37fe640 1 --2- 172.21.15.63:0/2162273291 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fadc0003be0 0x7fadc0004000 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fadc4009ae0 tx=0x7fadc401ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=652d88ae0c1c3c53 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:26:58.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.277+0000 7fadd27fc640 1 -- 172.21.15.63:0/2162273291 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc40096b0 con 0x7fadc0003be0 2024-04-26T04:26:58.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.277+0000 7fadd27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:58.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.277+0000 7fadd27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:26:58.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.277+0000 7fadd27fc640 10 monclient: dump: 2024-04-26T04:26:58.290 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:58.290 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:58.290 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:58.290 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:58.290 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:58.290 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:58.290 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:58.290 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:58.290 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:58.290 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:58.290 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.277+0000 7fadd27fc640 1 -- 172.21.15.63:0/2162273291 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fadc40a8410 con 0x7fadc0003be0 2024-04-26T04:26:58.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.277+0000 7fadda106640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T04:26:58.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.277+0000 7fadd27fc640 10 monclient: handle_config config(0 keys) 2024-04-26T04:26:58.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.277+0000 7fadda106640 10 monclient: get_monmap_and_config success 2024-04-26T04:26:58.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.277+0000 7fadd27fc640 1 -- 172.21.15.63:0/2162273291 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc40a8800 con 0x7fadc0003be0 2024-04-26T04:26:58.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.277+0000 7fadda106640 4 set_mon_vals no callback set 2024-04-26T04:26:58.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.277+0000 7fadd27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:58.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.277+0000 7fadd27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:26:58.291 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.277+0000 7fadd27fc640 10 monclient: dump: 2024-04-26T04:26:58.291 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:58.291 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:58.291 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:58.291 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:58.291 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:58.291 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:58.291 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:58.292 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:58.292 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:58.292 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:58.292 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.277+0000 7fadda106640 10 monclient: shutdown 2024-04-26T04:26:58.292 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.277+0000 7fadda106640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:26:58.292 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.277+0000 7fadda106640 1 -- 172.21.15.63:0/2162273291 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fadc0003be0 msgr2=0x7fadc0004000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:58.292 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.277+0000 7fadda106640 1 --2- 172.21.15.63:0/2162273291 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fadc0003be0 0x7fadc0004000 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fadc4009ae0 tx=0x7fadc401ca50 comp rx=0 tx=0).stop 2024-04-26T04:26:58.292 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.277+0000 7fadda106640 1 -- 172.21.15.63:0/2162273291 shutdown_connections 2024-04-26T04:26:58.292 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.277+0000 7fadda106640 1 --2- 172.21.15.63:0/2162273291 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fadc0003be0 0x7fadc0004000 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:58.292 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.277+0000 7fadda106640 1 -- 172.21.15.63:0/2162273291 >> 172.21.15.63:0/2162273291 conn(0x7fadd40a1ca0 msgr2=0x7fadd40a20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:26:58.292 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.277+0000 7fadda106640 1 -- 172.21.15.63:0/2162273291 shutdown_connections 2024-04-26T04:26:58.292 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.277+0000 7fadda106640 1 -- 172.21.15.63:0/2162273291 wait complete. 2024-04-26T04:26:58.292 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.281+0000 7fadda106640 10 monclient: build_initial_monmap 2024-04-26T04:26:58.292 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.281+0000 7fadda106640 10 monclient: monmap: 2024-04-26T04:26:58.292 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:26:58.293 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:26:58.293 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:26:58.293 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:26:58.293 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:26:58.293 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:58.293 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:26:58.293 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:26:58.293 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:26:58.293 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:58.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.281+0000 7fadda106640 1 Processor -- start 2024-04-26T04:26:58.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.281+0000 7fadda106640 1 -- start start 2024-04-26T04:26:58.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.281+0000 7fadda106640 10 monclient: init 2024-04-26T04:26:58.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.281+0000 7fadda106640 10 monclient: _reopen_session rank -1 2024-04-26T04:26:58.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.281+0000 7fadda106640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T04:26:58.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.281+0000 7fadda106640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fadc0003be0 0x7fadd41cf590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:58.293 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.281+0000 7fadda106640 10 monclient(hunting): picked mon.noname-c con 0x7fadc0003be0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:26:58.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.281+0000 7fadda106640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fadd41cfad0 0x7fadd41c0970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:58.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.281+0000 7fadda106640 10 monclient(hunting): picked mon.noname-b con 0x7fadd41cfad0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:26:58.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.281+0000 7fadda106640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fadd41cff10 0x7fadd41c0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:58.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.281+0000 7fadda106640 10 monclient(hunting): picked mon.noname-a con 0x7fadd41cff10 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:26:58.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.281+0000 7fadda106640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:58.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.281+0000 7fadda106640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fadd4098210 con 0x7fadd41cff10 2024-04-26T04:26:58.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.281+0000 7fadda106640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:58.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.281+0000 7fadda106640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fadd4098090 con 0x7fadd41cfad0 2024-04-26T04:26:58.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.281+0000 7fadda106640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:58.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.281+0000 7fadda106640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7fadd4098390 con 0x7fadc0003be0 2024-04-26T04:26:58.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.281+0000 7fadda106640 10 monclient(hunting): _renew_subs 2024-04-26T04:26:58.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.281+0000 7fadda106640 10 monclient(hunting): authenticate will time out at 1449.989429s 2024-04-26T04:26:58.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.281+0000 7fadd37fe640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fadc0003be0 0x7fadd41cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:58.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.281+0000 7fadd3fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fadd41cff10 0x7fadd41c0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:58.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.281+0000 7fadd37fe640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fadc0003be0 0x7fadd41cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:41132/0 (socket says 172.21.15.63:41132) 2024-04-26T04:26:58.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.281+0000 7fadd37fe640 1 -- 172.21.15.63:0/924561696 learned_addr learned my addr 172.21.15.63:0/924561696 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:26:58.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.281+0000 7fadd37fe640 10 monclient(hunting): get_auth_request con 0x7fadc0003be0 auth_method 0 2024-04-26T04:26:58.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.281+0000 7fadd37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:58.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.281+0000 7fadd37fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:58.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.281+0000 7fadd37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:58.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.281+0000 7fadd3fff640 10 monclient(hunting): get_auth_request con 0x7fadd41cff10 auth_method 0 2024-04-26T04:26:58.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.281+0000 7fadd3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:58.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.281+0000 7fadd3fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:58.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.281+0000 7fadd3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:58.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.281+0000 7fadd37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:26:58.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.281+0000 7fadd37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:26:58.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.281+0000 7fadd37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:26:58.295 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.281+0000 7fadd3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:26:58.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.281+0000 7fadd3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:26:58.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.281+0000 7fadd3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:26:58.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.281+0000 7fadd2ffd640 1 --2- 172.21.15.63:0/924561696 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fadd41cfad0 0x7fadd41c0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:58.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.281+0000 7fadd2ffd640 10 monclient(hunting): get_auth_request con 0x7fadd41cfad0 auth_method 0 2024-04-26T04:26:58.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.281+0000 7fadd2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:58.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.281+0000 7fadd2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:58.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.281+0000 7fadd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:58.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.281+0000 7fadd37fe640 10 monclient(hunting): handle_auth_done global_id 24719 payload 824 2024-04-26T04:26:58.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.281+0000 7fadd37fe640 1 -- 172.21.15.63:0/924561696 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fadd41cfad0 msgr2=0x7fadd41c0970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:58.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.281+0000 7fadd37fe640 1 --2- 172.21.15.63:0/924561696 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fadd41cfad0 0x7fadd41c0970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:58.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.281+0000 7fadd37fe640 1 -- 172.21.15.63:0/924561696 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fadd41cff10 msgr2=0x7fadd41c0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:58.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.281+0000 7fadd37fe640 1 --2- 172.21.15.63:0/924561696 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fadd41cff10 0x7fadd41c0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:58.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.281+0000 7fadd37fe640 10 monclient: _finish_hunting 0 2024-04-26T04:26:58.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.281+0000 7fadd37fe640 1 monclient: found mon.noname-c 2024-04-26T04:26:58.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.281+0000 7fadd37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:26:58.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.281+0000 7fadd37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.63:3301/0 2024-04-26T04:26:58.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.281+0000 7fadd37fe640 1 -- 172.21.15.63:0/924561696 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadd408dbf0 con 0x7fadc0003be0 2024-04-26T04:26:58.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.281+0000 7fadd3fff640 1 --2- 172.21.15.63:0/924561696 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fadd41cff10 0x7fadd41c0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T04:26:58.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.285+0000 7fadd37fe640 1 -- 172.21.15.63:0/924561696 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fadc0003be0 msgr2=0x7fadd41cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:26:58.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.285+0000 7fadd37fe640 1 --2- 172.21.15.63:0/924561696 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fadc0003be0 0x7fadd41cf590 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fadc401c880 tx=0x7fadc40197f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:26:58.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.285+0000 7fadcbfff640 1 -- 172.21.15.63:0/924561696 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc40bf070 con 0x7fadc0003be0 2024-04-26T04:26:58.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.285+0000 7fadcbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:58.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.285+0000 7fadcbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:26:58.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.285+0000 7fadcbfff640 10 monclient: dump: 2024-04-26T04:26:58.297 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:58.297 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:58.297 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:58.297 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:58.297 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:58.297 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:58.297 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:58.298 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:58.298 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:58.298 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:58.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.285+0000 7fadcbfff640 10 monclient: _finish_auth 0 2024-04-26T04:26:58.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.285+0000 7fadcbfff640 10 monclient: _check_auth_tickets 2024-04-26T04:26:58.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.285+0000 7fadcbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:26:58.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.285+0000 7fadcbfff640 1 -- 172.21.15.63:0/924561696 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fadc400f000 con 0x7fadc0003be0 2024-04-26T04:26:58.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.285+0000 7fadcbfff640 10 monclient: handle_config config(0 keys) 2024-04-26T04:26:58.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.285+0000 7fadcbfff640 1 -- 172.21.15.63:0/924561696 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc40a8da0 con 0x7fadc0003be0 2024-04-26T04:26:58.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.285+0000 7fadda106640 5 monclient: authenticate success, global_id 24719 2024-04-26T04:26:58.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.285+0000 7fadcbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:58.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.285+0000 7fadcbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:26:58.298 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.285+0000 7fadcbfff640 10 monclient: dump: 2024-04-26T04:26:58.298 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:58.298 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:58.299 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:58.299 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:58.299 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:58.299 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:58.299 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:58.299 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:58.299 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:58.299 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:58.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.285+0000 7fadda106640 10 monclient: _renew_subs 2024-04-26T04:26:58.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.285+0000 7fadda106640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:26:58.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.285+0000 7fadda106640 1 -- 172.21.15.63:0/924561696 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadd408ded0 con 0x7fadc0003be0 2024-04-26T04:26:58.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.285+0000 7fadd1ffb640 4 set_mon_vals no callback set 2024-04-26T04:26:58.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.285+0000 7fadda106640 10 monclient: _renew_subs 2024-04-26T04:26:58.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.285+0000 7fadda106640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:26:58.299 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.285+0000 7fadda106640 1 -- 172.21.15.63:0/924561696 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fadd408c2e0 con 0x7fadc0003be0 2024-04-26T04:26:58.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.285+0000 7fadda106640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:26:58.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.285+0000 7fadda106640 1 -- 172.21.15.63:0/924561696 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fadd4067fb0 con 0x7fadc0003be0 2024-04-26T04:26:58.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.285+0000 7fadcbfff640 1 -- 172.21.15.63:0/924561696 <== mon.2 v2:172.21.15.63:3301/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fadc40b0400 con 0x7fadc0003be0 2024-04-26T04:26:58.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.285+0000 7fadcbfff640 1 --2- 172.21.15.63:0/924561696 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fadb4035ec0 0x7fadb4038380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:58.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.285+0000 7fadcbfff640 1 -- 172.21.15.63:0/924561696 <== mon.2 v2:172.21.15.63:3301/0 5 ==== osd_map(323..323 src has 1..323) ==== 4346+0+0 (secure 0 0 0) 0x7fadc40e2b10 con 0x7fadc0003be0 2024-04-26T04:26:58.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.285+0000 7fadcbfff640 1 --2- 172.21.15.63:0/924561696 >> [v2:172.21.15.63:6800/2758152700,v1:172.21.15.63:6801/2758152700] conn(0x7fadb403bcc0 0x7fadb403e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:58.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.285+0000 7fadcbfff640 1 -- 172.21.15.63:0/924561696 --> [v2:172.21.15.63:6800/2758152700,v1:172.21.15.63:6801/2758152700] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fadc40bcec0 con 0x7fadb403bcc0 2024-04-26T04:26:58.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.285+0000 7fadcbfff640 1 -- 172.21.15.63:0/924561696 <== mon.2 v2:172.21.15.63:3301/0 6 ==== mon_get_version_reply(handle=1 version=323) ==== 24+0+0 (secure 0 0 0) 0x7fadc40e4e60 con 0x7fadc0003be0 2024-04-26T04:26:58.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.285+0000 7fadcbfff640 10 monclient: handle_get_version_reply finishing 1 version 323 2024-04-26T04:26:58.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.285+0000 7fadd3fff640 1 --2- 172.21.15.63:0/924561696 >> [v2:172.21.15.63:6800/2758152700,v1:172.21.15.63:6801/2758152700] conn(0x7fadb403bcc0 0x7fadb403e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:58.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.285+0000 7fadd3fff640 10 monclient: get_auth_request con 0x7fadb403bcc0 auth_method 0 2024-04-26T04:26:58.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.285+0000 7fadd2ffd640 1 --2- 172.21.15.63:0/924561696 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fadb4035ec0 0x7fadb4038380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:58.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.285+0000 7fadd2ffd640 10 monclient: get_auth_request con 0x7fadb4035ec0 auth_method 0 2024-04-26T04:26:58.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.289+0000 7fadd2ffd640 1 --2- 172.21.15.63:0/924561696 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fadb4035ec0 0x7fadb4038380 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fadd41c20f0 tx=0x7fadbc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:26:58.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.289+0000 7fadd3fff640 1 --2- 172.21.15.63:0/924561696 >> [v2:172.21.15.63:6800/2758152700,v1:172.21.15.63:6801/2758152700] conn(0x7fadb403bcc0 0x7fadb403e0e0 crc :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:26:58.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.289+0000 7fadcbfff640 1 -- 172.21.15.63:0/924561696 <== osd.0 v2:172.21.15.63:6800/2758152700 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fadc40bcec0 con 0x7fadb403bcc0 2024-04-26T04:26:58.305 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.305+0000 7fadda106640 1 -- 172.21.15.63:0/924561696 --> [v2:172.21.15.63:6800/2758152700,v1:172.21.15.63:6801/2758152700] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fadd4062e10 con 0x7fadb403bcc0 2024-04-26T04:26:58.306 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.305+0000 7fadcbfff640 1 -- 172.21.15.63:0/924561696 <== osd.0 v2:172.21.15.63:6800/2758152700 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fadd4062e10 con 0x7fadb403bcc0 2024-04-26T04:26:58.306 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-26T04:26:58.306 INFO:teuthology.orchestra.run.smithi063.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-26T04:26:58.306 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-26T04:26:58.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.305+0000 7fadda106640 1 -- 172.21.15.63:0/924561696 >> [v2:172.21.15.63:6800/2758152700,v1:172.21.15.63:6801/2758152700] conn(0x7fadb403bcc0 msgr2=0x7fadb403e0e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:58.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.305+0000 7fadda106640 1 --2- 172.21.15.63:0/924561696 >> [v2:172.21.15.63:6800/2758152700,v1:172.21.15.63:6801/2758152700] conn(0x7fadb403bcc0 0x7fadb403e0e0 crc :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:58.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.305+0000 7fadda106640 1 -- 172.21.15.63:0/924561696 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fadb4035ec0 msgr2=0x7fadb4038380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:58.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.305+0000 7fadda106640 1 --2- 172.21.15.63:0/924561696 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fadb4035ec0 0x7fadb4038380 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fadd41c20f0 tx=0x7fadbc009040 comp rx=0 tx=0).stop 2024-04-26T04:26:58.307 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.305+0000 7fadda106640 10 monclient: shutdown 2024-04-26T04:26:58.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.305+0000 7fadda106640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:26:58.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.305+0000 7fadda106640 1 -- 172.21.15.63:0/924561696 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fadc0003be0 msgr2=0x7fadd41cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:58.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.305+0000 7fadda106640 1 --2- 172.21.15.63:0/924561696 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fadc0003be0 0x7fadd41cf590 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fadc401c880 tx=0x7fadc40197f0 comp rx=0 tx=0).stop 2024-04-26T04:26:58.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.305+0000 7fadd3fff640 1 -- 172.21.15.63:0/924561696 reap_dead start 2024-04-26T04:26:58.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.305+0000 7fadda106640 1 -- 172.21.15.63:0/924561696 shutdown_connections 2024-04-26T04:26:58.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.305+0000 7fadda106640 1 -- 172.21.15.63:0/924561696 >> 172.21.15.63:0/924561696 conn(0x7fadd40a1ca0 msgr2=0x7fadd4097520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:26:58.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.305+0000 7fadda106640 1 -- 172.21.15.63:0/924561696 shutdown_connections 2024-04-26T04:26:58.308 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.305+0000 7fadda106640 1 -- 172.21.15.63:0/924561696 wait complete. 2024-04-26T04:26:58.322 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-04-26T04:26:58.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.433+0000 7f8c1f18f640 10 monclient: get_monmap_and_config 2024-04-26T04:26:58.438 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.437+0000 7f8c1f18f640 10 monclient: build_initial_monmap 2024-04-26T04:26:58.438 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.437+0000 7f8c1f18f640 10 monclient: monmap: 2024-04-26T04:26:58.438 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:26:58.438 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:58.438 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:26:58.437895+0000 2024-04-26T04:26:58.438 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:26:58.437895+0000 2024-04-26T04:26:58.438 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:26:58.438 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:58.438 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:26:58.438 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:26:58.438 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:26:58.438 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:26:58.439 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:26:58.439 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:58.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.437+0000 7f8c1f18f640 1 Processor -- start 2024-04-26T04:26:58.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.437+0000 7f8c1f18f640 1 -- start start 2024-04-26T04:26:58.441 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.437+0000 7f8c1f18f640 10 monclient: init 2024-04-26T04:26:58.441 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.437+0000 7f8c1f18f640 10 monclient: _reopen_session rank -1 2024-04-26T04:26:58.441 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.437+0000 7f8c1f18f640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-04-26T04:26:58.441 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.437+0000 7f8c1f18f640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f8c18072450 0x7f8c18133c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:58.441 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.437+0000 7f8c1f18f640 10 monclient(hunting): picked mon.noname-c con 0x7f8c18072450 addr v2:172.21.15.186:3300/0 2024-04-26T04:26:58.441 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.437+0000 7f8c1f18f640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f8c18072830 0x7f8c180a1430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:58.441 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.437+0000 7f8c1f18f640 10 monclient(hunting): picked mon.noname-a con 0x7f8c18072830 addr v2:172.21.15.63:3300/0 2024-04-26T04:26:58.441 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.437+0000 7f8c1f18f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8c18072cb0 addr v1:172.21.15.186:6789/0 2024-04-26T04:26:58.441 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.437+0000 7f8c1f18f640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c1806e660 con 0x7f8c18072cb0 2024-04-26T04:26:58.441 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.437+0000 7f8c1f18f640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:58.441 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.437+0000 7f8c1f18f640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7f8c1809f650 con 0x7f8c18072830 2024-04-26T04:26:58.441 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.437+0000 7f8c1f18f640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:58.441 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.437+0000 7f8c1f18f640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f8c1806e940 con 0x7f8c18072450 2024-04-26T04:26:58.441 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.437+0000 7f8c1f18f640 10 monclient(hunting): _renew_subs 2024-04-26T04:26:58.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.437+0000 7f8c1f18f640 10 monclient(hunting): authenticate will time out at 1450.145567s 2024-04-26T04:26:58.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.437+0000 7f8c17fff640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f8c18072830 0x7f8c180a1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:58.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.437+0000 7f8c17fff640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f8c18072830 0x7f8c180a1430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:55400/0 (socket says 172.21.15.63:55400) 2024-04-26T04:26:58.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.437+0000 7f8c17fff640 1 -- 172.21.15.63:0/2891099670 learned_addr learned my addr 172.21.15.63:0/2891099670 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:26:58.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.437+0000 7f8c17fff640 10 monclient(hunting): get_auth_request con 0x7f8c18072830 auth_method 0 2024-04-26T04:26:58.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.437+0000 7f8c17fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:58.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.437+0000 7f8c17fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:58.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.437+0000 7f8c17fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:58.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.437+0000 7f8c17fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:26:58.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.437+0000 7f8c17fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:26:58.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.437+0000 7f8c17fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:26:58.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.437+0000 7f8c1cf2b640 1 --2- 172.21.15.63:0/2891099670 >> v2:172.21.15.186:3300/0 conn(0x7f8c18072450 0x7f8c18133c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:58.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.437+0000 7f8c17fff640 10 monclient(hunting): handle_auth_done global_id 44163 payload 274 2024-04-26T04:26:58.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.437+0000 7f8c17fff640 1 -- 172.21.15.63:0/2891099670 >> v2:172.21.15.186:3300/0 conn(0x7f8c18072450 msgr2=0x7f8c18133c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:26:58.442 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.437+0000 7f8c17fff640 1 --2- 172.21.15.63:0/2891099670 >> v2:172.21.15.186:3300/0 conn(0x7f8c18072450 0x7f8c18133c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:58.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.437+0000 7f8c17fff640 1 -- 172.21.15.63:0/2891099670 >> v1:172.21.15.186:6789/0 conn(0x7f8c18072cb0 legacy=0x7f8c180a1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:26:58.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.437+0000 7f8c17fff640 10 monclient: _finish_hunting 0 2024-04-26T04:26:58.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.437+0000 7f8c17fff640 1 monclient: found mon.noname-a 2024-04-26T04:26:58.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.437+0000 7f8c17fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:26:58.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.437+0000 7f8c17fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:26:58.443 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.437+0000 7f8c17fff640 1 -- 172.21.15.63:0/2891099670 --> v2:172.21.15.63:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c1806ec20 con 0x7f8c18072830 2024-04-26T04:26:58.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.441+0000 7f8c17fff640 1 -- 172.21.15.63:0/2891099670 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8c18072830 msgr2=0x7f8c180a1430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:26:58.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.441+0000 7f8c17fff640 1 --2- 172.21.15.63:0/2891099670 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8c18072830 0x7f8c180a1430 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8c1809f590 tx=0x7f8c0c01c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e25b3f4d9fc8af39 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:26:58.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.441+0000 7f8c177fe640 1 -- 172.21.15.63:0/2891099670 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c0c0b0070 con 0x7f8c18072830 2024-04-26T04:26:58.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.441+0000 7f8c177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:58.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.441+0000 7f8c177fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:26:58.445 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.441+0000 7f8c177fe640 10 monclient: dump: 2024-04-26T04:26:58.445 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:58.446 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:58.446 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:58.446 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:58.446 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:58.446 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:58.446 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:58.446 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:58.446 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:58.446 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:58.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.441+0000 7f8c177fe640 10 monclient: _finish_auth 0 2024-04-26T04:26:58.446 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.441+0000 7f8c177fe640 10 monclient: _check_auth_tickets 2024-04-26T04:26:58.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.441+0000 7f8c177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:26:58.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.441+0000 7f8c177fe640 1 -- 172.21.15.63:0/2891099670 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c0c0ab600 con 0x7f8c18072830 2024-04-26T04:26:58.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.441+0000 7f8c1f18f640 5 monclient: authenticate success, global_id 44163 2024-04-26T04:26:58.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.441+0000 7f8c1f18f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T04:26:58.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.441+0000 7f8c177fe640 10 monclient: handle_config config(0 keys) 2024-04-26T04:26:58.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.441+0000 7f8c1f18f640 10 monclient: get_monmap_and_config success 2024-04-26T04:26:58.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.441+0000 7f8c1f18f640 4 set_mon_vals no callback set 2024-04-26T04:26:58.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.441+0000 7f8c177fe640 1 -- 172.21.15.63:0/2891099670 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c0c0ab9d0 con 0x7f8c18072830 2024-04-26T04:26:58.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.441+0000 7f8c177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:58.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.441+0000 7f8c177fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:26:58.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.441+0000 7f8c177fe640 10 monclient: dump: 2024-04-26T04:26:58.447 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:58.447 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:58.447 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:58.447 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:58.448 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:58.448 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:58.448 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:58.448 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:58.448 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:58.448 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:58.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.441+0000 7f8c1f18f640 10 monclient: shutdown 2024-04-26T04:26:58.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.441+0000 7f8c1f18f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:26:58.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.441+0000 7f8c1f18f640 1 -- 172.21.15.63:0/2891099670 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8c18072830 msgr2=0x7f8c180a1430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:58.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.441+0000 7f8c1f18f640 1 --2- 172.21.15.63:0/2891099670 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8c18072830 0x7f8c180a1430 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8c1809f590 tx=0x7f8c0c01c3a0 comp rx=0 tx=0).stop 2024-04-26T04:26:58.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.441+0000 7f8c1f18f640 1 -- 172.21.15.63:0/2891099670 shutdown_connections 2024-04-26T04:26:58.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.441+0000 7f8c1f18f640 1 --2- 172.21.15.63:0/2891099670 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8c18072830 0x7f8c180a1430 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:58.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.441+0000 7f8c1f18f640 1 --2- 172.21.15.63:0/2891099670 >> v2:172.21.15.186:3300/0 conn(0x7f8c18072450 0x7f8c18133c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:58.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.441+0000 7f8c1f18f640 1 -- 172.21.15.63:0/2891099670 >> 172.21.15.63:0/2891099670 conn(0x7f8c180a6290 msgr2=0x7f8c180a2520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:26:58.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.441+0000 7f8c1f18f640 1 -- 172.21.15.63:0/2891099670 shutdown_connections 2024-04-26T04:26:58.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.441+0000 7f8c1f18f640 1 -- 172.21.15.63:0/2891099670 wait complete. 2024-04-26T04:26:58.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.441+0000 7f8c1f18f640 10 monclient: build_initial_monmap 2024-04-26T04:26:58.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.441+0000 7f8c1f18f640 10 monclient: monmap: 2024-04-26T04:26:58.449 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:26:58.449 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:26:58.449 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:26:58.449 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:26:58.449 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:26:58.449 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:58.449 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:26:58.449 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:26:58.449 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:26:58.449 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:58.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.445+0000 7f8c1f18f640 1 Processor -- start 2024-04-26T04:26:58.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.445+0000 7f8c1f18f640 1 -- start start 2024-04-26T04:26:58.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.445+0000 7f8c1f18f640 10 monclient: init 2024-04-26T04:26:58.453 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.445+0000 7f8c1f18f640 10 monclient: _reopen_session rank -1 2024-04-26T04:26:58.453 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.445+0000 7f8c1f18f640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T04:26:58.453 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.445+0000 7f8c1f18f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c18072450 0x7f8c18130600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:58.453 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.445+0000 7f8c1f18f640 10 monclient(hunting): picked mon.noname-b con 0x7f8c18072450 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:26:58.453 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.445+0000 7f8c1f18f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8c18072830 0x7f8c18130b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:58.453 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.445+0000 7f8c1f18f640 10 monclient(hunting): picked mon.noname-a con 0x7f8c18072830 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:26:58.453 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.445+0000 7f8c1f18f640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f8c18072cb0 0x7f8c181ced40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:58.453 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.445+0000 7f8c1f18f640 10 monclient(hunting): picked mon.noname-c con 0x7f8c18072cb0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:26:58.453 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.445+0000 7f8c1f18f640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:58.453 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.445+0000 7f8c1f18f640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f8c181419b0 con 0x7f8c18072830 2024-04-26T04:26:58.453 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.445+0000 7f8c1f18f640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:58.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.445+0000 7f8c1f18f640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8c18141830 con 0x7f8c18072450 2024-04-26T04:26:58.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.445+0000 7f8c1f18f640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:58.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.445+0000 7f8c1f18f640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f8c18141b30 con 0x7f8c18072cb0 2024-04-26T04:26:58.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.445+0000 7f8c1f18f640 10 monclient(hunting): _renew_subs 2024-04-26T04:26:58.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.445+0000 7f8c1f18f640 10 monclient(hunting): authenticate will time out at 1450.152588s 2024-04-26T04:26:58.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.445+0000 7f8c17fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8c18072830 0x7f8c18130b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:58.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.445+0000 7f8c17fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8c18072830 0x7f8c18130b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:55404/0 (socket says 172.21.15.63:55404) 2024-04-26T04:26:58.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.445+0000 7f8c1d72c640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f8c18072cb0 0x7f8c181ced40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:58.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.445+0000 7f8c17fff640 1 -- 172.21.15.63:0/1599570998 learned_addr learned my addr 172.21.15.63:0/1599570998 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:26:58.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.445+0000 7f8c17fff640 10 monclient(hunting): get_auth_request con 0x7f8c18072830 auth_method 0 2024-04-26T04:26:58.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.445+0000 7f8c17fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:58.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.445+0000 7f8c17fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:58.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.445+0000 7f8c17fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:58.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.445+0000 7f8c1d72c640 10 monclient(hunting): get_auth_request con 0x7f8c18072cb0 auth_method 0 2024-04-26T04:26:58.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.445+0000 7f8c1d72c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:58.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.445+0000 7f8c1d72c640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:58.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.445+0000 7f8c1d72c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:58.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.445+0000 7f8c17fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:26:58.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.445+0000 7f8c17fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:26:58.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.445+0000 7f8c17fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:26:58.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.445+0000 7f8c1d72c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:26:58.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.445+0000 7f8c1d72c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:26:58.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.445+0000 7f8c1d72c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:26:58.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.445+0000 7f8c17fff640 10 monclient(hunting): handle_auth_done global_id 44166 payload 824 2024-04-26T04:26:58.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.445+0000 7f8c17fff640 1 -- 172.21.15.63:0/1599570998 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f8c18072cb0 msgr2=0x7f8c181ced40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:58.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.445+0000 7f8c17fff640 1 --2- 172.21.15.63:0/1599570998 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f8c18072cb0 0x7f8c181ced40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:58.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.445+0000 7f8c17fff640 1 -- 172.21.15.63:0/1599570998 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c18072450 msgr2=0x7f8c18130600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:58.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.445+0000 7f8c17fff640 1 --2- 172.21.15.63:0/1599570998 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c18072450 0x7f8c18130600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:58.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.445+0000 7f8c17fff640 10 monclient: _finish_hunting 0 2024-04-26T04:26:58.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.445+0000 7f8c17fff640 1 monclient: found mon.noname-a 2024-04-26T04:26:58.456 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.445+0000 7f8c17fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:26:58.456 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.445+0000 7f8c17fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:26:58.456 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.445+0000 7f8c17fff640 1 -- 172.21.15.63:0/1599570998 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c1809fd40 con 0x7f8c18072830 2024-04-26T04:26:58.456 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.445+0000 7f8c17fff640 1 -- 172.21.15.63:0/1599570998 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8c18072830 msgr2=0x7f8c18130b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:26:58.456 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.445+0000 7f8c17fff640 1 --2- 172.21.15.63:0/1599570998 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8c18072830 0x7f8c18130b40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8c0c01aa50 tx=0x7f8c0c019140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:26:58.456 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.445+0000 7f8c157fa640 1 -- 172.21.15.63:0/1599570998 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c0c0b0070 con 0x7f8c18072830 2024-04-26T04:26:58.456 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.445+0000 7f8c157fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:58.456 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.445+0000 7f8c157fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:26:58.456 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.445+0000 7f8c157fa640 10 monclient: dump: 2024-04-26T04:26:58.456 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:58.456 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:58.456 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:58.456 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:58.456 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:58.456 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:58.457 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:58.457 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:58.457 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:58.457 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:58.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.445+0000 7f8c157fa640 10 monclient: _finish_auth 0 2024-04-26T04:26:58.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.445+0000 7f8c157fa640 10 monclient: _check_auth_tickets 2024-04-26T04:26:58.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.445+0000 7f8c157fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:26:58.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.445+0000 7f8c157fa640 1 -- 172.21.15.63:0/1599570998 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c0c0b1040 con 0x7f8c18072830 2024-04-26T04:26:58.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.445+0000 7f8c1f18f640 5 monclient: authenticate success, global_id 44166 2024-04-26T04:26:58.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.445+0000 7f8c1f18f640 10 monclient: _renew_subs 2024-04-26T04:26:58.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.445+0000 7f8c1f18f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:26:58.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.445+0000 7f8c1f18f640 1 -- 172.21.15.63:0/1599570998 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c18132bf0 con 0x7f8c18072830 2024-04-26T04:26:58.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.445+0000 7f8c157fa640 10 monclient: handle_config config(0 keys) 2024-04-26T04:26:58.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.445+0000 7f8c157fa640 1 -- 172.21.15.63:0/1599570998 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c0c0aaa00 con 0x7f8c18072830 2024-04-26T04:26:58.458 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.445+0000 7f8c157fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:58.458 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.445+0000 7f8c157fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:26:58.458 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.445+0000 7f8c157fa640 10 monclient: dump: 2024-04-26T04:26:58.458 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:58.458 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:58.458 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:58.458 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:58.458 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:58.458 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:58.458 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:58.458 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:58.458 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:58.458 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:58.458 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.445+0000 7f8c16ffd640 4 set_mon_vals no callback set 2024-04-26T04:26:58.459 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.445+0000 7f8c1f18f640 10 monclient: _renew_subs 2024-04-26T04:26:58.459 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.445+0000 7f8c1f18f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:26:58.459 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.445+0000 7f8c1f18f640 1 -- 172.21.15.63:0/1599570998 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c1813d5d0 con 0x7f8c18072830 2024-04-26T04:26:58.459 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.449+0000 7f8c157fa640 1 -- 172.21.15.63:0/1599570998 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f8c0c0aaba0 con 0x7f8c18072830 2024-04-26T04:26:58.459 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.449+0000 7f8c1f18f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:26:58.459 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.449+0000 7f8c1f18f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:26:58.459 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.449+0000 7f8c1f18f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:26:58.459 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.449+0000 7f8c1f18f640 1 -- 172.21.15.63:0/1599570998 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c18062e10 con 0x7f8c18072830 2024-04-26T04:26:58.459 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.449+0000 7f8c157fa640 1 --2- 172.21.15.63:0/1599570998 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f8c04035f40 0x7f8c04038400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:58.459 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.449+0000 7f8c157fa640 1 -- 172.21.15.63:0/1599570998 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(323..323 src has 1..323) ==== 4346+0+0 (secure 0 0 0) 0x7f8c0c0e2a40 con 0x7f8c18072830 2024-04-26T04:26:58.459 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.449+0000 7f8c1cf2b640 1 --2- 172.21.15.63:0/1599570998 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f8c04035f40 0x7f8c04038400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:58.459 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.449+0000 7f8c1cf2b640 10 monclient: get_auth_request con 0x7f8c04035f40 auth_method 0 2024-04-26T04:26:58.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.453+0000 7f8c1cf2b640 1 --2- 172.21.15.63:0/1599570998 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f8c04035f40 0x7f8c04038400 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f8c1813d500 tx=0x7f8c08008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:26:58.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.453+0000 7f8c157fa640 1 -- 172.21.15.63:0/1599570998 DispatchQueue::entry inject delay of 0.002000 2024-04-26T04:26:58.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.453+0000 7f8c157fa640 1 -- 172.21.15.63:0/1599570998 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f8c0c0160b0 con 0x7f8c18072830 2024-04-26T04:26:58.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.453+0000 7f8c157fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:26:58.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.453+0000 7f8c157fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:26:58.612 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.609+0000 7f8c1f18f640 1 -- 172.21.15.63:0/1599570998 --> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7f8c180a1db0 con 0x7f8c04035f40 2024-04-26T04:26:58.613 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.609+0000 7f8c1cf2b640 1 -- 172.21.15.63:0/1599570998 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f8c04035f40 msgr2=0x7f8c04038400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-04-26T04:26:58.613 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.609+0000 7f8c1cf2b640 1 -- 172.21.15.63:0/1599570998 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f8c04035f40 msgr2=0x7f8c04038400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-26T04:26:58.613 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.609+0000 7f8c1cf2b640 1 --2- 172.21.15.63:0/1599570998 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f8c04035f40 0x7f8c04038400 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f8c1813d500 tx=0x7f8c08008040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-26T04:26:58.613 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.609+0000 7f8c1cf2b640 1 --2- 172.21.15.63:0/1599570998 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f8c04035f40 0x7f8c04038400 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f8c1813d500 tx=0x7f8c08008040 comp rx=0 tx=0).stop 2024-04-26T04:26:58.613 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.609+0000 7f8c157fa640 1 -- 172.21.15.63:0/1599570998 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f8c04035f40 msgr2=0x7f8c04038400 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-04-26T04:26:58.613 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:58.609+0000 7f8c157fa640 1 --2- 172.21.15.63:0/1599570998 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f8c04035f40 0x7f8c04038400 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:59.450 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.449+0000 7f8bff7fe640 1 --2- 172.21.15.63:0/1599570998 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f8c04035f40 0x7f8bf0005570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:59.450 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.449+0000 7f8bff7fe640 1 -- 172.21.15.63:0/1599570998 --> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7f8bf0005ad0 con 0x7f8c04035f40 2024-04-26T04:26:59.451 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.449+0000 7f8c1cf2b640 1 --2- 172.21.15.63:0/1599570998 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f8c04035f40 0x7f8bf0005570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:59.451 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.449+0000 7f8c1cf2b640 10 monclient: get_auth_request con 0x7f8c04035f40 auth_method 0 2024-04-26T04:26:59.454 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.453+0000 7f8c1cf2b640 1 --2- 172.21.15.63:0/1599570998 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f8c04035f40 0x7f8bf0005570 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f8c0800a5f0 tx=0x7f8c08016000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:26:59.455 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.453+0000 7f8c157fa640 1 -- 172.21.15.63:0/1599570998 <== mgr.4105 v2:172.21.15.186:6824/2583706639 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f8bf0005ad0 con 0x7f8c04035f40 2024-04-26T04:26:59.455 INFO:teuthology.orchestra.run.smithi063.stderr:instructed osd(s) 0 to deep-scrub 2024-04-26T04:26:59.458 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.457+0000 7f8c1f18f640 1 -- 172.21.15.63:0/1599570998 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f8c04035f40 msgr2=0x7f8bf0005570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:59.458 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.457+0000 7f8c1f18f640 1 --2- 172.21.15.63:0/1599570998 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f8c04035f40 0x7f8bf0005570 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f8c0800a5f0 tx=0x7f8c08016000 comp rx=0 tx=0).stop 2024-04-26T04:26:59.458 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.457+0000 7f8c1f18f640 10 monclient: shutdown 2024-04-26T04:26:59.458 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.457+0000 7f8c1f18f640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:26:59.458 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.457+0000 7f8c1f18f640 1 -- 172.21.15.63:0/1599570998 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8c18072830 msgr2=0x7f8c18130b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:59.458 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.457+0000 7f8c1f18f640 1 --2- 172.21.15.63:0/1599570998 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8c18072830 0x7f8c18130b40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8c0c01aa50 tx=0x7f8c0c019140 comp rx=0 tx=0).stop 2024-04-26T04:26:59.459 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.457+0000 7f8c1f18f640 1 -- 172.21.15.63:0/1599570998 shutdown_connections 2024-04-26T04:26:59.459 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.457+0000 7f8c1f18f640 1 --2- 172.21.15.63:0/1599570998 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f8c04035f40 0x7f8bf0005570 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:59.459 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.457+0000 7f8c1f18f640 1 --2- 172.21.15.63:0/1599570998 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f8c18072cb0 0x7f8c181ced40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:59.459 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.457+0000 7f8c1f18f640 1 --2- 172.21.15.63:0/1599570998 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8c18072830 0x7f8c18130b40 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:59.459 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.457+0000 7f8c1f18f640 1 --2- 172.21.15.63:0/1599570998 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c18072450 0x7f8c18130600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:59.459 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.457+0000 7f8c1f18f640 1 -- 172.21.15.63:0/1599570998 >> 172.21.15.63:0/1599570998 conn(0x7f8c180a6290 msgr2=0x7f8c18132310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:26:59.459 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.457+0000 7f8c1f18f640 1 -- 172.21.15.63:0/1599570998 shutdown_connections 2024-04-26T04:26:59.459 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.457+0000 7f8c1f18f640 1 -- 172.21.15.63:0/1599570998 wait complete. 2024-04-26T04:26:59.481 INFO:tasks.ceph:Scrubbing osd.1 2024-04-26T04:26:59.481 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-04-26T04:26:59.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.613+0000 7f61f84ca640 10 monclient: get_monmap_and_config 2024-04-26T04:26:59.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.613+0000 7f61f84ca640 10 monclient: build_initial_monmap 2024-04-26T04:26:59.615 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.613+0000 7f61f84ca640 10 monclient: monmap: 2024-04-26T04:26:59.615 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:26:59.615 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:59.615 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:26:59.614816+0000 2024-04-26T04:26:59.615 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:26:59.614816+0000 2024-04-26T04:26:59.615 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:26:59.615 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:59.615 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:26:59.615 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:26:59.615 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:26:59.615 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:26:59.615 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:26:59.615 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:59.617 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.613+0000 7f61f84ca640 1 Processor -- start 2024-04-26T04:26:59.617 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.613+0000 7f61f84ca640 1 -- start start 2024-04-26T04:26:59.617 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.613+0000 7f61f84ca640 10 monclient: init 2024-04-26T04:26:59.617 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.613+0000 7f61f84ca640 10 monclient: _reopen_session rank -1 2024-04-26T04:26:59.617 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.613+0000 7f61f84ca640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-04-26T04:26:59.617 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.613+0000 7f61f84ca640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f61f0072130 addr v1:172.21.15.186:6789/0 2024-04-26T04:26:59.617 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.613+0000 7f61f84ca640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f61f0137000 addr v1:172.21.15.63:6789/0 2024-04-26T04:26:59.618 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.613+0000 7f61f84ca640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f61f009f490 0x7f61f013f590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:59.618 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.613+0000 7f61f84ca640 10 monclient(hunting): picked mon.noname-c con 0x7f61f009f490 addr v2:172.21.15.186:3300/0 2024-04-26T04:26:59.618 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.613+0000 7f61f84ca640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61f00a3170 con 0x7f61f0137000 2024-04-26T04:26:59.618 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.613+0000 7f61f84ca640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61f009fc10 con 0x7f61f0072130 2024-04-26T04:26:59.618 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.613+0000 7f61f84ca640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:59.618 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.613+0000 7f61f84ca640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f61f0072cb0 con 0x7f61f009f490 2024-04-26T04:26:59.618 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.613+0000 7f61f84ca640 10 monclient(hunting): _renew_subs 2024-04-26T04:26:59.618 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.613+0000 7f61f84ca640 10 monclient(hunting): authenticate will time out at 1451.323256s 2024-04-26T04:26:59.618 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.613+0000 7f61f5a65640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f61f0137000 0x7f61f013be60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:37532/0 (socket says 172.21.15.63:37532) 2024-04-26T04:26:59.618 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.613+0000 7f61f5a65640 1 -- 172.21.15.63:0/415719227 learned_addr learned my addr 172.21.15.63:0/415719227 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-04-26T04:26:59.618 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.613+0000 7f61f6a67640 1 --2- 172.21.15.63:0/415719227 >> v2:172.21.15.186:3300/0 conn(0x7f61f009f490 0x7f61f013f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:59.618 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.613+0000 7f61f6a67640 10 monclient(hunting): get_auth_request con 0x7f61f009f490 auth_method 0 2024-04-26T04:26:59.618 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.613+0000 7f61f6a67640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:59.618 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.613+0000 7f61f6a67640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:59.618 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.613+0000 7f61f6a67640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:59.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.613+0000 7f61f6a67640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:26:59.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.613+0000 7f61f6a67640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:26:59.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.613+0000 7f61f6a67640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:26:59.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f5a65640 1 -- 172.21.15.63:0/415719227 >> v1:172.21.15.63:6789/0 conn(0x7f61f0137000 legacy=0x7f61f013be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:26:59.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f5264640 1 -- 172.21.15.63:0/415719227 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2824974594 0 0) 0x7f61f00a3170 con 0x7f61f0137000 2024-04-26T04:26:59.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f5264640 10 monclient(hunting): _negotiate 2024-04-26T04:26:59.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f5264640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:59.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f5264640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:59.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f5264640 10 monclient(hunting): my global_id is 44169 2024-04-26T04:26:59.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f5264640 1 -- 172.21.15.63:0/415719227 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f61dc003530 con 0x7f61f0137000 2024-04-26T04:26:59.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f6266640 1 -- 172.21.15.63:0/415719227 >> v1:172.21.15.186:6789/0 conn(0x7f61f0072130 legacy=0x7f61f0072510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:26:59.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f5264640 1 -- 172.21.15.63:0/415719227 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1352795957 0 0) 0x7f61dc003530 con 0x7f61f0137000 2024-04-26T04:26:59.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f5264640 1 -- 172.21.15.63:0/415719227 >> v2:172.21.15.186:3300/0 conn(0x7f61f009f490 msgr2=0x7f61f013f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:26:59.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f5264640 1 --2- 172.21.15.63:0/415719227 >> v2:172.21.15.186:3300/0 conn(0x7f61f009f490 0x7f61f013f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:59.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f5264640 1 -- 172.21.15.63:0/415719227 >> v1:172.21.15.186:6789/0 conn(0x7f61f0072130 legacy=0x7f61f0072510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:59.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f5264640 10 monclient: _finish_hunting 0 2024-04-26T04:26:59.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f5264640 1 monclient: found mon.noname-a-legacy 2024-04-26T04:26:59.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f5264640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:26:59.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f5264640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.63:6789/0 2024-04-26T04:26:59.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f5264640 1 -- 172.21.15.63:0/415719227 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61f00a33b0 con 0x7f61f0137000 2024-04-26T04:26:59.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f5264640 10 monclient: _finish_auth 0 2024-04-26T04:26:59.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f5264640 10 monclient: _check_auth_tickets 2024-04-26T04:26:59.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f5264640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:26:59.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f5264640 1 -- 172.21.15.63:0/415719227 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2612289768 0 0) 0x7f61e0001930 con 0x7f61f0137000 2024-04-26T04:26:59.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f5264640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:59.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f5264640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-26T04:26:59.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f5264640 10 monclient: dump: 2024-04-26T04:26:59.624 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:59.624 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:59.624 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:59.624 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:59.624 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:59.624 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:59.624 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:59.624 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:59.624 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:59.624 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:59.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f5264640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] but i'm connected to v1:172.21.15.63:6789/0, reconnecting 2024-04-26T04:26:59.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f5264640 10 monclient: _reopen_session rank -1 2024-04-26T04:26:59.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f5264640 1 -- 172.21.15.63:0/415719227 >> v1:172.21.15.63:6789/0 conn(0x7f61f0137000 legacy=0x7f61f013be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:59.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f5264640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T04:26:59.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f5264640 1 --2- 172.21.15.63:0/415719227 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f61dc003af0 0x7f61dc003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:59.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f5264640 10 monclient(hunting): picked mon.c con 0x7f61dc003af0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:26:59.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f5264640 1 --2- 172.21.15.63:0/415719227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61dc004810 0x7f61dc004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:59.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f5264640 10 monclient(hunting): picked mon.b con 0x7f61dc004810 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:26:59.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f5264640 1 --2- 172.21.15.63:0/415719227 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61dc0053d0 0x7f61dc00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:59.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f5264640 10 monclient(hunting): picked mon.a con 0x7f61dc0053d0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:26:59.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f5264640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:59.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f5264640 1 -- 172.21.15.63:0/415719227 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f61f0072cb0 con 0x7f61dc0053d0 2024-04-26T04:26:59.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f5264640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:59.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f5264640 1 -- 172.21.15.63:0/415719227 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f61dc00c260 con 0x7f61dc004810 2024-04-26T04:26:59.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f5264640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:59.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f5264640 1 -- 172.21.15.63:0/415719227 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f61dc00c460 con 0x7f61dc003af0 2024-04-26T04:26:59.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f5264640 10 monclient(hunting): _renew_subs 2024-04-26T04:26:59.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f5264640 10 monclient(hunting): _finish_auth 0 2024-04-26T04:26:59.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f6266640 1 --2- 172.21.15.63:0/415719227 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f61dc003af0 0x7f61dc003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:59.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f6a67640 1 --2- 172.21.15.63:0/415719227 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61dc0053d0 0x7f61dc00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:59.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f6266640 10 monclient(hunting): get_auth_request con 0x7f61dc003af0 auth_method 0 2024-04-26T04:26:59.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f6266640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:59.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f6266640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:59.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f6266640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:26:59.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f6a67640 10 monclient(hunting): get_auth_request con 0x7f61dc0053d0 auth_method 0 2024-04-26T04:26:59.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f6a67640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:59.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f6a67640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:59.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f6a67640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:26:59.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f6a67640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:26:59.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f6a67640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:26:59.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f6a67640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T04:26:59.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f6266640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:26:59.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f6266640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:26:59.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f6266640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T04:26:59.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f5a65640 1 --2- 172.21.15.63:0/415719227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61dc004810 0x7f61dc004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:59.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f5a65640 10 monclient(hunting): get_auth_request con 0x7f61dc004810 auth_method 0 2024-04-26T04:26:59.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f5a65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:59.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f5a65640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:59.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f5a65640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:26:59.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f6a67640 10 monclient(hunting): handle_auth_done global_id 44169 payload 293 2024-04-26T04:26:59.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f6a67640 1 -- 172.21.15.63:0/415719227 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f61dc003af0 msgr2=0x7f61dc003f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:26:59.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f6a67640 1 --2- 172.21.15.63:0/415719227 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f61dc003af0 0x7f61dc003f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:59.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f6a67640 1 -- 172.21.15.63:0/415719227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61dc004810 msgr2=0x7f61dc004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:26:59.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f6a67640 1 --2- 172.21.15.63:0/415719227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61dc004810 0x7f61dc004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:59.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f6a67640 10 monclient: _finish_hunting 0 2024-04-26T04:26:59.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f6a67640 1 monclient: found mon.a 2024-04-26T04:26:59.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f6a67640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:26:59.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f6a67640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:26:59.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f6a67640 1 -- 172.21.15.63:0/415719227 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61dc003530 con 0x7f61dc0053d0 2024-04-26T04:26:59.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f6a67640 10 monclient: _finish_auth 0 2024-04-26T04:26:59.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f6a67640 10 monclient: _check_auth_tickets 2024-04-26T04:26:59.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f6a67640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:26:59.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f84ca640 5 monclient: authenticate success, global_id 44169 2024-04-26T04:26:59.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f84ca640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T04:26:59.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.617+0000 7f61f6266640 1 --2- 172.21.15.63:0/415719227 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f61dc003af0 0x7f61dc003f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T04:26:59.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.621+0000 7f61f6a67640 1 -- 172.21.15.63:0/415719227 reap_dead start 2024-04-26T04:26:59.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.621+0000 7f61f6a67640 1 -- 172.21.15.63:0/415719227 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61dc0053d0 msgr2=0x7f61dc00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:26:59.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.621+0000 7f61f6a67640 1 --2- 172.21.15.63:0/415719227 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61dc0053d0 0x7f61dc00b9d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f61e8009f90 tx=0x7f61e801cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38179530c829f000 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:26:59.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.621+0000 7f61f5264640 1 -- 172.21.15.63:0/415719227 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61ec001680 con 0x7f61dc0053d0 2024-04-26T04:26:59.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.621+0000 7f61f5264640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:59.634 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.621+0000 7f61f5264640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:26:59.634 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.621+0000 7f61f5264640 10 monclient: dump: 2024-04-26T04:26:59.634 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:59.634 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:59.634 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:59.634 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:59.634 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:59.634 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:59.634 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:59.634 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:59.634 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:59.634 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:59.634 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.621+0000 7f61f84ca640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T04:26:59.634 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.621+0000 7f61f5264640 1 -- 172.21.15.63:0/415719227 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61e80a5c70 con 0x7f61dc0053d0 2024-04-26T04:26:59.634 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.621+0000 7f61f5264640 10 monclient: handle_config config(0 keys) 2024-04-26T04:26:59.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.621+0000 7f61f84ca640 10 monclient: get_monmap_and_config success 2024-04-26T04:26:59.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.621+0000 7f61f84ca640 4 set_mon_vals no callback set 2024-04-26T04:26:59.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.621+0000 7f61f5264640 1 -- 172.21.15.63:0/415719227 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61e8009c10 con 0x7f61dc0053d0 2024-04-26T04:26:59.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.621+0000 7f61f5264640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:59.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.621+0000 7f61f5264640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:26:59.635 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.621+0000 7f61f5264640 10 monclient: dump: 2024-04-26T04:26:59.635 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:59.635 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:59.635 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:59.635 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:59.635 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:59.635 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:59.635 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:59.635 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:59.636 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:59.636 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:59.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.621+0000 7f61f84ca640 10 monclient: shutdown 2024-04-26T04:26:59.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.621+0000 7f61f84ca640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:26:59.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.621+0000 7f61f84ca640 1 -- 172.21.15.63:0/415719227 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61dc0053d0 msgr2=0x7f61dc00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:59.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.621+0000 7f61f84ca640 1 --2- 172.21.15.63:0/415719227 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61dc0053d0 0x7f61dc00b9d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f61e8009f90 tx=0x7f61e801cc60 comp rx=0 tx=0).stop 2024-04-26T04:26:59.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.621+0000 7f61f84ca640 1 -- 172.21.15.63:0/415719227 shutdown_connections 2024-04-26T04:26:59.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.621+0000 7f61f84ca640 1 --2- 172.21.15.63:0/415719227 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61dc0053d0 0x7f61dc00b9d0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:59.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.621+0000 7f61f84ca640 1 -- 172.21.15.63:0/415719227 >> 172.21.15.63:0/415719227 conn(0x7f61f00a6090 msgr2=0x7f61f00a2590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:26:59.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.621+0000 7f61f84ca640 1 -- 172.21.15.63:0/415719227 shutdown_connections 2024-04-26T04:26:59.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.621+0000 7f61f84ca640 1 -- 172.21.15.63:0/415719227 wait complete. 2024-04-26T04:26:59.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.625+0000 7f61f84ca640 10 monclient: build_initial_monmap 2024-04-26T04:26:59.636 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.625+0000 7f61f84ca640 10 monclient: monmap: 2024-04-26T04:26:59.636 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:26:59.636 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:26:59.637 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:26:59.637 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:26:59.637 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:26:59.637 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:59.637 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:26:59.637 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:26:59.637 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:26:59.637 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:59.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.625+0000 7f61f84ca640 1 Processor -- start 2024-04-26T04:26:59.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.625+0000 7f61f84ca640 1 -- start start 2024-04-26T04:26:59.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.625+0000 7f61f84ca640 10 monclient: init 2024-04-26T04:26:59.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.625+0000 7f61f84ca640 10 monclient: _reopen_session rank -1 2024-04-26T04:26:59.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.625+0000 7f61f84ca640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T04:26:59.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.625+0000 7f61f84ca640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61dc0053d0 0x7f61f01cbee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:59.637 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.625+0000 7f61f84ca640 10 monclient(hunting): picked mon.noname-a con 0x7f61dc0053d0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:26:59.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.625+0000 7f61f84ca640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f61f012f3a0 0x7f61f012f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:59.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.625+0000 7f61f84ca640 10 monclient(hunting): picked mon.noname-c con 0x7f61f012f3a0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:26:59.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.625+0000 7f61f84ca640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61f012fce0 0x7f61f011ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:59.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.625+0000 7f61f84ca640 10 monclient(hunting): picked mon.noname-b con 0x7f61f012fce0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:26:59.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.625+0000 7f61f84ca640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:59.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.625+0000 7f61f84ca640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f61f0072690 con 0x7f61dc0053d0 2024-04-26T04:26:59.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.625+0000 7f61f84ca640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:59.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.625+0000 7f61f84ca640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f61f0072510 con 0x7f61f012fce0 2024-04-26T04:26:59.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.625+0000 7f61f84ca640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:59.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.625+0000 7f61f84ca640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f61f0072810 con 0x7f61f012f3a0 2024-04-26T04:26:59.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.625+0000 7f61f84ca640 10 monclient(hunting): _renew_subs 2024-04-26T04:26:59.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.625+0000 7f61f84ca640 10 monclient(hunting): authenticate will time out at 1451.333154s 2024-04-26T04:26:59.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.625+0000 7f61f6266640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61dc0053d0 0x7f61f01cbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:59.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.625+0000 7f61f5a65640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f61f012f3a0 0x7f61f012f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:59.638 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.625+0000 7f61f6266640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61dc0053d0 0x7f61f01cbee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:47490/0 (socket says 172.21.15.63:47490) 2024-04-26T04:26:59.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.625+0000 7f61f5a65640 10 monclient(hunting): get_auth_request con 0x7f61f012f3a0 auth_method 0 2024-04-26T04:26:59.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.625+0000 7f61f6266640 1 -- 172.21.15.63:0/1989371354 learned_addr learned my addr 172.21.15.63:0/1989371354 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:26:59.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.625+0000 7f61f5a65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:59.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.625+0000 7f61f5a65640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:59.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.625+0000 7f61f5a65640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:59.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.625+0000 7f61f6266640 10 monclient(hunting): get_auth_request con 0x7f61dc0053d0 auth_method 0 2024-04-26T04:26:59.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.625+0000 7f61f6266640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:59.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.625+0000 7f61f6266640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:59.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.625+0000 7f61f6266640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:59.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.625+0000 7f61f5a65640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:26:59.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.625+0000 7f61f5a65640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:26:59.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.625+0000 7f61f5a65640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:26:59.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.625+0000 7f61f6266640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:26:59.639 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.625+0000 7f61f6266640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:26:59.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.625+0000 7f61f6266640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:26:59.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.625+0000 7f61f6a67640 1 --2- 172.21.15.63:0/1989371354 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61f012fce0 0x7f61f011ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:59.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.625+0000 7f61f6a67640 10 monclient(hunting): get_auth_request con 0x7f61f012fce0 auth_method 0 2024-04-26T04:26:59.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.625+0000 7f61f6a67640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:59.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.625+0000 7f61f6a67640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:59.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.625+0000 7f61f6a67640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:59.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.625+0000 7f61f5a65640 10 monclient(hunting): handle_auth_done global_id 24725 payload 824 2024-04-26T04:26:59.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.625+0000 7f61f5a65640 1 -- 172.21.15.63:0/1989371354 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61f012fce0 msgr2=0x7f61f011ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:59.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.625+0000 7f61f5a65640 1 --2- 172.21.15.63:0/1989371354 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61f012fce0 0x7f61f011ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:59.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.625+0000 7f61f5a65640 1 -- 172.21.15.63:0/1989371354 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61dc0053d0 msgr2=0x7f61f01cbee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:59.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.625+0000 7f61f5a65640 1 --2- 172.21.15.63:0/1989371354 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61dc0053d0 0x7f61f01cbee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:59.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.625+0000 7f61f5a65640 10 monclient: _finish_hunting 0 2024-04-26T04:26:59.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.625+0000 7f61f5a65640 1 monclient: found mon.noname-c 2024-04-26T04:26:59.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.625+0000 7f61f5a65640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:26:59.640 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.625+0000 7f61f5a65640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.63:3301/0 2024-04-26T04:26:59.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.625+0000 7f61f5a65640 1 -- 172.21.15.63:0/1989371354 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61f0120da0 con 0x7f61f012f3a0 2024-04-26T04:26:59.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.625+0000 7f61f6266640 1 --2- 172.21.15.63:0/1989371354 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61dc0053d0 0x7f61f01cbee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T04:26:59.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.625+0000 7f61f5a65640 1 -- 172.21.15.63:0/1989371354 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f61f012f3a0 msgr2=0x7f61f012f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:26:59.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.625+0000 7f61f5a65640 1 --2- 172.21.15.63:0/1989371354 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f61f012f3a0 0x7f61f012f7a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f61f00a2560 tx=0x7f61e0011ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:26:59.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.629+0000 7f61e6ffd640 1 -- 172.21.15.63:0/1989371354 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61e0010040 con 0x7f61f012f3a0 2024-04-26T04:26:59.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.629+0000 7f61e6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:59.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.629+0000 7f61e6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:26:59.641 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.629+0000 7f61e6ffd640 10 monclient: dump: 2024-04-26T04:26:59.641 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:59.641 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:59.641 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:59.641 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:59.641 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:59.641 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:59.641 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:59.642 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:59.642 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:59.642 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:59.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.629+0000 7f61e6ffd640 10 monclient: _finish_auth 0 2024-04-26T04:26:59.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.629+0000 7f61e6ffd640 10 monclient: _check_auth_tickets 2024-04-26T04:26:59.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.629+0000 7f61e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:26:59.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.629+0000 7f61f84ca640 5 monclient: authenticate success, global_id 24725 2024-04-26T04:26:59.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.629+0000 7f61e6ffd640 1 -- 172.21.15.63:0/1989371354 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61e001a070 con 0x7f61f012f3a0 2024-04-26T04:26:59.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.629+0000 7f61e6ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T04:26:59.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.629+0000 7f61e6ffd640 1 -- 172.21.15.63:0/1989371354 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61e000fd90 con 0x7f61f012f3a0 2024-04-26T04:26:59.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.629+0000 7f61f84ca640 10 monclient: _renew_subs 2024-04-26T04:26:59.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.629+0000 7f61f84ca640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:26:59.642 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.629+0000 7f61f84ca640 1 -- 172.21.15.63:0/1989371354 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61f009f8e0 con 0x7f61f012f3a0 2024-04-26T04:26:59.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.629+0000 7f61f4a63640 4 set_mon_vals no callback set 2024-04-26T04:26:59.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.629+0000 7f61e6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:59.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.629+0000 7f61e6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:26:59.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.629+0000 7f61e6ffd640 10 monclient: dump: 2024-04-26T04:26:59.643 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:59.643 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:59.643 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:59.643 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:59.643 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:59.643 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:59.643 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:59.643 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:59.643 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:59.643 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:59.643 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.629+0000 7f61f84ca640 10 monclient: _renew_subs 2024-04-26T04:26:59.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.629+0000 7f61f84ca640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:26:59.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.629+0000 7f61f84ca640 1 -- 172.21.15.63:0/1989371354 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f61f011f120 con 0x7f61f012f3a0 2024-04-26T04:26:59.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.629+0000 7f61f84ca640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:26:59.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.629+0000 7f61f84ca640 1 -- 172.21.15.63:0/1989371354 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f61f0067fb0 con 0x7f61f012f3a0 2024-04-26T04:26:59.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.629+0000 7f61e6ffd640 1 -- 172.21.15.63:0/1989371354 <== mon.2 v2:172.21.15.63:3301/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f61e001f460 con 0x7f61f012f3a0 2024-04-26T04:26:59.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.629+0000 7f61e6ffd640 1 --2- 172.21.15.63:0/1989371354 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f61d4035ec0 0x7f61d4038380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:59.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.629+0000 7f61e6ffd640 1 -- 172.21.15.63:0/1989371354 <== mon.2 v2:172.21.15.63:3301/0 5 ==== osd_map(323..323 src has 1..323) ==== 4346+0+0 (secure 0 0 0) 0x7f61e004bb40 con 0x7f61f012f3a0 2024-04-26T04:26:59.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.629+0000 7f61e6ffd640 1 --2- 172.21.15.63:0/1989371354 >> [v2:172.21.15.63:6816/3596874741,v1:172.21.15.63:6817/3596874741] conn(0x7f61d403bd30 0x7f61d403e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:59.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.629+0000 7f61e6ffd640 1 -- 172.21.15.63:0/1989371354 --> [v2:172.21.15.63:6816/3596874741,v1:172.21.15.63:6817/3596874741] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f61e0023ec0 con 0x7f61d403bd30 2024-04-26T04:26:59.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.629+0000 7f61e6ffd640 1 -- 172.21.15.63:0/1989371354 <== mon.2 v2:172.21.15.63:3301/0 6 ==== mon_get_version_reply(handle=1 version=323) ==== 24+0+0 (secure 0 0 0) 0x7f61e00502a0 con 0x7f61f012f3a0 2024-04-26T04:26:59.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.629+0000 7f61e6ffd640 10 monclient: handle_get_version_reply finishing 1 version 323 2024-04-26T04:26:59.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.629+0000 7f61f6a67640 1 --2- 172.21.15.63:0/1989371354 >> [v2:172.21.15.63:6816/3596874741,v1:172.21.15.63:6817/3596874741] conn(0x7f61d403bd30 0x7f61d403e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:59.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.629+0000 7f61f6a67640 10 monclient: get_auth_request con 0x7f61d403bd30 auth_method 0 2024-04-26T04:26:59.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.629+0000 7f61f6266640 1 --2- 172.21.15.63:0/1989371354 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f61d4035ec0 0x7f61d4038380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:59.644 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.629+0000 7f61f6266640 10 monclient: get_auth_request con 0x7f61d4035ec0 auth_method 0 2024-04-26T04:26:59.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.633+0000 7f61f6266640 1 --2- 172.21.15.63:0/1989371354 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f61d4035ec0 0x7f61d4038380 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f61ec007a90 tx=0x7f61ec009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:26:59.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.633+0000 7f61f6a67640 1 --2- 172.21.15.63:0/1989371354 >> [v2:172.21.15.63:6816/3596874741,v1:172.21.15.63:6817/3596874741] conn(0x7f61d403bd30 0x7f61d403e150 crc :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:26:59.645 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.637+0000 7f61e6ffd640 1 -- 172.21.15.63:0/1989371354 <== osd.1 v2:172.21.15.63:6816/3596874741 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f61e0023ec0 con 0x7f61d403bd30 2024-04-26T04:26:59.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.649+0000 7f61f84ca640 1 -- 172.21.15.63:0/1989371354 --> [v2:172.21.15.63:6816/3596874741,v1:172.21.15.63:6817/3596874741] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f61f0120690 con 0x7f61d403bd30 2024-04-26T04:26:59.651 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.649+0000 7f61e6ffd640 1 -- 172.21.15.63:0/1989371354 <== osd.1 v2:172.21.15.63:6816/3596874741 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f61e80a5a70 con 0x7f61d403bd30 2024-04-26T04:26:59.651 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-26T04:26:59.651 INFO:teuthology.orchestra.run.smithi063.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-26T04:26:59.652 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-26T04:26:59.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.649+0000 7f61f84ca640 1 -- 172.21.15.63:0/1989371354 >> [v2:172.21.15.63:6816/3596874741,v1:172.21.15.63:6817/3596874741] conn(0x7f61d403bd30 msgr2=0x7f61d403e150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:59.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.649+0000 7f61f84ca640 1 --2- 172.21.15.63:0/1989371354 >> [v2:172.21.15.63:6816/3596874741,v1:172.21.15.63:6817/3596874741] conn(0x7f61d403bd30 0x7f61d403e150 crc :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:59.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.649+0000 7f61f84ca640 1 -- 172.21.15.63:0/1989371354 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f61d4035ec0 msgr2=0x7f61d4038380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:59.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.649+0000 7f61f84ca640 1 --2- 172.21.15.63:0/1989371354 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f61d4035ec0 0x7f61d4038380 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f61ec007a90 tx=0x7f61ec009040 comp rx=0 tx=0).stop 2024-04-26T04:26:59.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.649+0000 7f61f84ca640 10 monclient: shutdown 2024-04-26T04:26:59.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.649+0000 7f61f84ca640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:26:59.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.649+0000 7f61f84ca640 1 -- 172.21.15.63:0/1989371354 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f61f012f3a0 msgr2=0x7f61f012f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:59.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.649+0000 7f61f84ca640 1 --2- 172.21.15.63:0/1989371354 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f61f012f3a0 0x7f61f012f7a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f61f00a2560 tx=0x7f61e0011ca0 comp rx=0 tx=0).stop 2024-04-26T04:26:59.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.649+0000 7f61f6a67640 1 -- 172.21.15.63:0/1989371354 reap_dead start 2024-04-26T04:26:59.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.649+0000 7f61f84ca640 1 -- 172.21.15.63:0/1989371354 shutdown_connections 2024-04-26T04:26:59.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.649+0000 7f61f84ca640 1 -- 172.21.15.63:0/1989371354 >> 172.21.15.63:0/1989371354 conn(0x7f61f00a6090 msgr2=0x7f61f0130860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:26:59.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.649+0000 7f61f84ca640 1 -- 172.21.15.63:0/1989371354 shutdown_connections 2024-04-26T04:26:59.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.649+0000 7f61f84ca640 1 -- 172.21.15.63:0/1989371354 wait complete. 2024-04-26T04:26:59.667 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-04-26T04:26:59.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.773+0000 7f38242f4640 10 monclient: get_monmap_and_config 2024-04-26T04:26:59.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.773+0000 7f38242f4640 10 monclient: build_initial_monmap 2024-04-26T04:26:59.781 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.773+0000 7f38242f4640 10 monclient: monmap: 2024-04-26T04:26:59.781 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:26:59.781 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:59.781 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:26:59.777784+0000 2024-04-26T04:26:59.781 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:26:59.777784+0000 2024-04-26T04:26:59.782 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:26:59.782 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:59.782 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:26:59.782 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:26:59.782 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:26:59.782 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:26:59.782 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:26:59.782 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:59.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.777+0000 7f38242f4640 1 Processor -- start 2024-04-26T04:26:59.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.777+0000 7f38242f4640 1 -- start start 2024-04-26T04:26:59.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.777+0000 7f38242f4640 10 monclient: init 2024-04-26T04:26:59.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.777+0000 7f38242f4640 10 monclient: _reopen_session rank -1 2024-04-26T04:26:59.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.777+0000 7f38242f4640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2024-04-26T04:26:59.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.777+0000 7f38242f4640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f381c09b3b0 0x7f381c09e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:59.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.777+0000 7f38242f4640 10 monclient(hunting): picked mon.noname-c con 0x7f381c09b3b0 addr v2:172.21.15.186:3300/0 2024-04-26T04:26:59.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.777+0000 7f38242f4640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f381c0a0bb0 0x7f381c0a0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:59.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.777+0000 7f38242f4640 10 monclient(hunting): picked mon.noname-b con 0x7f381c0a0bb0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:26:59.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.777+0000 7f38242f4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f381c0a14d0 addr v1:172.21.15.63:6789/0 2024-04-26T04:26:59.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.777+0000 7f38242f4640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f381c09dde0 con 0x7f381c0a14d0 2024-04-26T04:26:59.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.777+0000 7f38242f4640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:59.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.777+0000 7f38242f4640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f381c09c100 con 0x7f381c09b3b0 2024-04-26T04:26:59.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.777+0000 7f38242f4640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:59.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.777+0000 7f38242f4640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f381c09b790 con 0x7f381c0a0bb0 2024-04-26T04:26:59.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.777+0000 7f38242f4640 10 monclient(hunting): _renew_subs 2024-04-26T04:26:59.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.777+0000 7f38242f4640 10 monclient(hunting): authenticate will time out at 1451.485362s 2024-04-26T04:26:59.783 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.777+0000 7f3822891640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f381c0a14d0 0x7f381c13b340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:37540/0 (socket says 172.21.15.63:37540) 2024-04-26T04:26:59.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.777+0000 7f382188f640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f381c0a0bb0 0x7f381c0a0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:59.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.777+0000 7f3822891640 1 -- 172.21.15.63:0/1605967498 learned_addr learned my addr 172.21.15.63:0/1605967498 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-04-26T04:26:59.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.777+0000 7f382188f640 10 monclient(hunting): get_auth_request con 0x7f381c0a0bb0 auth_method 0 2024-04-26T04:26:59.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.777+0000 7f382188f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:59.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.777+0000 7f382188f640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:59.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.777+0000 7f382188f640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:59.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.777+0000 7f382188f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:26:59.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.777+0000 7f382188f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:26:59.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.777+0000 7f382188f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:26:59.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.777+0000 7f382188f640 10 monclient(hunting): handle_auth_done global_id 24728 payload 274 2024-04-26T04:26:59.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.777+0000 7f3822090640 1 --2- 172.21.15.63:0/1605967498 >> v2:172.21.15.186:3300/0 conn(0x7f381c09b3b0 0x7f381c09e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:59.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.777+0000 7f382188f640 1 -- 172.21.15.63:0/1605967498 >> v2:172.21.15.186:3300/0 conn(0x7f381c09b3b0 msgr2=0x7f381c09e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:26:59.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.777+0000 7f382188f640 1 --2- 172.21.15.63:0/1605967498 >> v2:172.21.15.186:3300/0 conn(0x7f381c09b3b0 0x7f381c09e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:59.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.777+0000 7f382188f640 1 -- 172.21.15.63:0/1605967498 >> v1:172.21.15.63:6789/0 conn(0x7f381c0a14d0 legacy=0x7f381c13b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:26:59.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.777+0000 7f382188f640 10 monclient: _finish_hunting 0 2024-04-26T04:26:59.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.777+0000 7f382188f640 1 monclient: found mon.noname-b 2024-04-26T04:26:59.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.777+0000 7f382188f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:26:59.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.777+0000 7f382188f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.63:3301/0 2024-04-26T04:26:59.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.777+0000 7f382188f640 1 -- 172.21.15.63:0/1605967498 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f381c09c9b0 con 0x7f381c0a0bb0 2024-04-26T04:26:59.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.777+0000 7f3822090640 10 monclient: get_auth_request con 0x7f381c09b3b0 auth_method 0 2024-04-26T04:26:59.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.777+0000 7f3822090640 1 --2- 172.21.15.63:0/1605967498 >> v2:172.21.15.186:3300/0 conn(0x7f381c09b3b0 0x7f381c09e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T04:26:59.788 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.781+0000 7f382188f640 1 -- 172.21.15.63:0/1605967498 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f381c0a0bb0 msgr2=0x7f381c0a0f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:26:59.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.781+0000 7f382188f640 1 --2- 172.21.15.63:0/1605967498 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f381c0a0bb0 0x7f381c0a0f90 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f380c0097c0 tx=0x7f380c01c140 comp rx=0 tx=0).ready entity=mon.2 client_cookie=61d18a1c003fecad server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:26:59.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.781+0000 7f382108e640 1 -- 172.21.15.63:0/1605967498 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f380c0b0070 con 0x7f381c0a0bb0 2024-04-26T04:26:59.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.781+0000 7f382108e640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:59.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.781+0000 7f382108e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T04:26:59.789 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.781+0000 7f382108e640 10 monclient: dump: 2024-04-26T04:26:59.789 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:59.789 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:59.789 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:59.789 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:59.789 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:59.789 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:59.789 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:59.790 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:59.790 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:59.790 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:59.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.781+0000 7f382108e640 10 monclient: _finish_auth 0 2024-04-26T04:26:59.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.781+0000 7f382108e640 10 monclient: _check_auth_tickets 2024-04-26T04:26:59.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.781+0000 7f382108e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:26:59.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.781+0000 7f382108e640 1 -- 172.21.15.63:0/1605967498 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f380c0ab600 con 0x7f381c0a0bb0 2024-04-26T04:26:59.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.781+0000 7f38242f4640 5 monclient: authenticate success, global_id 24728 2024-04-26T04:26:59.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.781+0000 7f38242f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T04:26:59.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.781+0000 7f382108e640 10 monclient: handle_config config(0 keys) 2024-04-26T04:26:59.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.781+0000 7f38242f4640 10 monclient: get_monmap_and_config success 2024-04-26T04:26:59.790 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.781+0000 7f38242f4640 4 set_mon_vals no callback set 2024-04-26T04:26:59.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.781+0000 7f382108e640 1 -- 172.21.15.63:0/1605967498 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f380c0ab9d0 con 0x7f381c0a0bb0 2024-04-26T04:26:59.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.781+0000 7f382108e640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:59.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.781+0000 7f382108e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:26:59.791 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.781+0000 7f382108e640 10 monclient: dump: 2024-04-26T04:26:59.791 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:59.791 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:59.791 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:59.791 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:59.791 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:59.791 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:59.791 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:59.791 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:59.791 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:59.792 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:59.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.781+0000 7f38242f4640 10 monclient: shutdown 2024-04-26T04:26:59.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.781+0000 7f38242f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:26:59.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.781+0000 7f38242f4640 1 -- 172.21.15.63:0/1605967498 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f381c0a0bb0 msgr2=0x7f381c0a0f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:59.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.781+0000 7f38242f4640 1 --2- 172.21.15.63:0/1605967498 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f381c0a0bb0 0x7f381c0a0f90 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f380c0097c0 tx=0x7f380c01c140 comp rx=0 tx=0).stop 2024-04-26T04:26:59.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.781+0000 7f38242f4640 1 -- 172.21.15.63:0/1605967498 shutdown_connections 2024-04-26T04:26:59.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.781+0000 7f38242f4640 1 --2- 172.21.15.63:0/1605967498 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f381c0a0bb0 0x7f381c0a0f90 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:59.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.781+0000 7f38242f4640 1 --2- 172.21.15.63:0/1605967498 >> v2:172.21.15.186:3300/0 conn(0x7f381c09b3b0 0x7f381c09e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:59.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.781+0000 7f38242f4640 1 -- 172.21.15.63:0/1605967498 >> 172.21.15.63:0/1605967498 conn(0x7f381c0a1e70 msgr2=0x7f381c0a2280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:26:59.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.781+0000 7f38242f4640 1 -- 172.21.15.63:0/1605967498 shutdown_connections 2024-04-26T04:26:59.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.781+0000 7f38242f4640 1 -- 172.21.15.63:0/1605967498 wait complete. 2024-04-26T04:26:59.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.785+0000 7f38242f4640 10 monclient: build_initial_monmap 2024-04-26T04:26:59.792 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.785+0000 7f38242f4640 10 monclient: monmap: 2024-04-26T04:26:59.792 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:26:59.792 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:26:59.793 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:26:59.793 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:26:59.793 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:26:59.793 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:59.793 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:26:59.793 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:26:59.793 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:26:59.793 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:59.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.785+0000 7f38242f4640 1 Processor -- start 2024-04-26T04:26:59.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.785+0000 7f38242f4640 1 -- start start 2024-04-26T04:26:59.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.785+0000 7f38242f4640 10 monclient: init 2024-04-26T04:26:59.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.785+0000 7f38242f4640 10 monclient: _reopen_session rank -1 2024-04-26T04:26:59.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.785+0000 7f38242f4640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T04:26:59.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.785+0000 7f38242f4640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f381c09b3b0 0x7f381c1ccc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:59.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.785+0000 7f38242f4640 10 monclient(hunting): picked mon.noname-c con 0x7f381c09b3b0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:26:59.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.785+0000 7f38242f4640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f381c0a0bb0 0x7f381c0982f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:59.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.785+0000 7f38242f4640 10 monclient(hunting): picked mon.noname-a con 0x7f381c0a0bb0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:26:59.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.785+0000 7f38242f4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f381c0a14d0 0x7f381c098830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:59.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.785+0000 7f38242f4640 10 monclient(hunting): picked mon.noname-b con 0x7f381c0a14d0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:26:59.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.785+0000 7f38242f4640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:59.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.785+0000 7f38242f4640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f381c13f400 con 0x7f381c0a0bb0 2024-04-26T04:26:59.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.785+0000 7f38242f4640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:59.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.785+0000 7f38242f4640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f381c13f280 con 0x7f381c0a14d0 2024-04-26T04:26:59.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.785+0000 7f38242f4640 10 monclient(hunting): start opening mon connection 2024-04-26T04:26:59.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.785+0000 7f38242f4640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f381c13f580 con 0x7f381c09b3b0 2024-04-26T04:26:59.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.785+0000 7f38242f4640 10 monclient(hunting): _renew_subs 2024-04-26T04:26:59.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.785+0000 7f38242f4640 10 monclient(hunting): authenticate will time out at 1451.493659s 2024-04-26T04:26:59.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.785+0000 7f382188f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f381c0a0bb0 0x7f381c0982f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:59.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.785+0000 7f3822090640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f381c09b3b0 0x7f381c1ccc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:59.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.785+0000 7f382188f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f381c0a0bb0 0x7f381c0982f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:47502/0 (socket says 172.21.15.63:47502) 2024-04-26T04:26:59.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.785+0000 7f382188f640 1 -- 172.21.15.63:0/1013191647 learned_addr learned my addr 172.21.15.63:0/1013191647 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:26:59.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.785+0000 7f382188f640 10 monclient(hunting): get_auth_request con 0x7f381c0a0bb0 auth_method 0 2024-04-26T04:26:59.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.785+0000 7f382188f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:59.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.785+0000 7f382188f640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:59.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.785+0000 7f382188f640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:59.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.785+0000 7f3822090640 10 monclient(hunting): get_auth_request con 0x7f381c09b3b0 auth_method 0 2024-04-26T04:26:59.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.785+0000 7f3822090640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:26:59.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.785+0000 7f3822090640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:26:59.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.785+0000 7f3822090640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:26:59.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.785+0000 7f382188f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:26:59.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.785+0000 7f382188f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:26:59.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.785+0000 7f382188f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:26:59.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.785+0000 7f3822090640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:26:59.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.785+0000 7f3822090640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:26:59.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.785+0000 7f3822090640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:26:59.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.785+0000 7f3822891640 1 --2- 172.21.15.63:0/1013191647 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f381c0a14d0 0x7f381c098830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:59.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.785+0000 7f382188f640 10 monclient(hunting): handle_auth_done global_id 44175 payload 824 2024-04-26T04:26:59.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.785+0000 7f382188f640 1 -- 172.21.15.63:0/1013191647 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f381c09b3b0 msgr2=0x7f381c1ccc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:59.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.785+0000 7f382188f640 1 --2- 172.21.15.63:0/1013191647 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f381c09b3b0 0x7f381c1ccc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:59.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.785+0000 7f382188f640 1 -- 172.21.15.63:0/1013191647 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f381c0a14d0 msgr2=0x7f381c098830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:59.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.785+0000 7f382188f640 1 --2- 172.21.15.63:0/1013191647 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f381c0a14d0 0x7f381c098830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:59.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.785+0000 7f382188f640 10 monclient: _finish_hunting 0 2024-04-26T04:26:59.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.785+0000 7f382188f640 1 monclient: found mon.noname-a 2024-04-26T04:26:59.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.785+0000 7f382188f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:26:59.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.785+0000 7f382188f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:26:59.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.785+0000 7f382188f640 1 -- 172.21.15.63:0/1013191647 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f381c0917b0 con 0x7f381c0a0bb0 2024-04-26T04:26:59.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.785+0000 7f3822891640 10 monclient: get_auth_request con 0x7f381c0a14d0 auth_method 0 2024-04-26T04:26:59.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.785+0000 7f3822891640 1 --2- 172.21.15.63:0/1013191647 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f381c0a14d0 0x7f381c098830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T04:26:59.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.789+0000 7f382188f640 1 -- 172.21.15.63:0/1013191647 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f381c0a0bb0 msgr2=0x7f381c0982f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:26:59.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.789+0000 7f382188f640 1 --2- 172.21.15.63:0/1013191647 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f381c0a0bb0 0x7f381c0982f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f380c01c9e0 tx=0x7f380c018ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:26:59.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.789+0000 7f3812ffd640 1 -- 172.21.15.63:0/1013191647 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f380c0b0040 con 0x7f381c0a0bb0 2024-04-26T04:26:59.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.789+0000 7f3812ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:59.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.789+0000 7f3812ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:26:59.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.789+0000 7f3812ffd640 10 monclient: dump: 2024-04-26T04:26:59.799 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:59.799 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:59.799 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:59.799 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:59.799 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:59.799 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:59.799 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:59.799 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:59.800 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:59.800 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:59.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.789+0000 7f3812ffd640 10 monclient: _finish_auth 0 2024-04-26T04:26:59.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.789+0000 7f3812ffd640 10 monclient: _check_auth_tickets 2024-04-26T04:26:59.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.789+0000 7f3812ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:26:59.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.789+0000 7f3812ffd640 1 -- 172.21.15.63:0/1013191647 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f380c0b9070 con 0x7f381c0a0bb0 2024-04-26T04:26:59.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.789+0000 7f38242f4640 5 monclient: authenticate success, global_id 44175 2024-04-26T04:26:59.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.789+0000 7f38242f4640 10 monclient: _renew_subs 2024-04-26T04:26:59.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.789+0000 7f38242f4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:26:59.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.789+0000 7f38242f4640 1 -- 172.21.15.63:0/1013191647 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f381c091a90 con 0x7f381c0a0bb0 2024-04-26T04:26:59.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.789+0000 7f3812ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T04:26:59.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.789+0000 7f3812ffd640 1 -- 172.21.15.63:0/1013191647 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f380c0aaa10 con 0x7f381c0a0bb0 2024-04-26T04:26:59.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.789+0000 7f3812ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:26:59.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.789+0000 7f3812ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:26:59.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.789+0000 7f382088d640 4 set_mon_vals no callback set 2024-04-26T04:26:59.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.789+0000 7f3812ffd640 10 monclient: dump: 2024-04-26T04:26:59.801 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:26:59.801 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:26:59.801 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:59.801 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:26:59.801 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:26:59.801 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:26:59.801 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:26:59.801 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:26:59.801 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:26:59.801 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:26:59.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.789+0000 7f38242f4640 10 monclient: _renew_subs 2024-04-26T04:26:59.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.789+0000 7f38242f4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:26:59.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.789+0000 7f38242f4640 1 -- 172.21.15.63:0/1013191647 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f381c09cc50 con 0x7f381c0a0bb0 2024-04-26T04:26:59.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.789+0000 7f3812ffd640 1 -- 172.21.15.63:0/1013191647 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f380c0ab4c0 con 0x7f381c0a0bb0 2024-04-26T04:26:59.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.789+0000 7f38242f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:26:59.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.789+0000 7f38242f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:26:59.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.789+0000 7f38242f4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:26:59.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.789+0000 7f38242f4640 1 -- 172.21.15.63:0/1013191647 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f381c062e10 con 0x7f381c0a0bb0 2024-04-26T04:26:59.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.789+0000 7f3812ffd640 1 --2- 172.21.15.63:0/1013191647 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f3800035f10 0x7f38000383d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:26:59.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.789+0000 7f3812ffd640 1 -- 172.21.15.63:0/1013191647 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(323..323 src has 1..323) ==== 4346+0+0 (secure 0 0 0) 0x7f380c0e2b80 con 0x7f381c0a0bb0 2024-04-26T04:26:59.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.789+0000 7f3822090640 1 --2- 172.21.15.63:0/1013191647 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f3800035f10 0x7f38000383d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:26:59.802 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.789+0000 7f3822090640 10 monclient: get_auth_request con 0x7f3800035f10 auth_method 0 2024-04-26T04:26:59.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.793+0000 7f3822090640 1 --2- 172.21.15.63:0/1013191647 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f3800035f10 0x7f38000383d0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f381800ab60 tx=0x7f3818008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:26:59.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.793+0000 7f3812ffd640 1 -- 172.21.15.63:0/1013191647 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f381c09dde0 con 0x7f381c0a0bb0 2024-04-26T04:26:59.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.793+0000 7f3812ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:26:59.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.793+0000 7f3812ffd640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:26:59.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.949+0000 7f38242f4640 1 -- 172.21.15.63:0/1013191647 --> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7f381c072a00 con 0x7f3800035f10 2024-04-26T04:26:59.951 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.949+0000 7f3812ffd640 1 -- 172.21.15.63:0/1013191647 <== mgr.4105 v2:172.21.15.186:6824/2583706639 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f381c072a00 con 0x7f3800035f10 2024-04-26T04:26:59.951 INFO:teuthology.orchestra.run.smithi063.stderr:instructed osd(s) 1 to deep-scrub 2024-04-26T04:26:59.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.953+0000 7f38242f4640 1 -- 172.21.15.63:0/1013191647 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f3800035f10 msgr2=0x7f38000383d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:59.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.953+0000 7f38242f4640 1 --2- 172.21.15.63:0/1013191647 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f3800035f10 0x7f38000383d0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f381800ab60 tx=0x7f3818008040 comp rx=0 tx=0).stop 2024-04-26T04:26:59.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.953+0000 7f38242f4640 10 monclient: shutdown 2024-04-26T04:26:59.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.953+0000 7f38242f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:26:59.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.953+0000 7f38242f4640 1 -- 172.21.15.63:0/1013191647 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f381c0a0bb0 msgr2=0x7f381c0982f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:26:59.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.953+0000 7f38242f4640 1 --2- 172.21.15.63:0/1013191647 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f381c0a0bb0 0x7f381c0982f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f380c01c9e0 tx=0x7f380c018ee0 comp rx=0 tx=0).stop 2024-04-26T04:26:59.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.953+0000 7f38242f4640 1 -- 172.21.15.63:0/1013191647 shutdown_connections 2024-04-26T04:26:59.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.953+0000 7f38242f4640 1 --2- 172.21.15.63:0/1013191647 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f3800035f10 0x7f38000383d0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:59.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.953+0000 7f38242f4640 1 --2- 172.21.15.63:0/1013191647 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f381c0a14d0 0x7f381c098830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:59.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.953+0000 7f38242f4640 1 --2- 172.21.15.63:0/1013191647 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f381c0a0bb0 0x7f381c0982f0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:59.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.953+0000 7f38242f4640 1 --2- 172.21.15.63:0/1013191647 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f381c09b3b0 0x7f381c1ccc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:26:59.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.953+0000 7f38242f4640 1 -- 172.21.15.63:0/1013191647 >> 172.21.15.63:0/1013191647 conn(0x7f381c0a1e70 msgr2=0x7f381c06ead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:26:59.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.953+0000 7f38242f4640 1 -- 172.21.15.63:0/1013191647 shutdown_connections 2024-04-26T04:26:59.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:26:59.953+0000 7f38242f4640 1 -- 172.21.15.63:0/1013191647 wait complete. 2024-04-26T04:26:59.969 INFO:tasks.ceph:Scrubbing osd.2 2024-04-26T04:26:59.969 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-04-26T04:27:00.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.093+0000 7ff27579e640 10 monclient: get_monmap_and_config 2024-04-26T04:27:00.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.093+0000 7ff27579e640 10 monclient: build_initial_monmap 2024-04-26T04:27:00.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.093+0000 7ff27579e640 10 monclient: monmap: 2024-04-26T04:27:00.095 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:27:00.095 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:27:00.095 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:27:00.095167+0000 2024-04-26T04:27:00.095 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:27:00.095167+0000 2024-04-26T04:27:00.095 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:27:00.095 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:27:00.095 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:27:00.095 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:27:00.095 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:27:00.095 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:27:00.095 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:27:00.096 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:00.098 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.093+0000 7ff27579e640 1 Processor -- start 2024-04-26T04:27:00.098 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.093+0000 7ff27579e640 1 -- start start 2024-04-26T04:27:00.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.093+0000 7ff27579e640 10 monclient: init 2024-04-26T04:27:00.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.093+0000 7ff27579e640 10 monclient: _reopen_session rank -1 2024-04-26T04:27:00.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.093+0000 7ff27579e640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-04-26T04:27:00.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.093+0000 7ff27579e640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff270137210 0x7ff2700a3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:00.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.093+0000 7ff27579e640 10 monclient(hunting): picked mon.noname-b con 0x7ff270137210 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:27:00.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.093+0000 7ff27579e640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7ff2701375f0 0x7ff2700a4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:00.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.093+0000 7ff27579e640 10 monclient(hunting): picked mon.noname-a con 0x7ff2701375f0 addr v2:172.21.15.63:3300/0 2024-04-26T04:27:00.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.093+0000 7ff27579e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff2700a5500 0x7ff2700a5990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:00.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.093+0000 7ff27579e640 10 monclient(hunting): picked mon.noname-c con 0x7ff2700a5500 addr v2:172.21.15.186:3300/0 2024-04-26T04:27:00.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.093+0000 7ff27579e640 10 monclient(hunting): start opening mon connection 2024-04-26T04:27:00.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.093+0000 7ff27579e640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7ff270136bc0 con 0x7ff2701375f0 2024-04-26T04:27:00.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.093+0000 7ff27579e640 10 monclient(hunting): start opening mon connection 2024-04-26T04:27:00.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.093+0000 7ff27579e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7ff270073730 con 0x7ff2700a5500 2024-04-26T04:27:00.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.093+0000 7ff27579e640 10 monclient(hunting): start opening mon connection 2024-04-26T04:27:00.100 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.093+0000 7ff27579e640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7ff2700a0e90 con 0x7ff270137210 2024-04-26T04:27:00.100 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.093+0000 7ff27579e640 10 monclient(hunting): _renew_subs 2024-04-26T04:27:00.100 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.093+0000 7ff27579e640 10 monclient(hunting): authenticate will time out at 1451.803228s 2024-04-26T04:27:00.100 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.093+0000 7ff26effd640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff270137210 0x7ff2700a3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:00.100 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.093+0000 7ff26e7fc640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7ff2701375f0 0x7ff2700a4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:00.100 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.093+0000 7ff26effd640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff270137210 0x7ff2700a3640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:39842/0 (socket says 172.21.15.63:39842) 2024-04-26T04:27:00.100 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.093+0000 7ff26effd640 1 -- 172.21.15.63:0/1578836609 learned_addr learned my addr 172.21.15.63:0/1578836609 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:27:00.100 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.093+0000 7ff26effd640 10 monclient(hunting): get_auth_request con 0x7ff270137210 auth_method 0 2024-04-26T04:27:00.100 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.093+0000 7ff26effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:27:00.100 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.093+0000 7ff26effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:27:00.100 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.093+0000 7ff26effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:27:00.100 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.093+0000 7ff26e7fc640 10 monclient(hunting): get_auth_request con 0x7ff2701375f0 auth_method 0 2024-04-26T04:27:00.100 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.093+0000 7ff26e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:27:00.100 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.093+0000 7ff26e7fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:27:00.100 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.093+0000 7ff26e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:27:00.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.093+0000 7ff26effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:27:00.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.093+0000 7ff26effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:27:00.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.093+0000 7ff26effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:27:00.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.093+0000 7ff26e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:27:00.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.093+0000 7ff26e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:27:00.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.093+0000 7ff26e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:27:00.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.093+0000 7ff26f7fe640 1 --2- 172.21.15.63:0/1578836609 >> v2:172.21.15.186:3300/0 conn(0x7ff2700a5500 0x7ff2700a5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:00.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.093+0000 7ff26f7fe640 10 monclient(hunting): get_auth_request con 0x7ff2700a5500 auth_method 0 2024-04-26T04:27:00.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.093+0000 7ff26f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:27:00.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.093+0000 7ff26f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:27:00.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.093+0000 7ff26f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:27:00.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.093+0000 7ff26e7fc640 10 monclient(hunting): handle_auth_done global_id 44178 payload 274 2024-04-26T04:27:00.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.093+0000 7ff26e7fc640 1 -- 172.21.15.63:0/1578836609 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff270137210 msgr2=0x7ff2700a3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:27:00.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.093+0000 7ff26e7fc640 1 --2- 172.21.15.63:0/1578836609 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff270137210 0x7ff2700a3640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:00.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.093+0000 7ff26e7fc640 1 -- 172.21.15.63:0/1578836609 >> v2:172.21.15.186:3300/0 conn(0x7ff2700a5500 msgr2=0x7ff2700a5990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:27:00.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.093+0000 7ff26e7fc640 1 --2- 172.21.15.63:0/1578836609 >> v2:172.21.15.186:3300/0 conn(0x7ff2700a5500 0x7ff2700a5990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:00.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.093+0000 7ff26e7fc640 10 monclient: _finish_hunting 0 2024-04-26T04:27:00.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.093+0000 7ff26e7fc640 1 monclient: found mon.noname-a 2024-04-26T04:27:00.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.093+0000 7ff26e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:27:00.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.093+0000 7ff26e7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:27:00.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.093+0000 7ff26e7fc640 1 -- 172.21.15.63:0/1578836609 --> v2:172.21.15.63:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2700a1510 con 0x7ff2701375f0 2024-04-26T04:27:00.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.093+0000 7ff26effd640 1 --2- 172.21.15.63:0/1578836609 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff270137210 0x7ff2700a3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T04:27:00.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.097+0000 7ff26e7fc640 1 -- 172.21.15.63:0/1578836609 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff2701375f0 msgr2=0x7ff2700a4fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:27:00.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.097+0000 7ff26e7fc640 1 --2- 172.21.15.63:0/1578836609 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff2701375f0 0x7ff2700a4fc0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff270134760 tx=0x7ff26401c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3223c32f4b3870f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:27:00.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff26dffb640 1 -- 172.21.15.63:0/1578836609 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2640b0070 con 0x7ff2701375f0 2024-04-26T04:27:00.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff26dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:27:00.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff26dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:27:00.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff26dffb640 10 monclient: dump: 2024-04-26T04:27:00.106 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:27:00.106 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:27:00.106 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:00.106 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:00.107 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:27:00.107 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:27:00.107 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:27:00.107 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:27:00.107 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:27:00.107 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:00.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff26dffb640 10 monclient: _finish_auth 0 2024-04-26T04:27:00.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff26dffb640 10 monclient: _check_auth_tickets 2024-04-26T04:27:00.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff26dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:27:00.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff26dffb640 1 -- 172.21.15.63:0/1578836609 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2640ab600 con 0x7ff2701375f0 2024-04-26T04:27:00.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff26dffb640 10 monclient: handle_config config(0 keys) 2024-04-26T04:27:00.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff26dffb640 1 -- 172.21.15.63:0/1578836609 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2640ab9d0 con 0x7ff2701375f0 2024-04-26T04:27:00.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff27579e640 5 monclient: authenticate success, global_id 44178 2024-04-26T04:27:00.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff27579e640 10 monclient: get_monmap_and_config success 2024-04-26T04:27:00.107 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff27579e640 4 set_mon_vals no callback set 2024-04-26T04:27:00.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff26dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:27:00.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff26dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:27:00.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff26dffb640 10 monclient: dump: 2024-04-26T04:27:00.108 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:27:00.108 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:27:00.108 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:00.108 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:00.108 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:27:00.108 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:27:00.108 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:27:00.108 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:27:00.108 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:27:00.108 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:00.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff27579e640 10 monclient: shutdown 2024-04-26T04:27:00.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff27579e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:27:00.108 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff27579e640 1 -- 172.21.15.63:0/1578836609 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff2701375f0 msgr2=0x7ff2700a4fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:27:00.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff27579e640 1 --2- 172.21.15.63:0/1578836609 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff2701375f0 0x7ff2700a4fc0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff270134760 tx=0x7ff26401c380 comp rx=0 tx=0).stop 2024-04-26T04:27:00.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff27579e640 1 -- 172.21.15.63:0/1578836609 shutdown_connections 2024-04-26T04:27:00.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff27579e640 1 --2- 172.21.15.63:0/1578836609 >> v2:172.21.15.186:3300/0 conn(0x7ff2700a5500 0x7ff2700a5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:00.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff27579e640 1 --2- 172.21.15.63:0/1578836609 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff2701375f0 0x7ff2700a4fc0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:00.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff27579e640 1 --2- 172.21.15.63:0/1578836609 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff270137210 0x7ff2700a3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:00.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff27579e640 1 -- 172.21.15.63:0/1578836609 >> 172.21.15.63:0/1578836609 conn(0x7ff2700a60b0 msgr2=0x7ff2700a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:27:00.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff27579e640 1 -- 172.21.15.63:0/1578836609 shutdown_connections 2024-04-26T04:27:00.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff27579e640 1 -- 172.21.15.63:0/1578836609 wait complete. 2024-04-26T04:27:00.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff27579e640 10 monclient: build_initial_monmap 2024-04-26T04:27:00.109 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff27579e640 10 monclient: monmap: 2024-04-26T04:27:00.109 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:27:00.109 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:27:00.109 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:27:00.109 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:27:00.109 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:27:00.110 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:27:00.110 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:27:00.110 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:27:00.110 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:27:00.110 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:00.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff27579e640 1 Processor -- start 2024-04-26T04:27:00.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff27579e640 1 -- start start 2024-04-26T04:27:00.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff27579e640 10 monclient: init 2024-04-26T04:27:00.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff27579e640 10 monclient: _reopen_session rank -1 2024-04-26T04:27:00.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff27579e640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T04:27:00.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff27579e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff2700a5500 0x7ff2701ceb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:00.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff27579e640 10 monclient(hunting): picked mon.noname-a con 0x7ff2700a5500 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:27:00.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff27579e640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff270137210 0x7ff270134240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:00.110 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff27579e640 10 monclient(hunting): picked mon.noname-c con 0x7ff270137210 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:27:00.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff27579e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff2701375f0 0x7ff270134780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:00.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff27579e640 10 monclient(hunting): picked mon.noname-b con 0x7ff2701375f0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:27:00.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff27579e640 10 monclient(hunting): start opening mon connection 2024-04-26T04:27:00.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff27579e640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff2700a6fb0 con 0x7ff2700a5500 2024-04-26T04:27:00.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff27579e640 10 monclient(hunting): start opening mon connection 2024-04-26T04:27:00.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff27579e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff2700a6e30 con 0x7ff2701375f0 2024-04-26T04:27:00.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff27579e640 10 monclient(hunting): start opening mon connection 2024-04-26T04:27:00.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff27579e640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7ff270140710 con 0x7ff270137210 2024-04-26T04:27:00.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff27579e640 10 monclient(hunting): _renew_subs 2024-04-26T04:27:00.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff27579e640 10 monclient(hunting): authenticate will time out at 1451.810862s 2024-04-26T04:27:00.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff26effd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff2700a5500 0x7ff2701ceb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:00.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff26e7fc640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff270137210 0x7ff270134240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:00.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff26effd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff2700a5500 0x7ff2701ceb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:47528/0 (socket says 172.21.15.63:47528) 2024-04-26T04:27:00.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff26effd640 1 -- 172.21.15.63:0/2417096039 learned_addr learned my addr 172.21.15.63:0/2417096039 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:27:00.111 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff26effd640 10 monclient(hunting): get_auth_request con 0x7ff2700a5500 auth_method 0 2024-04-26T04:27:00.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff26e7fc640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff270137210 0x7ff270134240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:39848/0 (socket says 172.21.15.63:39848) 2024-04-26T04:27:00.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff26effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:27:00.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff26effd640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:27:00.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff26effd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:27:00.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff26e7fc640 10 monclient(hunting): get_auth_request con 0x7ff270137210 auth_method 0 2024-04-26T04:27:00.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff26e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:27:00.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff26e7fc640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:27:00.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff26e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:27:00.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff26effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:27:00.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff26effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:27:00.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff26effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:27:00.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff26e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:27:00.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff26e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:27:00.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff26e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:27:00.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff26f7fe640 1 --2- 172.21.15.63:0/2417096039 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff2701375f0 0x7ff270134780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:00.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff26f7fe640 10 monclient(hunting): get_auth_request con 0x7ff2701375f0 auth_method 0 2024-04-26T04:27:00.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff26f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:27:00.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff26f7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:27:00.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff26f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:27:00.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff26effd640 10 monclient(hunting): handle_auth_done global_id 44181 payload 824 2024-04-26T04:27:00.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff26effd640 1 -- 172.21.15.63:0/2417096039 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff270137210 msgr2=0x7ff270134240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:27:00.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff26effd640 1 --2- 172.21.15.63:0/2417096039 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff270137210 0x7ff270134240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:00.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff26effd640 1 -- 172.21.15.63:0/2417096039 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff2701375f0 msgr2=0x7ff270134780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:27:00.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff26effd640 1 --2- 172.21.15.63:0/2417096039 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff2701375f0 0x7ff270134780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:00.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff26effd640 10 monclient: _finish_hunting 0 2024-04-26T04:27:00.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff26effd640 1 monclient: found mon.noname-a 2024-04-26T04:27:00.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff26effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:27:00.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff26effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:27:00.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff26effd640 1 -- 172.21.15.63:0/2417096039 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2700a1740 con 0x7ff2700a5500 2024-04-26T04:27:00.114 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.101+0000 7ff26e7fc640 1 --2- 172.21.15.63:0/2417096039 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff270137210 0x7ff270134240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T04:27:00.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.105+0000 7ff26effd640 1 -- 172.21.15.63:0/2417096039 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff2700a5500 msgr2=0x7ff2701ceb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:27:00.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.105+0000 7ff26effd640 1 --2- 172.21.15.63:0/2417096039 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff2700a5500 0x7ff2701ceb80 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff26000e9d0 tx=0x7ff26000ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:27:00.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.105+0000 7ff2577fe640 1 -- 172.21.15.63:0/2417096039 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff260017070 con 0x7ff2700a5500 2024-04-26T04:27:00.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.105+0000 7ff2577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:27:00.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.105+0000 7ff2577fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:27:00.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.105+0000 7ff2577fe640 10 monclient: dump: 2024-04-26T04:27:00.116 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:27:00.117 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:27:00.117 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:00.117 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:00.117 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:27:00.117 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:27:00.117 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:27:00.117 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:27:00.117 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:27:00.117 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:00.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.105+0000 7ff2577fe640 10 monclient: _finish_auth 0 2024-04-26T04:27:00.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.105+0000 7ff2577fe640 10 monclient: _check_auth_tickets 2024-04-26T04:27:00.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.105+0000 7ff2577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:27:00.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.105+0000 7ff2577fe640 1 -- 172.21.15.63:0/2417096039 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff26000cc70 con 0x7ff2700a5500 2024-04-26T04:27:00.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.105+0000 7ff27579e640 5 monclient: authenticate success, global_id 44181 2024-04-26T04:27:00.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.105+0000 7ff27579e640 10 monclient: _renew_subs 2024-04-26T04:27:00.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.105+0000 7ff27579e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:27:00.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.105+0000 7ff27579e640 1 -- 172.21.15.63:0/2417096039 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2701363d0 con 0x7ff2700a5500 2024-04-26T04:27:00.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.105+0000 7ff2577fe640 10 monclient: handle_config config(0 keys) 2024-04-26T04:27:00.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.105+0000 7ff2577fe640 1 -- 172.21.15.63:0/2417096039 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff260013670 con 0x7ff2700a5500 2024-04-26T04:27:00.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.105+0000 7ff2577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:27:00.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.105+0000 7ff2577fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:27:00.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.105+0000 7ff2577fe640 10 monclient: dump: 2024-04-26T04:27:00.118 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:27:00.118 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:27:00.118 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:00.118 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:00.118 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:27:00.118 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:27:00.118 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:27:00.119 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:27:00.119 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:27:00.119 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:00.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.105+0000 7ff26d7fa640 4 set_mon_vals no callback set 2024-04-26T04:27:00.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.105+0000 7ff27579e640 10 monclient: _renew_subs 2024-04-26T04:27:00.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.105+0000 7ff27579e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:27:00.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.105+0000 7ff27579e640 1 -- 172.21.15.63:0/2417096039 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff27012fa00 con 0x7ff2700a5500 2024-04-26T04:27:00.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.105+0000 7ff27579e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:27:00.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.105+0000 7ff27579e640 1 -- 172.21.15.63:0/2417096039 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff270067fb0 con 0x7ff2700a5500 2024-04-26T04:27:00.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.105+0000 7ff2577fe640 1 -- 172.21.15.63:0/2417096039 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7ff26001a070 con 0x7ff2700a5500 2024-04-26T04:27:00.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.105+0000 7ff2577fe640 1 --2- 172.21.15.63:0/2417096039 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7ff25c035f10 0x7ff25c0383d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:00.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.105+0000 7ff2577fe640 1 -- 172.21.15.63:0/2417096039 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(323..323 src has 1..323) ==== 4346+0+0 (secure 0 0 0) 0x7ff260048c10 con 0x7ff2700a5500 2024-04-26T04:27:00.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.105+0000 7ff2577fe640 1 --2- 172.21.15.63:0/2417096039 >> [v2:172.21.15.63:6808/413829082,v1:172.21.15.63:6809/413829082] conn(0x7ff25c03be00 0x7ff25c03e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:00.119 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.105+0000 7ff2577fe640 1 -- 172.21.15.63:0/2417096039 --> [v2:172.21.15.63:6808/413829082,v1:172.21.15.63:6809/413829082] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff26000be50 con 0x7ff25c03be00 2024-04-26T04:27:00.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.105+0000 7ff2577fe640 1 -- 172.21.15.63:0/2417096039 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_get_version_reply(handle=1 version=323) ==== 24+0+0 (secure 0 0 0) 0x7ff26004d2a0 con 0x7ff2700a5500 2024-04-26T04:27:00.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.105+0000 7ff2577fe640 10 monclient: handle_get_version_reply finishing 1 version 323 2024-04-26T04:27:00.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.105+0000 7ff26f7fe640 1 --2- 172.21.15.63:0/2417096039 >> [v2:172.21.15.63:6808/413829082,v1:172.21.15.63:6809/413829082] conn(0x7ff25c03be00 0x7ff25c03e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:00.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.105+0000 7ff26f7fe640 10 monclient: get_auth_request con 0x7ff25c03be00 auth_method 0 2024-04-26T04:27:00.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.105+0000 7ff26e7fc640 1 --2- 172.21.15.63:0/2417096039 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7ff25c035f10 0x7ff25c0383d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:00.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.105+0000 7ff26e7fc640 10 monclient: get_auth_request con 0x7ff25c035f10 auth_method 0 2024-04-26T04:27:00.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.109+0000 7ff26e7fc640 1 --2- 172.21.15.63:0/2417096039 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7ff25c035f10 0x7ff25c0383d0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7ff27006be80 tx=0x7ff2640a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:27:00.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.109+0000 7ff26f7fe640 1 --2- 172.21.15.63:0/2417096039 >> [v2:172.21.15.63:6808/413829082,v1:172.21.15.63:6809/413829082] conn(0x7ff25c03be00 0x7ff25c03e220 crc :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:27:00.120 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.113+0000 7ff2577fe640 1 -- 172.21.15.63:0/2417096039 <== osd.2 v2:172.21.15.63:6808/413829082 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff26000be50 con 0x7ff25c03be00 2024-04-26T04:27:00.126 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.125+0000 7ff27579e640 1 -- 172.21.15.63:0/2417096039 --> [v2:172.21.15.63:6808/413829082,v1:172.21.15.63:6809/413829082] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7ff270062e10 con 0x7ff25c03be00 2024-04-26T04:27:00.127 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.125+0000 7ff2577fe640 1 -- 172.21.15.63:0/2417096039 <== osd.2 v2:172.21.15.63:6808/413829082 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7ff270062e10 con 0x7ff25c03be00 2024-04-26T04:27:00.127 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-26T04:27:00.127 INFO:teuthology.orchestra.run.smithi063.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-26T04:27:00.127 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-26T04:27:00.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.125+0000 7ff27579e640 1 -- 172.21.15.63:0/2417096039 >> [v2:172.21.15.63:6808/413829082,v1:172.21.15.63:6809/413829082] conn(0x7ff25c03be00 msgr2=0x7ff25c03e220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:27:00.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.125+0000 7ff27579e640 1 --2- 172.21.15.63:0/2417096039 >> [v2:172.21.15.63:6808/413829082,v1:172.21.15.63:6809/413829082] conn(0x7ff25c03be00 0x7ff25c03e220 crc :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:00.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.125+0000 7ff27579e640 1 -- 172.21.15.63:0/2417096039 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7ff25c035f10 msgr2=0x7ff25c0383d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:27:00.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.125+0000 7ff27579e640 1 --2- 172.21.15.63:0/2417096039 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7ff25c035f10 0x7ff25c0383d0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7ff27006be80 tx=0x7ff2640a6000 comp rx=0 tx=0).stop 2024-04-26T04:27:00.129 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.125+0000 7ff27579e640 10 monclient: shutdown 2024-04-26T04:27:00.129 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.125+0000 7ff27579e640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:27:00.129 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.125+0000 7ff27579e640 1 -- 172.21.15.63:0/2417096039 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff2700a5500 msgr2=0x7ff2701ceb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:27:00.129 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.125+0000 7ff27579e640 1 --2- 172.21.15.63:0/2417096039 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff2700a5500 0x7ff2701ceb80 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff26000e9d0 tx=0x7ff26000ee90 comp rx=0 tx=0).stop 2024-04-26T04:27:00.129 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.125+0000 7ff26f7fe640 1 -- 172.21.15.63:0/2417096039 reap_dead start 2024-04-26T04:27:00.129 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.125+0000 7ff27579e640 1 -- 172.21.15.63:0/2417096039 shutdown_connections 2024-04-26T04:27:00.129 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.125+0000 7ff27579e640 1 -- 172.21.15.63:0/2417096039 >> 172.21.15.63:0/2417096039 conn(0x7ff2700a60b0 msgr2=0x7ff2700a8240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:27:00.129 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.125+0000 7ff27579e640 1 -- 172.21.15.63:0/2417096039 shutdown_connections 2024-04-26T04:27:00.129 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.125+0000 7ff27579e640 1 -- 172.21.15.63:0/2417096039 wait complete. 2024-04-26T04:27:00.142 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-04-26T04:27:00.251 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.249+0000 7f6cd9d68640 10 monclient: get_monmap_and_config 2024-04-26T04:27:00.251 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.249+0000 7f6cd9d68640 10 monclient: build_initial_monmap 2024-04-26T04:27:00.251 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.249+0000 7f6cd9d68640 10 monclient: monmap: 2024-04-26T04:27:00.251 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:27:00.251 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:27:00.251 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:27:00.251112+0000 2024-04-26T04:27:00.251 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:27:00.251112+0000 2024-04-26T04:27:00.252 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:27:00.252 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:27:00.252 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:27:00.252 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:27:00.252 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:27:00.252 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:27:00.252 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:27:00.252 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:00.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.249+0000 7f6cd9d68640 1 Processor -- start 2024-04-26T04:27:00.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.249+0000 7f6cd9d68640 1 -- start start 2024-04-26T04:27:00.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.249+0000 7f6cd9d68640 10 monclient: init 2024-04-26T04:27:00.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.249+0000 7f6cd9d68640 10 monclient: _reopen_session rank -1 2024-04-26T04:27:00.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.249+0000 7f6cd9d68640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-04-26T04:27:00.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.249+0000 7f6cd9d68640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6cd4072ec0 addr v1:172.21.15.186:6789/0 2024-04-26T04:27:00.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.249+0000 7f6cd9d68640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6cd40732a0 addr v1:172.21.15.63:6789/0 2024-04-26T04:27:00.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.249+0000 7f6cd9d68640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f6cd41423c0 0x7f6cd41447b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:00.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.249+0000 7f6cd9d68640 10 monclient(hunting): picked mon.noname-c con 0x7f6cd41423c0 addr v2:172.21.15.186:3300/0 2024-04-26T04:27:00.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.249+0000 7f6cd9d68640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6cd4099fc0 con 0x7f6cd40732a0 2024-04-26T04:27:00.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.249+0000 7f6cd9d68640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6cd409bba0 con 0x7f6cd4072ec0 2024-04-26T04:27:00.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.249+0000 7f6cd9d68640 10 monclient(hunting): start opening mon connection 2024-04-26T04:27:00.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.249+0000 7f6cd9d68640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f6cd409b5e0 con 0x7f6cd41423c0 2024-04-26T04:27:00.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.249+0000 7f6cd9d68640 10 monclient(hunting): _renew_subs 2024-04-26T04:27:00.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.249+0000 7f6cd9d68640 10 monclient(hunting): authenticate will time out at 1451.958357s 2024-04-26T04:27:00.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.249+0000 7f6ccaffd640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f6cd40732a0 0x7f6cd40998b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:37556/0 (socket says 172.21.15.63:37556) 2024-04-26T04:27:00.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.249+0000 7f6ccaffd640 1 -- 172.21.15.63:0/1174785255 learned_addr learned my addr 172.21.15.63:0/1174785255 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-04-26T04:27:00.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.249+0000 7f6cd3fff640 1 --2- 172.21.15.63:0/1174785255 >> v2:172.21.15.186:3300/0 conn(0x7f6cd41423c0 0x7f6cd41447b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:00.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.249+0000 7f6cd3fff640 10 monclient(hunting): get_auth_request con 0x7f6cd41423c0 auth_method 0 2024-04-26T04:27:00.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.249+0000 7f6cd3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:27:00.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.249+0000 7f6cd3fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:27:00.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.249+0000 7f6cd3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:27:00.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.249+0000 7f6cd3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:27:00.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.249+0000 7f6cd3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:27:00.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.249+0000 7f6cd3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:27:00.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6ccaffd640 1 -- 172.21.15.63:0/1174785255 >> v1:172.21.15.63:6789/0 conn(0x7f6cd40732a0 legacy=0x7f6cd40998b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:27:00.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6cd2ffd640 1 -- 172.21.15.63:0/1174785255 <== mon.0 v1:172.21.15.63:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2306280520 0 0) 0x7f6cd4099fc0 con 0x7f6cd40732a0 2024-04-26T04:27:00.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6cd2ffd640 10 monclient(hunting): _negotiate 2024-04-26T04:27:00.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6cd2ffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:27:00.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6cd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:27:00.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6cd2ffd640 10 monclient(hunting): my global_id is 44184 2024-04-26T04:27:00.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6cd2ffd640 1 -- 172.21.15.63:0/1174785255 --> v1:172.21.15.63:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6cc0003530 con 0x7f6cd40732a0 2024-04-26T04:27:00.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6cd2ffd640 1 -- 172.21.15.63:0/1174785255 <== mon.0 v1:172.21.15.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 664460902 0 0) 0x7f6cc0003530 con 0x7f6cd40732a0 2024-04-26T04:27:00.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6cd2ffd640 1 -- 172.21.15.63:0/1174785255 >> v2:172.21.15.186:3300/0 conn(0x7f6cd41423c0 msgr2=0x7f6cd41447b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:27:00.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6cd2ffd640 1 --2- 172.21.15.63:0/1174785255 >> v2:172.21.15.186:3300/0 conn(0x7f6cd41423c0 0x7f6cd41447b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:00.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6cd2ffd640 1 -- 172.21.15.63:0/1174785255 >> v1:172.21.15.186:6789/0 conn(0x7f6cd4072ec0 legacy=0x7f6cd40991a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:27:00.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6cd2ffd640 10 monclient: _finish_hunting 0 2024-04-26T04:27:00.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6cd2ffd640 1 monclient: found mon.noname-a-legacy 2024-04-26T04:27:00.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6cd2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:27:00.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6cd2ffd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.63:6789/0 2024-04-26T04:27:00.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6cd2ffd640 1 -- 172.21.15.63:0/1174785255 --> v1:172.21.15.63:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cd40a18c0 con 0x7f6cd40732a0 2024-04-26T04:27:00.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6cd2ffd640 10 monclient: _finish_auth 0 2024-04-26T04:27:00.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6cd2ffd640 10 monclient: _check_auth_tickets 2024-04-26T04:27:00.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6cd2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:27:00.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6cd9d68640 5 monclient: authenticate success, global_id 44184 2024-04-26T04:27:00.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6cd9d68640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T04:27:00.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6cd2ffd640 1 -- 172.21.15.63:0/1174785255 <== mon.0 v1:172.21.15.63:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2612289768 0 0) 0x7f6cbc0019f0 con 0x7f6cd40732a0 2024-04-26T04:27:00.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6cd2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:27:00.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6cd2ffd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-04-26T04:27:00.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6cd2ffd640 10 monclient: dump: 2024-04-26T04:27:00.260 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:27:00.260 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:27:00.260 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:00.260 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:00.260 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:27:00.260 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:27:00.261 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:27:00.261 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:27:00.261 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:27:00.261 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:00.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6cd2ffd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] but i'm connected to v1:172.21.15.63:6789/0, reconnecting 2024-04-26T04:27:00.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6cd2ffd640 10 monclient: _reopen_session rank -1 2024-04-26T04:27:00.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6cd2ffd640 1 -- 172.21.15.63:0/1174785255 >> v1:172.21.15.63:6789/0 conn(0x7f6cd40732a0 legacy=0x7f6cd40998b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:27:00.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6cd2ffd640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T04:27:00.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6cd2ffd640 1 --2- 172.21.15.63:0/1174785255 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6cc0003aa0 0x7f6cc0003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:00.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6cd2ffd640 10 monclient(hunting): picked mon.a con 0x7f6cc0003aa0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:27:00.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6cd2ffd640 1 --2- 172.21.15.63:0/1174785255 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f6cc0004810 0x7f6cc0004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:00.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6cd2ffd640 10 monclient(hunting): picked mon.c con 0x7f6cc0004810 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:27:00.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6cd2ffd640 1 --2- 172.21.15.63:0/1174785255 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6cc0005320 0x7f6cc000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:00.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6cd2ffd640 10 monclient(hunting): picked mon.b con 0x7f6cc0005320 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:27:00.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6cd2ffd640 10 monclient(hunting): start opening mon connection 2024-04-26T04:27:00.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6cd2ffd640 1 -- 172.21.15.63:0/1174785255 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f6cd409b5e0 con 0x7f6cc0003aa0 2024-04-26T04:27:00.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6cd2ffd640 10 monclient(hunting): start opening mon connection 2024-04-26T04:27:00.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6cd2ffd640 1 -- 172.21.15.63:0/1174785255 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6cc000c260 con 0x7f6cc0005320 2024-04-26T04:27:00.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6cd2ffd640 10 monclient(hunting): start opening mon connection 2024-04-26T04:27:00.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6cd37fe640 1 --2- 172.21.15.63:0/1174785255 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6cc0003aa0 0x7f6cc0003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:00.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6cd2ffd640 1 -- 172.21.15.63:0/1174785255 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f6cc000c460 con 0x7f6cc0004810 2024-04-26T04:27:00.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6cd2ffd640 10 monclient(hunting): _renew_subs 2024-04-26T04:27:00.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6cd2ffd640 10 monclient(hunting): _finish_auth 0 2024-04-26T04:27:00.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6cd9d68640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-04-26T04:27:00.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6cd37fe640 10 monclient(hunting): get_auth_request con 0x7f6cc0003aa0 auth_method 0 2024-04-26T04:27:00.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6cd37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:27:00.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6cd37fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:27:00.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6cd37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:27:00.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6ccaffd640 1 --2- 172.21.15.63:0/1174785255 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f6cc0004810 0x7f6cc0004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:00.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6cd37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:27:00.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6cd37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:27:00.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6cd37fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-04-26T04:27:00.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6ccaffd640 10 monclient(hunting): get_auth_request con 0x7f6cc0004810 auth_method 0 2024-04-26T04:27:00.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6ccaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:27:00.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6ccaffd640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:27:00.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6ccaffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-04-26T04:27:00.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6cd37fe640 10 monclient(hunting): handle_auth_done global_id 44184 payload 293 2024-04-26T04:27:00.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6cd37fe640 1 -- 172.21.15.63:0/1174785255 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f6cc0004810 msgr2=0x7f6cc0004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:27:00.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6cd37fe640 1 --2- 172.21.15.63:0/1174785255 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f6cc0004810 0x7f6cc0004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:00.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6cd37fe640 1 -- 172.21.15.63:0/1174785255 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6cc0005320 msgr2=0x7f6cc000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:27:00.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6cd37fe640 1 --2- 172.21.15.63:0/1174785255 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6cc0005320 0x7f6cc000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:00.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6cd37fe640 10 monclient: _finish_hunting 0 2024-04-26T04:27:00.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6cd37fe640 1 monclient: found mon.a 2024-04-26T04:27:00.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6cd37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:27:00.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6cd3fff640 1 -- 172.21.15.63:0/1174785255 reap_dead start 2024-04-26T04:27:00.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6cd37fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:27:00.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6cd37fe640 1 -- 172.21.15.63:0/1174785255 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cc0003530 con 0x7f6cc0003aa0 2024-04-26T04:27:00.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6cd37fe640 10 monclient: _finish_auth 0 2024-04-26T04:27:00.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6cd37fe640 10 monclient: _check_auth_tickets 2024-04-26T04:27:00.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6cd37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:27:00.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.253+0000 7f6ccaffd640 1 --2- 172.21.15.63:0/1174785255 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f6cc0004810 0x7f6cc0004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T04:27:00.269 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.257+0000 7f6cd37fe640 1 -- 172.21.15.63:0/1174785255 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6cc0003aa0 msgr2=0x7f6cc0003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:27:00.269 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.257+0000 7f6cd37fe640 1 --2- 172.21.15.63:0/1174785255 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6cc0003aa0 0x7f6cc0003ea0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6cc4003db0 tx=0x7f6cc401c2b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1afeed934143847b server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:27:00.269 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.257+0000 7f6cd2ffd640 1 -- 172.21.15.63:0/1174785255 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cc40b0070 con 0x7f6cc0003aa0 2024-04-26T04:27:00.269 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.257+0000 7f6cd2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:27:00.270 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.257+0000 7f6cd2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:27:00.270 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.257+0000 7f6cd2ffd640 10 monclient: dump: 2024-04-26T04:27:00.270 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:27:00.270 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:27:00.270 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:00.270 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:00.270 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:27:00.270 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:27:00.270 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:27:00.270 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:27:00.270 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:27:00.270 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:00.270 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.257+0000 7f6cd9d68640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T04:27:00.270 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.257+0000 7f6cd2ffd640 1 -- 172.21.15.63:0/1174785255 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cc40ab620 con 0x7f6cc0003aa0 2024-04-26T04:27:00.270 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.257+0000 7f6cd2ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T04:27:00.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.257+0000 7f6cd9d68640 10 monclient: get_monmap_and_config success 2024-04-26T04:27:00.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.257+0000 7f6cd9d68640 4 set_mon_vals no callback set 2024-04-26T04:27:00.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.257+0000 7f6cd2ffd640 1 -- 172.21.15.63:0/1174785255 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cc40039b0 con 0x7f6cc0003aa0 2024-04-26T04:27:00.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.257+0000 7f6cd2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:27:00.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.257+0000 7f6cd2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:27:00.271 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.257+0000 7f6cd2ffd640 10 monclient: dump: 2024-04-26T04:27:00.271 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:27:00.271 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:27:00.271 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:00.271 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:00.271 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:27:00.271 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:27:00.271 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:27:00.271 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:27:00.271 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:27:00.272 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:00.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.257+0000 7f6cd9d68640 10 monclient: shutdown 2024-04-26T04:27:00.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.257+0000 7f6cd9d68640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:27:00.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.257+0000 7f6cd9d68640 1 -- 172.21.15.63:0/1174785255 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6cc0003aa0 msgr2=0x7f6cc0003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:27:00.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.257+0000 7f6cd9d68640 1 --2- 172.21.15.63:0/1174785255 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6cc0003aa0 0x7f6cc0003ea0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6cc4003db0 tx=0x7f6cc401c2b0 comp rx=0 tx=0).stop 2024-04-26T04:27:00.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.257+0000 7f6cd9d68640 1 -- 172.21.15.63:0/1174785255 shutdown_connections 2024-04-26T04:27:00.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.257+0000 7f6cd9d68640 1 --2- 172.21.15.63:0/1174785255 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6cc0003aa0 0x7f6cc0003ea0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:00.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.257+0000 7f6cd9d68640 1 -- 172.21.15.63:0/1174785255 >> 172.21.15.63:0/1174785255 conn(0x7f6cd40a1e70 msgr2=0x7f6cd40a2280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:27:00.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.257+0000 7f6cd9d68640 1 -- 172.21.15.63:0/1174785255 shutdown_connections 2024-04-26T04:27:00.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.257+0000 7f6cd9d68640 1 -- 172.21.15.63:0/1174785255 wait complete. 2024-04-26T04:27:00.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.257+0000 7f6cd9d68640 10 monclient: build_initial_monmap 2024-04-26T04:27:00.272 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.257+0000 7f6cd9d68640 10 monclient: monmap: 2024-04-26T04:27:00.272 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:27:00.272 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:27:00.272 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:27:00.273 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:27:00.273 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:27:00.273 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:27:00.273 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:27:00.273 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:27:00.273 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:27:00.273 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:00.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.257+0000 7f6cd9d68640 1 Processor -- start 2024-04-26T04:27:00.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.257+0000 7f6cd9d68640 1 -- start start 2024-04-26T04:27:00.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.257+0000 7f6cd9d68640 10 monclient: init 2024-04-26T04:27:00.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.257+0000 7f6cd9d68640 10 monclient: _reopen_session rank -1 2024-04-26T04:27:00.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.257+0000 7f6cd9d68640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T04:27:00.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.257+0000 7f6cd9d68640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f6cc0003aa0 0x7f6cd4096060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:00.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.257+0000 7f6cd9d68640 10 monclient(hunting): picked mon.noname-c con 0x7f6cc0003aa0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:27:00.273 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.257+0000 7f6cd9d68640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6cd40965a0 0x7f6cd40969a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:00.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.257+0000 7f6cd9d68640 10 monclient(hunting): picked mon.noname-b con 0x7f6cd40965a0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:27:00.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.257+0000 7f6cd9d68640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6cd4096ee0 0x7f6cd41d95a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:00.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.257+0000 7f6cd9d68640 10 monclient(hunting): picked mon.noname-a con 0x7f6cd4096ee0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:27:00.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.257+0000 7f6cd9d68640 10 monclient(hunting): start opening mon connection 2024-04-26T04:27:00.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.257+0000 7f6cd9d68640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f6cd4073800 con 0x7f6cd4096ee0 2024-04-26T04:27:00.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.257+0000 7f6cd9d68640 10 monclient(hunting): start opening mon connection 2024-04-26T04:27:00.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.257+0000 7f6cd9d68640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6cd4073680 con 0x7f6cd40965a0 2024-04-26T04:27:00.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.257+0000 7f6cd9d68640 10 monclient(hunting): start opening mon connection 2024-04-26T04:27:00.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.257+0000 7f6cd9d68640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f6cd40a0e10 con 0x7f6cc0003aa0 2024-04-26T04:27:00.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.257+0000 7f6cd9d68640 10 monclient(hunting): _renew_subs 2024-04-26T04:27:00.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.261+0000 7f6cd9d68640 10 monclient(hunting): authenticate will time out at 1451.968010s 2024-04-26T04:27:00.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.261+0000 7f6cd37fe640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f6cc0003aa0 0x7f6cd4096060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:00.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.261+0000 7f6cd3fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6cd4096ee0 0x7f6cd41d95a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:00.274 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.261+0000 7f6cd37fe640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f6cc0003aa0 0x7f6cd4096060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:39860/0 (socket says 172.21.15.63:39860) 2024-04-26T04:27:00.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.261+0000 7f6cd37fe640 1 -- 172.21.15.63:0/3298051190 learned_addr learned my addr 172.21.15.63:0/3298051190 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:27:00.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.261+0000 7f6cd37fe640 10 monclient(hunting): get_auth_request con 0x7f6cc0003aa0 auth_method 0 2024-04-26T04:27:00.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.261+0000 7f6cd37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:27:00.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.261+0000 7f6cd37fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:27:00.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.261+0000 7f6cd37fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:27:00.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.261+0000 7f6cd3fff640 10 monclient(hunting): get_auth_request con 0x7f6cd4096ee0 auth_method 0 2024-04-26T04:27:00.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.261+0000 7f6cd3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:27:00.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.261+0000 7f6cd3fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:27:00.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.261+0000 7f6cd3fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:27:00.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.261+0000 7f6cd37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:27:00.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.261+0000 7f6cd37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:27:00.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.261+0000 7f6cd37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:27:00.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.261+0000 7f6cd3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:27:00.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.261+0000 7f6cd3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:27:00.275 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.261+0000 7f6cd3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:27:00.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.261+0000 7f6ccaffd640 1 --2- 172.21.15.63:0/3298051190 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6cd40965a0 0x7f6cd40969a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:00.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.261+0000 7f6cd37fe640 10 monclient(hunting): handle_auth_done global_id 24740 payload 824 2024-04-26T04:27:00.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.261+0000 7f6cd37fe640 1 -- 172.21.15.63:0/3298051190 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6cd40965a0 msgr2=0x7f6cd40969a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:27:00.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.261+0000 7f6cd37fe640 1 --2- 172.21.15.63:0/3298051190 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6cd40965a0 0x7f6cd40969a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:00.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.261+0000 7f6cd37fe640 1 -- 172.21.15.63:0/3298051190 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6cd4096ee0 msgr2=0x7f6cd41d95a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:27:00.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.261+0000 7f6cd37fe640 1 --2- 172.21.15.63:0/3298051190 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6cd4096ee0 0x7f6cd41d95a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:00.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.261+0000 7f6cd37fe640 10 monclient: _finish_hunting 0 2024-04-26T04:27:00.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.261+0000 7f6cd37fe640 1 monclient: found mon.noname-c 2024-04-26T04:27:00.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.261+0000 7f6cd37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:27:00.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.261+0000 7f6cd37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.63:3301/0 2024-04-26T04:27:00.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.261+0000 7f6cd37fe640 1 -- 172.21.15.63:0/3298051190 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cd406ea60 con 0x7f6cc0003aa0 2024-04-26T04:27:00.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.261+0000 7f6ccaffd640 10 monclient: get_auth_request con 0x7f6cd40965a0 auth_method 0 2024-04-26T04:27:00.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.261+0000 7f6ccaffd640 1 --2- 172.21.15.63:0/3298051190 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6cd40965a0 0x7f6cd40969a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T04:27:00.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.261+0000 7f6cd37fe640 1 -- 172.21.15.63:0/3298051190 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f6cc0003aa0 msgr2=0x7f6cd4096060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:27:00.276 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.261+0000 7f6cd37fe640 1 --2- 172.21.15.63:0/3298051190 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f6cc0003aa0 0x7f6cd4096060 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6cc401c790 tx=0x7f6cc0005840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:27:00.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.261+0000 7f6cd0ff9640 1 -- 172.21.15.63:0/3298051190 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cc40bb070 con 0x7f6cc0003aa0 2024-04-26T04:27:00.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.261+0000 7f6cd0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:27:00.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.261+0000 7f6cd0ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:27:00.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.261+0000 7f6cd0ff9640 10 monclient: dump: 2024-04-26T04:27:00.277 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:27:00.277 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:27:00.277 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:00.277 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:00.277 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:27:00.277 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:27:00.277 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:27:00.277 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:27:00.277 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:27:00.277 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:00.277 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.261+0000 7f6cd0ff9640 10 monclient: _finish_auth 0 2024-04-26T04:27:00.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.261+0000 7f6cd0ff9640 10 monclient: _check_auth_tickets 2024-04-26T04:27:00.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.261+0000 7f6cd0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:27:00.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.261+0000 7f6cd9d68640 5 monclient: authenticate success, global_id 24740 2024-04-26T04:27:00.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.261+0000 7f6cd9d68640 10 monclient: _renew_subs 2024-04-26T04:27:00.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.261+0000 7f6cd9d68640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:27:00.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.261+0000 7f6cd9d68640 1 -- 172.21.15.63:0/3298051190 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cd406ed40 con 0x7f6cc0003aa0 2024-04-26T04:27:00.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.261+0000 7f6cd0ff9640 1 -- 172.21.15.63:0/3298051190 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cc40abe30 con 0x7f6cc0003aa0 2024-04-26T04:27:00.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.261+0000 7f6cd0ff9640 10 monclient: handle_config config(0 keys) 2024-04-26T04:27:00.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.261+0000 7f6cd0ff9640 1 -- 172.21.15.63:0/3298051190 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cc40b0040 con 0x7f6cc0003aa0 2024-04-26T04:27:00.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.261+0000 7f6cd0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:27:00.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.261+0000 7f6cd0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:27:00.278 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.261+0000 7f6cd0ff9640 10 monclient: dump: 2024-04-26T04:27:00.278 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:27:00.278 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:27:00.278 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:00.279 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:00.279 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:27:00.279 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:27:00.279 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:27:00.279 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:27:00.279 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:27:00.279 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:00.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.261+0000 7f6cd27fc640 4 set_mon_vals no callback set 2024-04-26T04:27:00.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.261+0000 7f6cd9d68640 10 monclient: _renew_subs 2024-04-26T04:27:00.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.261+0000 7f6cd9d68640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:27:00.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.261+0000 7f6cd9d68640 1 -- 172.21.15.63:0/3298051190 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cd409d0e0 con 0x7f6cc0003aa0 2024-04-26T04:27:00.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.261+0000 7f6cd9d68640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:27:00.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.261+0000 7f6cd9d68640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:27:00.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.261+0000 7f6cd9d68640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:27:00.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.261+0000 7f6cd9d68640 1 -- 172.21.15.63:0/3298051190 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cd4062e10 con 0x7f6cc0003aa0 2024-04-26T04:27:00.279 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.261+0000 7f6cd0ff9640 1 -- 172.21.15.63:0/3298051190 <== mon.2 v2:172.21.15.63:3301/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f6cc40be070 con 0x7f6cc0003aa0 2024-04-26T04:27:00.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.261+0000 7f6cd0ff9640 1 --2- 172.21.15.63:0/3298051190 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f6cb4035ec0 0x7f6cb4038380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:00.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.265+0000 7f6cd0ff9640 1 -- 172.21.15.63:0/3298051190 <== mon.2 v2:172.21.15.63:3301/0 5 ==== osd_map(323..323 src has 1..323) ==== 4346+0+0 (secure 0 0 0) 0x7f6cc40e3040 con 0x7f6cc0003aa0 2024-04-26T04:27:00.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.265+0000 7f6ccaffd640 1 --2- 172.21.15.63:0/3298051190 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f6cb4035ec0 0x7f6cb4038380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:00.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.265+0000 7f6ccaffd640 10 monclient: get_auth_request con 0x7f6cb4035ec0 auth_method 0 2024-04-26T04:27:00.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.265+0000 7f6ccaffd640 1 --2- 172.21.15.63:0/3298051190 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f6cb4035ec0 0x7f6cb4038380 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f6cbc001470 tx=0x7f6cbc000ee0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:27:00.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.269+0000 7f6cd0ff9640 1 -- 172.21.15.63:0/3298051190 <== mon.2 v2:172.21.15.63:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f6cc4015ff0 con 0x7f6cc0003aa0 2024-04-26T04:27:00.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.269+0000 7f6cd0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:27:00.280 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.269+0000 7f6cd0ff9640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:27:00.426 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.425+0000 7f6cd9d68640 1 -- 172.21.15.63:0/3298051190 --> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7f6cd40729f0 con 0x7f6cb4035ec0 2024-04-26T04:27:00.427 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.425+0000 7f6cd0ff9640 1 -- 172.21.15.63:0/3298051190 <== mgr.4105 v2:172.21.15.186:6824/2583706639 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f6cd40729f0 con 0x7f6cb4035ec0 2024-04-26T04:27:00.427 INFO:teuthology.orchestra.run.smithi063.stderr:instructed osd(s) 2 to deep-scrub 2024-04-26T04:27:00.429 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.425+0000 7f6cd9d68640 1 -- 172.21.15.63:0/3298051190 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f6cb4035ec0 msgr2=0x7f6cb4038380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:27:00.429 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.425+0000 7f6cd9d68640 1 --2- 172.21.15.63:0/3298051190 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f6cb4035ec0 0x7f6cb4038380 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f6cbc001470 tx=0x7f6cbc000ee0 comp rx=0 tx=0).stop 2024-04-26T04:27:00.429 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.425+0000 7f6cd9d68640 10 monclient: shutdown 2024-04-26T04:27:00.429 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.425+0000 7f6cd9d68640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:27:00.429 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.425+0000 7f6cd9d68640 1 -- 172.21.15.63:0/3298051190 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f6cc0003aa0 msgr2=0x7f6cd4096060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:27:00.429 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.425+0000 7f6cd9d68640 1 --2- 172.21.15.63:0/3298051190 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f6cc0003aa0 0x7f6cd4096060 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6cc401c790 tx=0x7f6cc0005840 comp rx=0 tx=0).stop 2024-04-26T04:27:00.429 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.425+0000 7f6cd9d68640 1 -- 172.21.15.63:0/3298051190 shutdown_connections 2024-04-26T04:27:00.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.425+0000 7f6cd9d68640 1 --2- 172.21.15.63:0/3298051190 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f6cb4035ec0 0x7f6cb4038380 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:00.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.425+0000 7f6cd9d68640 1 --2- 172.21.15.63:0/3298051190 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6cd4096ee0 0x7f6cd41d95a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:00.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.425+0000 7f6cd9d68640 1 --2- 172.21.15.63:0/3298051190 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6cd40965a0 0x7f6cd40969a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:00.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.425+0000 7f6cd9d68640 1 --2- 172.21.15.63:0/3298051190 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f6cc0003aa0 0x7f6cd4096060 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:00.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.425+0000 7f6cd9d68640 1 -- 172.21.15.63:0/3298051190 >> 172.21.15.63:0/3298051190 conn(0x7f6cd40a1e70 msgr2=0x7f6cd413b920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:27:00.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.425+0000 7f6cd9d68640 1 -- 172.21.15.63:0/3298051190 shutdown_connections 2024-04-26T04:27:00.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.425+0000 7f6cd9d68640 1 -- 172.21.15.63:0/3298051190 wait complete. 2024-04-26T04:27:00.445 INFO:tasks.ceph:Scrubbing osd.3 2024-04-26T04:27:00.445 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-04-26T04:27:00.571 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.569+0000 7faabccfc640 10 monclient: get_monmap_and_config 2024-04-26T04:27:00.571 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.569+0000 7faabccfc640 10 monclient: build_initial_monmap 2024-04-26T04:27:00.571 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.569+0000 7faabccfc640 10 monclient: monmap: 2024-04-26T04:27:00.571 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:27:00.572 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:27:00.572 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:27:00.571653+0000 2024-04-26T04:27:00.572 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:27:00.571653+0000 2024-04-26T04:27:00.572 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:27:00.572 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:27:00.572 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:27:00.572 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:27:00.572 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:27:00.572 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:27:00.572 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:27:00.572 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:00.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.569+0000 7faabccfc640 1 Processor -- start 2024-04-26T04:27:00.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.569+0000 7faabccfc640 1 -- start start 2024-04-26T04:27:00.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.569+0000 7faabccfc640 10 monclient: init 2024-04-26T04:27:00.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.569+0000 7faabccfc640 10 monclient: _reopen_session rank -1 2024-04-26T04:27:00.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.569+0000 7faabccfc640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2024-04-26T04:27:00.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.569+0000 7faabccfc640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7faab809e760 0x7faab80a0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:00.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.569+0000 7faabccfc640 10 monclient(hunting): picked mon.noname-c con 0x7faab809e760 addr v2:172.21.15.186:3300/0 2024-04-26T04:27:00.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.569+0000 7faabccfc640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7faab809cfc0 0x7faab80a10f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:00.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.569+0000 7faabccfc640 10 monclient(hunting): picked mon.noname-b con 0x7faab809cfc0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:27:00.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.569+0000 7faabccfc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7faab80a1630 addr v1:172.21.15.186:6789/0 2024-04-26T04:27:00.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.569+0000 7faabccfc640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faab809f0b0 con 0x7faab80a1630 2024-04-26T04:27:00.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.569+0000 7faabccfc640 10 monclient(hunting): start opening mon connection 2024-04-26T04:27:00.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.569+0000 7faabccfc640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7faab809c860 con 0x7faab809e760 2024-04-26T04:27:00.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.569+0000 7faabccfc640 10 monclient(hunting): start opening mon connection 2024-04-26T04:27:00.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.569+0000 7faabccfc640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7faab809eb40 con 0x7faab809cfc0 2024-04-26T04:27:00.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.569+0000 7faabccfc640 10 monclient(hunting): _renew_subs 2024-04-26T04:27:00.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.569+0000 7faabccfc640 10 monclient(hunting): authenticate will time out at 1452.279844s 2024-04-26T04:27:00.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.573+0000 7faab5d9b640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7faab809cfc0 0x7faab80a10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:00.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.573+0000 7faab5d9b640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7faab809cfc0 0x7faab80a10f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:39868/0 (socket says 172.21.15.63:39868) 2024-04-26T04:27:00.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.573+0000 7faab5d9b640 1 -- 172.21.15.63:0/3938022644 learned_addr learned my addr 172.21.15.63:0/3938022644 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:27:00.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.573+0000 7faab5d9b640 10 monclient(hunting): get_auth_request con 0x7faab809cfc0 auth_method 0 2024-04-26T04:27:00.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.573+0000 7faab5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:27:00.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.573+0000 7faab5d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:27:00.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.573+0000 7faab5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:27:00.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.573+0000 7faab659c640 1 --2- 172.21.15.63:0/3938022644 >> v2:172.21.15.186:3300/0 conn(0x7faab809e760 0x7faab80a0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:00.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.573+0000 7faab659c640 10 monclient(hunting): get_auth_request con 0x7faab809e760 auth_method 0 2024-04-26T04:27:00.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.573+0000 7faab659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:27:00.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.573+0000 7faab659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:27:00.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.573+0000 7faab659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:27:00.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.573+0000 7faab5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:27:00.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.573+0000 7faab5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:27:00.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.573+0000 7faab5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:27:00.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.573+0000 7faab659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:27:00.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.573+0000 7faab659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:27:00.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.573+0000 7faab659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:27:00.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.573+0000 7faab5d9b640 10 monclient(hunting): handle_auth_done global_id 24743 payload 274 2024-04-26T04:27:00.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.573+0000 7faab5d9b640 1 -- 172.21.15.63:0/3938022644 >> v2:172.21.15.186:3300/0 conn(0x7faab809e760 msgr2=0x7faab80a0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:27:00.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.573+0000 7faab5d9b640 1 --2- 172.21.15.63:0/3938022644 >> v2:172.21.15.186:3300/0 conn(0x7faab809e760 0x7faab80a0bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:00.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.573+0000 7faab5d9b640 1 -- 172.21.15.63:0/3938022644 >> v1:172.21.15.186:6789/0 conn(0x7faab80a1630 legacy=0x7faab813b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:27:00.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.573+0000 7faab5d9b640 10 monclient: _finish_hunting 0 2024-04-26T04:27:00.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.573+0000 7faab5d9b640 1 monclient: found mon.noname-b 2024-04-26T04:27:00.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.573+0000 7faab5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:27:00.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.573+0000 7faab5d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.63:3301/0 2024-04-26T04:27:00.578 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.573+0000 7faab5d9b640 1 -- 172.21.15.63:0/3938022644 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faab809def0 con 0x7faab809cfc0 2024-04-26T04:27:00.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.577+0000 7faab5d9b640 1 -- 172.21.15.63:0/3938022644 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7faab809cfc0 msgr2=0x7faab80a10f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:27:00.579 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.577+0000 7faab5d9b640 1 --2- 172.21.15.63:0/3938022644 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7faab809cfc0 0x7faab80a10f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7faaac009870 tx=0x7faaac01c470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=390955a04387f53 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:27:00.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.577+0000 7faab559a640 1 -- 172.21.15.63:0/3938022644 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaac0ab430 con 0x7faab809cfc0 2024-04-26T04:27:00.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.577+0000 7faab559a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:27:00.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.577+0000 7faab559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T04:27:00.581 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.577+0000 7faab559a640 10 monclient: dump: 2024-04-26T04:27:00.581 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:27:00.581 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:27:00.582 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:00.582 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:00.582 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:27:00.582 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:27:00.582 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:27:00.582 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:27:00.582 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:27:00.582 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:00.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.577+0000 7faab559a640 10 monclient: _finish_auth 0 2024-04-26T04:27:00.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.577+0000 7faab559a640 10 monclient: _check_auth_tickets 2024-04-26T04:27:00.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.577+0000 7faab559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:27:00.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.577+0000 7faab559a640 1 -- 172.21.15.63:0/3938022644 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faaac0b0070 con 0x7faab809cfc0 2024-04-26T04:27:00.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.577+0000 7faab559a640 10 monclient: handle_config config(0 keys) 2024-04-26T04:27:00.582 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.577+0000 7faab559a640 1 -- 172.21.15.63:0/3938022644 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaac0ab7f0 con 0x7faab809cfc0 2024-04-26T04:27:00.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.577+0000 7faabccfc640 5 monclient: authenticate success, global_id 24743 2024-04-26T04:27:00.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.577+0000 7faabccfc640 10 monclient: get_monmap_and_config success 2024-04-26T04:27:00.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.577+0000 7faabccfc640 4 set_mon_vals no callback set 2024-04-26T04:27:00.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.577+0000 7faab559a640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:27:00.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.577+0000 7faab559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:27:00.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.577+0000 7faab559a640 10 monclient: dump: 2024-04-26T04:27:00.583 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:27:00.583 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:27:00.583 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:00.583 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:00.583 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:27:00.583 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:27:00.583 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:27:00.583 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:27:00.583 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:27:00.584 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:00.584 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.577+0000 7faabccfc640 10 monclient: shutdown 2024-04-26T04:27:00.584 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.577+0000 7faabccfc640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:27:00.584 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.577+0000 7faabccfc640 1 -- 172.21.15.63:0/3938022644 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7faab809cfc0 msgr2=0x7faab80a10f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:27:00.584 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.577+0000 7faabccfc640 1 --2- 172.21.15.63:0/3938022644 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7faab809cfc0 0x7faab80a10f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7faaac009870 tx=0x7faaac01c470 comp rx=0 tx=0).stop 2024-04-26T04:27:00.584 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.577+0000 7faabccfc640 1 -- 172.21.15.63:0/3938022644 shutdown_connections 2024-04-26T04:27:00.584 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.577+0000 7faabccfc640 1 --2- 172.21.15.63:0/3938022644 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7faab809cfc0 0x7faab80a10f0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:00.584 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.577+0000 7faabccfc640 1 --2- 172.21.15.63:0/3938022644 >> v2:172.21.15.186:3300/0 conn(0x7faab809e760 0x7faab80a0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:00.584 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.577+0000 7faabccfc640 1 -- 172.21.15.63:0/3938022644 >> 172.21.15.63:0/3938022644 conn(0x7faab80a1ca0 msgr2=0x7faab80a20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:27:00.584 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.577+0000 7faabccfc640 1 -- 172.21.15.63:0/3938022644 shutdown_connections 2024-04-26T04:27:00.584 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.577+0000 7faabccfc640 1 -- 172.21.15.63:0/3938022644 wait complete. 2024-04-26T04:27:00.584 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.581+0000 7faabccfc640 10 monclient: build_initial_monmap 2024-04-26T04:27:00.584 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.581+0000 7faabccfc640 10 monclient: monmap: 2024-04-26T04:27:00.584 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:27:00.584 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:27:00.585 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:27:00.585 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:27:00.585 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:27:00.585 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:27:00.585 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:27:00.585 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:27:00.585 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:27:00.585 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:00.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.581+0000 7faabccfc640 1 Processor -- start 2024-04-26T04:27:00.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.581+0000 7faabccfc640 1 -- start start 2024-04-26T04:27:00.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.581+0000 7faabccfc640 10 monclient: init 2024-04-26T04:27:00.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.581+0000 7faabccfc640 10 monclient: _reopen_session rank -1 2024-04-26T04:27:00.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.581+0000 7faabccfc640 10 monclient: _add_conns ranks=[0,2,1] 2024-04-26T04:27:00.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.581+0000 7faabccfc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faab809cfc0 0x7faab81ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:00.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.581+0000 7faabccfc640 10 monclient(hunting): picked mon.noname-a con 0x7faab809cfc0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:27:00.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.581+0000 7faabccfc640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7faab809e760 0x7faab80998d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:00.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.581+0000 7faabccfc640 10 monclient(hunting): picked mon.noname-c con 0x7faab809e760 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:27:00.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.581+0000 7faabccfc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faab80a1630 0x7faab8099e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:00.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.581+0000 7faabccfc640 10 monclient(hunting): picked mon.noname-b con 0x7faab80a1630 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:27:00.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.581+0000 7faabccfc640 10 monclient(hunting): start opening mon connection 2024-04-26T04:27:00.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.581+0000 7faabccfc640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7faab813f400 con 0x7faab809cfc0 2024-04-26T04:27:00.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.581+0000 7faabccfc640 10 monclient(hunting): start opening mon connection 2024-04-26T04:27:00.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.581+0000 7faabccfc640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7faab813f280 con 0x7faab80a1630 2024-04-26T04:27:00.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.581+0000 7faabccfc640 10 monclient(hunting): start opening mon connection 2024-04-26T04:27:00.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.581+0000 7faabccfc640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7faab813f580 con 0x7faab809e760 2024-04-26T04:27:00.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.581+0000 7faabccfc640 10 monclient(hunting): _renew_subs 2024-04-26T04:27:00.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.581+0000 7faabccfc640 10 monclient(hunting): authenticate will time out at 1452.289073s 2024-04-26T04:27:00.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.581+0000 7faab659c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faab809cfc0 0x7faab81ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:00.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.581+0000 7faab5d9b640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7faab809e760 0x7faab80998d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:00.590 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.581+0000 7faab659c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faab809cfc0 0x7faab81ceb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:47558/0 (socket says 172.21.15.63:47558) 2024-04-26T04:27:00.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.581+0000 7faab5d9b640 10 monclient(hunting): get_auth_request con 0x7faab809e760 auth_method 0 2024-04-26T04:27:00.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.581+0000 7faab659c640 1 -- 172.21.15.63:0/852286448 learned_addr learned my addr 172.21.15.63:0/852286448 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:27:00.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.581+0000 7faab5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:27:00.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.581+0000 7faab5d9b640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:27:00.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.581+0000 7faab5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:27:00.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.581+0000 7faab659c640 10 monclient(hunting): get_auth_request con 0x7faab809cfc0 auth_method 0 2024-04-26T04:27:00.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.581+0000 7faab659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:27:00.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.581+0000 7faab659c640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:27:00.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.581+0000 7faab659c640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:27:00.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.581+0000 7faab5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:27:00.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.581+0000 7faab5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:27:00.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.581+0000 7faab5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:27:00.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.581+0000 7faab659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:27:00.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.581+0000 7faab659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:27:00.591 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.581+0000 7faab659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:27:00.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.581+0000 7faab659c640 10 monclient(hunting): handle_auth_done global_id 44190 payload 824 2024-04-26T04:27:00.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.581+0000 7faab6d9d640 1 --2- 172.21.15.63:0/852286448 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faab80a1630 0x7faab8099e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:00.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.581+0000 7faab659c640 1 -- 172.21.15.63:0/852286448 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7faab809e760 msgr2=0x7faab80998d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:27:00.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.581+0000 7faab659c640 1 --2- 172.21.15.63:0/852286448 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7faab809e760 0x7faab80998d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:00.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.581+0000 7faab659c640 1 -- 172.21.15.63:0/852286448 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faab80a1630 msgr2=0x7faab8099e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:27:00.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.581+0000 7faab659c640 1 --2- 172.21.15.63:0/852286448 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faab80a1630 0x7faab8099e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:00.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.581+0000 7faab659c640 10 monclient: _finish_hunting 0 2024-04-26T04:27:00.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.581+0000 7faab659c640 1 monclient: found mon.noname-a 2024-04-26T04:27:00.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.581+0000 7faab659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:27:00.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.581+0000 7faab659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:27:00.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.581+0000 7faab659c640 1 -- 172.21.15.63:0/852286448 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faab809ed80 con 0x7faab809cfc0 2024-04-26T04:27:00.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.581+0000 7faab6d9d640 10 monclient: get_auth_request con 0x7faab80a1630 auth_method 0 2024-04-26T04:27:00.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.581+0000 7faab5d9b640 1 --2- 172.21.15.63:0/852286448 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7faab809e760 0x7faab80998d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T04:27:00.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.581+0000 7faab6d9d640 1 --2- 172.21.15.63:0/852286448 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faab80a1630 0x7faab8099e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T04:27:00.592 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.581+0000 7faab659c640 1 -- 172.21.15.63:0/852286448 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faab809cfc0 msgr2=0x7faab81ceb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:27:00.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.581+0000 7faab659c640 1 --2- 172.21.15.63:0/852286448 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faab809cfc0 0x7faab81ceb00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7faaa800e960 tx=0x7faaa800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:27:00.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.585+0000 7faaa6ffd640 1 -- 172.21.15.63:0/852286448 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaa8017070 con 0x7faab809cfc0 2024-04-26T04:27:00.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.585+0000 7faaa6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:27:00.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.585+0000 7faaa6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:27:00.593 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.585+0000 7faaa6ffd640 10 monclient: dump: 2024-04-26T04:27:00.594 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:27:00.594 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:27:00.594 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:00.594 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:00.594 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:27:00.594 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:27:00.594 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:27:00.594 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:27:00.594 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:27:00.594 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:00.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.585+0000 7faaa6ffd640 10 monclient: _finish_auth 0 2024-04-26T04:27:00.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.585+0000 7faaa6ffd640 10 monclient: _check_auth_tickets 2024-04-26T04:27:00.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.585+0000 7faaa6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:27:00.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.585+0000 7faabccfc640 5 monclient: authenticate success, global_id 44190 2024-04-26T04:27:00.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.585+0000 7faabccfc640 10 monclient: _renew_subs 2024-04-26T04:27:00.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.585+0000 7faaa6ffd640 1 -- 172.21.15.63:0/852286448 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faaa800cc10 con 0x7faab809cfc0 2024-04-26T04:27:00.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.585+0000 7faabccfc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:27:00.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.585+0000 7faabccfc640 1 -- 172.21.15.63:0/852286448 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faab80960c0 con 0x7faab809cfc0 2024-04-26T04:27:00.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.585+0000 7faaa6ffd640 10 monclient: handle_config config(0 keys) 2024-04-26T04:27:00.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.585+0000 7faaa6ffd640 1 -- 172.21.15.63:0/852286448 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaa8013470 con 0x7faab809cfc0 2024-04-26T04:27:00.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.585+0000 7faaa6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:27:00.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.585+0000 7faaa6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:27:00.595 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.585+0000 7faaa6ffd640 10 monclient: dump: 2024-04-26T04:27:00.595 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:27:00.595 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:27:00.595 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:00.595 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:00.595 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:27:00.595 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:27:00.596 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:27:00.596 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:27:00.596 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:27:00.596 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:00.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.585+0000 7faabccfc640 10 monclient: _renew_subs 2024-04-26T04:27:00.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.585+0000 7faabccfc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:27:00.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.585+0000 7faab4d99640 4 set_mon_vals no callback set 2024-04-26T04:27:00.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.585+0000 7faabccfc640 1 -- 172.21.15.63:0/852286448 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faab81cf040 con 0x7faab809cfc0 2024-04-26T04:27:00.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.585+0000 7faabccfc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:27:00.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.585+0000 7faabccfc640 1 -- 172.21.15.63:0/852286448 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7faab8067fb0 con 0x7faab809cfc0 2024-04-26T04:27:00.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.585+0000 7faaa6ffd640 1 -- 172.21.15.63:0/852286448 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7faaa801a070 con 0x7faab809cfc0 2024-04-26T04:27:00.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.585+0000 7faaa6ffd640 1 --2- 172.21.15.63:0/852286448 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7faa98035ec0 0x7faa98038380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:00.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.585+0000 7faaa6ffd640 1 -- 172.21.15.63:0/852286448 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(323..323 src has 1..323) ==== 4346+0+0 (secure 0 0 0) 0x7faaa8049060 con 0x7faab809cfc0 2024-04-26T04:27:00.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.585+0000 7faaa6ffd640 1 --2- 172.21.15.63:0/852286448 >> [v2:172.21.15.186:6807/4227496981,v1:172.21.15.186:6808/4227496981] conn(0x7faa9803bd30 0x7faa9803e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:00.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.585+0000 7faaa6ffd640 1 -- 172.21.15.63:0/852286448 --> [v2:172.21.15.186:6807/4227496981,v1:172.21.15.186:6808/4227496981] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7faaa8022e70 con 0x7faa9803bd30 2024-04-26T04:27:00.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.585+0000 7faaa6ffd640 1 -- 172.21.15.63:0/852286448 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_get_version_reply(handle=1 version=323) ==== 24+0+0 (secure 0 0 0) 0x7faaa8049340 con 0x7faab809cfc0 2024-04-26T04:27:00.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.585+0000 7faaa6ffd640 10 monclient: handle_get_version_reply finishing 1 version 323 2024-04-26T04:27:00.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.585+0000 7faab5d9b640 1 --2- 172.21.15.63:0/852286448 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7faa98035ec0 0x7faa98038380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:00.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.585+0000 7faab6d9d640 1 --2- 172.21.15.63:0/852286448 >> [v2:172.21.15.186:6807/4227496981,v1:172.21.15.186:6808/4227496981] conn(0x7faa9803bd30 0x7faa9803e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:00.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.585+0000 7faab5d9b640 10 monclient: get_auth_request con 0x7faa98035ec0 auth_method 0 2024-04-26T04:27:00.597 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.585+0000 7faab6d9d640 10 monclient: get_auth_request con 0x7faa9803bd30 auth_method 0 2024-04-26T04:27:00.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.589+0000 7faab5d9b640 1 --2- 172.21.15.63:0/852286448 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7faa98035ec0 0x7faa98038380 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7faaac0ac000 tx=0x7faaac008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:27:00.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.589+0000 7faab6d9d640 1 --2- 172.21.15.63:0/852286448 >> [v2:172.21.15.186:6807/4227496981,v1:172.21.15.186:6808/4227496981] conn(0x7faa9803bd30 0x7faa9803e150 crc :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:27:00.598 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.593+0000 7faaa6ffd640 1 -- 172.21.15.63:0/852286448 <== osd.3 v2:172.21.15.186:6807/4227496981 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7faaa8022e70 con 0x7faa9803bd30 2024-04-26T04:27:00.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.605+0000 7faabccfc640 1 -- 172.21.15.63:0/852286448 --> [v2:172.21.15.186:6807/4227496981,v1:172.21.15.186:6808/4227496981] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7faab80926b0 con 0x7faa9803bd30 2024-04-26T04:27:00.608 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.605+0000 7faaa6ffd640 1 -- 172.21.15.63:0/852286448 <== osd.3 v2:172.21.15.186:6807/4227496981 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7faab80926b0 con 0x7faa9803bd30 2024-04-26T04:27:00.608 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-26T04:27:00.608 INFO:teuthology.orchestra.run.smithi063.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-26T04:27:00.608 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-26T04:27:00.609 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.605+0000 7faabccfc640 1 -- 172.21.15.63:0/852286448 >> [v2:172.21.15.186:6807/4227496981,v1:172.21.15.186:6808/4227496981] conn(0x7faa9803bd30 msgr2=0x7faa9803e150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:27:00.609 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.605+0000 7faabccfc640 1 --2- 172.21.15.63:0/852286448 >> [v2:172.21.15.186:6807/4227496981,v1:172.21.15.186:6808/4227496981] conn(0x7faa9803bd30 0x7faa9803e150 crc :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:00.609 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.605+0000 7faabccfc640 1 -- 172.21.15.63:0/852286448 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7faa98035ec0 msgr2=0x7faa98038380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:27:00.609 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.605+0000 7faabccfc640 1 --2- 172.21.15.63:0/852286448 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7faa98035ec0 0x7faa98038380 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7faaac0ac000 tx=0x7faaac008040 comp rx=0 tx=0).stop 2024-04-26T04:27:00.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.605+0000 7faabccfc640 10 monclient: shutdown 2024-04-26T04:27:00.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.605+0000 7faabccfc640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:27:00.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.605+0000 7faabccfc640 1 -- 172.21.15.63:0/852286448 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faab809cfc0 msgr2=0x7faab81ceb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:27:00.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.605+0000 7faabccfc640 1 --2- 172.21.15.63:0/852286448 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faab809cfc0 0x7faab81ceb00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7faaa800e960 tx=0x7faaa800ee20 comp rx=0 tx=0).stop 2024-04-26T04:27:00.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.605+0000 7faab6d9d640 1 -- 172.21.15.63:0/852286448 reap_dead start 2024-04-26T04:27:00.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.605+0000 7faabccfc640 1 -- 172.21.15.63:0/852286448 shutdown_connections 2024-04-26T04:27:00.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.605+0000 7faabccfc640 1 -- 172.21.15.63:0/852286448 >> 172.21.15.63:0/852286448 conn(0x7faab80a1ca0 msgr2=0x7faab813bd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:27:00.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.605+0000 7faabccfc640 1 -- 172.21.15.63:0/852286448 shutdown_connections 2024-04-26T04:27:00.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.605+0000 7faabccfc640 1 -- 172.21.15.63:0/852286448 wait complete. 2024-04-26T04:27:00.624 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-04-26T04:27:00.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.733+0000 7fb29badf640 10 monclient: get_monmap_and_config 2024-04-26T04:27:00.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.733+0000 7fb29badf640 10 monclient: build_initial_monmap 2024-04-26T04:27:00.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.733+0000 7fb29badf640 10 monclient: monmap: 2024-04-26T04:27:00.737 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:27:00.737 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:27:00.737 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:27:00.737242+0000 2024-04-26T04:27:00.738 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:27:00.737242+0000 2024-04-26T04:27:00.738 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:27:00.738 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:27:00.738 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:27:00.738 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:27:00.738 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:27:00.738 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:27:00.738 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:27:00.738 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:00.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.737+0000 7fb29badf640 1 Processor -- start 2024-04-26T04:27:00.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.737+0000 7fb29badf640 1 -- start start 2024-04-26T04:27:00.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.737+0000 7fb29badf640 10 monclient: init 2024-04-26T04:27:00.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.737+0000 7fb29badf640 10 monclient: _reopen_session rank -1 2024-04-26T04:27:00.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.737+0000 7fb29badf640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-04-26T04:27:00.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.737+0000 7fb29badf640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb2940a2cd0 addr v1:172.21.15.63:6789/0 2024-04-26T04:27:00.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.737+0000 7fb29badf640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7fb2940a30b0 0x7fb29413c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:00.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.737+0000 7fb29badf640 10 monclient(hunting): picked mon.noname-a con 0x7fb2940a30b0 addr v2:172.21.15.63:3300/0 2024-04-26T04:27:00.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.737+0000 7fb29badf640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb29413c540 addr v1:172.21.15.186:6789/0 2024-04-26T04:27:00.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.737+0000 7fb29badf640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2940a4ee0 con 0x7fb2940a2cd0 2024-04-26T04:27:00.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.737+0000 7fb29badf640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2940a0fb0 con 0x7fb29413c540 2024-04-26T04:27:00.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.737+0000 7fb29badf640 10 monclient(hunting): start opening mon connection 2024-04-26T04:27:00.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.737+0000 7fb29badf640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7fb2940a5380 con 0x7fb2940a30b0 2024-04-26T04:27:00.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.737+0000 7fb29badf640 10 monclient(hunting): _renew_subs 2024-04-26T04:27:00.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.737+0000 7fb29badf640 10 monclient(hunting): authenticate will time out at 1452.444915s 2024-04-26T04:27:00.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.737+0000 7fb29907a640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7fb2940a30b0 0x7fb29413c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:00.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.737+0000 7fb29987b640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7fb2940a2cd0 0x7fb2940a47d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:37564/0 (socket says 172.21.15.63:37564) 2024-04-26T04:27:00.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.737+0000 7fb29987b640 1 -- 172.21.15.63:0/464480912 learned_addr learned my addr 172.21.15.63:0/464480912 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-04-26T04:27:00.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.737+0000 7fb29907a640 10 monclient(hunting): get_auth_request con 0x7fb2940a30b0 auth_method 0 2024-04-26T04:27:00.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.737+0000 7fb29907a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:27:00.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.737+0000 7fb29907a640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:27:00.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.737+0000 7fb29907a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:27:00.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.737+0000 7fb29907a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:27:00.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.737+0000 7fb29907a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:27:00.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.737+0000 7fb29907a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:27:00.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.737+0000 7fb29907a640 10 monclient(hunting): handle_auth_done global_id 44193 payload 274 2024-04-26T04:27:00.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.737+0000 7fb29907a640 1 -- 172.21.15.63:0/464480912 >> v1:172.21.15.186:6789/0 conn(0x7fb29413c540 legacy=0x7fb29413e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:27:00.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.737+0000 7fb29907a640 1 -- 172.21.15.63:0/464480912 >> v1:172.21.15.63:6789/0 conn(0x7fb2940a2cd0 legacy=0x7fb2940a47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:27:00.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.737+0000 7fb29907a640 10 monclient: _finish_hunting 0 2024-04-26T04:27:00.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.737+0000 7fb29907a640 1 monclient: found mon.noname-a 2024-04-26T04:27:00.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.737+0000 7fb29907a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:27:00.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.737+0000 7fb29907a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:27:00.743 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.737+0000 7fb29907a640 1 -- 172.21.15.63:0/464480912 --> v2:172.21.15.63:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2940a1290 con 0x7fb2940a30b0 2024-04-26T04:27:00.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.741+0000 7fb29907a640 1 -- 172.21.15.63:0/464480912 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb2940a30b0 msgr2=0x7fb29413c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:27:00.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.741+0000 7fb29907a640 1 --2- 172.21.15.63:0/464480912 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb2940a30b0 0x7fb29413c000 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb2840097c0 tx=0x7fb28401c370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9f56b7516f15ee1 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:27:00.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.741+0000 7fb298879640 1 -- 172.21.15.63:0/464480912 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2840b0070 con 0x7fb2940a30b0 2024-04-26T04:27:00.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.741+0000 7fb298879640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:27:00.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.741+0000 7fb298879640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:27:00.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.741+0000 7fb298879640 10 monclient: dump: 2024-04-26T04:27:00.748 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:27:00.748 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:27:00.748 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:00.748 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:00.748 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:27:00.748 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:27:00.748 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:27:00.749 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:27:00.749 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:27:00.749 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:00.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.741+0000 7fb298879640 10 monclient: _finish_auth 0 2024-04-26T04:27:00.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.741+0000 7fb298879640 10 monclient: _check_auth_tickets 2024-04-26T04:27:00.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.741+0000 7fb298879640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:27:00.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.741+0000 7fb298879640 1 -- 172.21.15.63:0/464480912 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2840ab600 con 0x7fb2940a30b0 2024-04-26T04:27:00.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.741+0000 7fb298879640 10 monclient: handle_config config(0 keys) 2024-04-26T04:27:00.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.741+0000 7fb29badf640 5 monclient: authenticate success, global_id 44193 2024-04-26T04:27:00.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.741+0000 7fb298879640 1 -- 172.21.15.63:0/464480912 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2840ab9d0 con 0x7fb2940a30b0 2024-04-26T04:27:00.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.741+0000 7fb29badf640 10 monclient: get_monmap_and_config success 2024-04-26T04:27:00.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.741+0000 7fb29badf640 4 set_mon_vals no callback set 2024-04-26T04:27:00.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.741+0000 7fb298879640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:27:00.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.741+0000 7fb298879640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:27:00.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.741+0000 7fb298879640 10 monclient: dump: 2024-04-26T04:27:00.750 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:27:00.750 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:27:00.750 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:00.751 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:00.751 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:27:00.751 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:27:00.751 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:27:00.751 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:27:00.751 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:27:00.751 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:00.751 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.741+0000 7fb29badf640 10 monclient: shutdown 2024-04-26T04:27:00.751 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.741+0000 7fb29badf640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:27:00.751 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.741+0000 7fb29badf640 1 -- 172.21.15.63:0/464480912 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb2940a30b0 msgr2=0x7fb29413c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:27:00.751 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.741+0000 7fb29badf640 1 --2- 172.21.15.63:0/464480912 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb2940a30b0 0x7fb29413c000 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb2840097c0 tx=0x7fb28401c370 comp rx=0 tx=0).stop 2024-04-26T04:27:00.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.741+0000 7fb29badf640 1 -- 172.21.15.63:0/464480912 shutdown_connections 2024-04-26T04:27:00.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.741+0000 7fb29badf640 1 --2- 172.21.15.63:0/464480912 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb2940a30b0 0x7fb29413c000 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:00.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.741+0000 7fb29badf640 1 -- 172.21.15.63:0/464480912 >> 172.21.15.63:0/464480912 conn(0x7fb2940a5f40 msgr2=0x7fb2940a6320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:27:00.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.741+0000 7fb29badf640 1 -- 172.21.15.63:0/464480912 shutdown_connections 2024-04-26T04:27:00.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.741+0000 7fb29badf640 1 -- 172.21.15.63:0/464480912 wait complete. 2024-04-26T04:27:00.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.741+0000 7fb29badf640 10 monclient: build_initial_monmap 2024-04-26T04:27:00.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.741+0000 7fb29badf640 10 monclient: monmap: 2024-04-26T04:27:00.752 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:27:00.752 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:27:00.752 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:27:00.752 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:27:00.752 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:27:00.753 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:27:00.753 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:27:00.753 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:27:00.753 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:27:00.753 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:00.753 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.741+0000 7fb29badf640 1 Processor -- start 2024-04-26T04:27:00.753 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.741+0000 7fb29badf640 1 -- start start 2024-04-26T04:27:00.753 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.741+0000 7fb29badf640 10 monclient: init 2024-04-26T04:27:00.753 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb29badf640 10 monclient: _reopen_session rank -1 2024-04-26T04:27:00.753 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb29badf640 10 monclient: _add_conns ranks=[2,0,1] 2024-04-26T04:27:00.753 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb29badf640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fb2940a2cd0 0x7fb2941cef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:00.753 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb29badf640 10 monclient(hunting): picked mon.noname-c con 0x7fb2940a2cd0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:27:00.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb29badf640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb2940a30b0 0x7fb2941cf740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:00.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb29badf640 10 monclient(hunting): picked mon.noname-a con 0x7fb2940a30b0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:27:00.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb29badf640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb29413c540 0x7fb29406f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:00.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb29badf640 10 monclient(hunting): picked mon.noname-b con 0x7fb29413c540 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:27:00.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb29badf640 10 monclient(hunting): start opening mon connection 2024-04-26T04:27:00.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb29badf640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fb294142c00 con 0x7fb2940a30b0 2024-04-26T04:27:00.754 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb29badf640 10 monclient(hunting): start opening mon connection 2024-04-26T04:27:00.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb29badf640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb294142a80 con 0x7fb29413c540 2024-04-26T04:27:00.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb29badf640 10 monclient(hunting): start opening mon connection 2024-04-26T04:27:00.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb29badf640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7fb294142d80 con 0x7fb2940a2cd0 2024-04-26T04:27:00.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb29badf640 10 monclient(hunting): _renew_subs 2024-04-26T04:27:00.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb29badf640 10 monclient(hunting): authenticate will time out at 1452.452123s 2024-04-26T04:27:00.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb29987b640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fb2940a2cd0 0x7fb2941cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:00.755 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb29907a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb2940a30b0 0x7fb2941cf740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:00.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb29907a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb2940a30b0 0x7fb2941cf740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:47580/0 (socket says 172.21.15.63:47580) 2024-04-26T04:27:00.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb29987b640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fb2940a2cd0 0x7fb2941cef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:39888/0 (socket says 172.21.15.63:39888) 2024-04-26T04:27:00.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb29907a640 1 -- 172.21.15.63:0/3027087705 learned_addr learned my addr 172.21.15.63:0/3027087705 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:27:00.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb29907a640 10 monclient(hunting): get_auth_request con 0x7fb2940a30b0 auth_method 0 2024-04-26T04:27:00.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb29907a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:27:00.756 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb29907a640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:27:00.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb29907a640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:27:00.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb29987b640 10 monclient(hunting): get_auth_request con 0x7fb2940a2cd0 auth_method 0 2024-04-26T04:27:00.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb29987b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:27:00.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb29987b640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:27:00.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb29987b640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:27:00.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb29907a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:27:00.757 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb29907a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:27:00.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb29907a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:27:00.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb29987b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:27:00.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb29987b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:27:00.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb29987b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:27:00.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb29a07c640 1 --2- 172.21.15.63:0/3027087705 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb29413c540 0x7fb29406f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:00.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb29907a640 10 monclient(hunting): handle_auth_done global_id 44196 payload 824 2024-04-26T04:27:00.758 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb29907a640 1 -- 172.21.15.63:0/3027087705 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fb2940a2cd0 msgr2=0x7fb2941cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:27:00.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb29907a640 1 --2- 172.21.15.63:0/3027087705 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fb2940a2cd0 0x7fb2941cef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:00.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb29907a640 1 -- 172.21.15.63:0/3027087705 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb29413c540 msgr2=0x7fb29406f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:27:00.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb29907a640 1 --2- 172.21.15.63:0/3027087705 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb29413c540 0x7fb29406f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:00.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb29907a640 10 monclient: _finish_hunting 0 2024-04-26T04:27:00.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb29907a640 1 monclient: found mon.noname-a 2024-04-26T04:27:00.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb29907a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:27:00.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb29907a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:27:00.759 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb29907a640 1 -- 172.21.15.63:0/3027087705 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb294070c60 con 0x7fb2940a30b0 2024-04-26T04:27:00.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb29a07c640 10 monclient: get_auth_request con 0x7fb29413c540 auth_method 0 2024-04-26T04:27:00.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb29987b640 1 --2- 172.21.15.63:0/3027087705 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fb2940a2cd0 0x7fb2941cef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T04:27:00.760 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb29a07c640 1 --2- 172.21.15.63:0/3027087705 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb29413c540 0x7fb29406f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T04:27:00.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb29907a640 1 -- 172.21.15.63:0/3027087705 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb2940a30b0 msgr2=0x7fb2941cf740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:27:00.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb29907a640 1 --2- 172.21.15.63:0/3027087705 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb2940a30b0 0x7fb2941cf740 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb284019fa0 tx=0x7fb28401abb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:27:00.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb28a7fc640 1 -- 172.21.15.63:0/3027087705 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2840b0050 con 0x7fb2940a30b0 2024-04-26T04:27:00.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb28a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:27:00.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb28a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:27:00.764 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb28a7fc640 10 monclient: dump: 2024-04-26T04:27:00.764 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:27:00.764 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:27:00.764 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:00.764 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:00.765 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:27:00.765 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:27:00.765 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:27:00.765 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:27:00.765 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:27:00.765 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:00.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb28a7fc640 10 monclient: _finish_auth 0 2024-04-26T04:27:00.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb28a7fc640 10 monclient: _check_auth_tickets 2024-04-26T04:27:00.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb28a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:27:00.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb28a7fc640 1 -- 172.21.15.63:0/3027087705 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2840bc070 con 0x7fb2940a30b0 2024-04-26T04:27:00.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb29badf640 5 monclient: authenticate success, global_id 44196 2024-04-26T04:27:00.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb29badf640 10 monclient: _renew_subs 2024-04-26T04:27:00.766 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb29badf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:27:00.766 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb29badf640 1 -- 172.21.15.63:0/3027087705 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2940726b0 con 0x7fb2940a30b0 2024-04-26T04:27:00.766 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb28a7fc640 10 monclient: handle_config config(0 keys) 2024-04-26T04:27:00.766 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb28a7fc640 1 -- 172.21.15.63:0/3027087705 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2840b5c80 con 0x7fb2940a30b0 2024-04-26T04:27:00.766 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb28a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:27:00.766 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb28a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:27:00.766 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb28a7fc640 10 monclient: dump: 2024-04-26T04:27:00.766 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:27:00.766 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:27:00.766 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:00.766 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:00.766 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:27:00.766 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:27:00.767 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:27:00.767 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:27:00.767 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:27:00.767 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:00.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb28bfff640 4 set_mon_vals no callback set 2024-04-26T04:27:00.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb29badf640 10 monclient: _renew_subs 2024-04-26T04:27:00.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb29badf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:27:00.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb29badf640 1 -- 172.21.15.63:0/3027087705 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2941cfc80 con 0x7fb2940a30b0 2024-04-26T04:27:00.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb29badf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:27:00.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb29badf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:27:00.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb29badf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:27:00.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.745+0000 7fb29badf640 1 -- 172.21.15.63:0/3027087705 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb294062e10 con 0x7fb2940a30b0 2024-04-26T04:27:00.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.749+0000 7fb28a7fc640 1 -- 172.21.15.63:0/3027087705 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fb2840ab4c0 con 0x7fb2940a30b0 2024-04-26T04:27:00.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.749+0000 7fb28a7fc640 1 --2- 172.21.15.63:0/3027087705 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fb278035f10 0x7fb2780383d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:00.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.749+0000 7fb28a7fc640 1 -- 172.21.15.63:0/3027087705 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(323..323 src has 1..323) ==== 4346+0+0 (secure 0 0 0) 0x7fb2840e36a0 con 0x7fb2940a30b0 2024-04-26T04:27:00.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.749+0000 7fb29987b640 1 --2- 172.21.15.63:0/3027087705 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fb278035f10 0x7fb2780383d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:00.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.749+0000 7fb29987b640 10 monclient: get_auth_request con 0x7fb278035f10 auth_method 0 2024-04-26T04:27:00.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.749+0000 7fb29987b640 1 --2- 172.21.15.63:0/3027087705 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fb278035f10 0x7fb2780383d0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fb290007ca0 tx=0x7fb290007bf0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:27:00.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.753+0000 7fb28a7fc640 1 -- 172.21.15.63:0/3027087705 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fb284015ff0 con 0x7fb2940a30b0 2024-04-26T04:27:00.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.753+0000 7fb28a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:27:00.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.753+0000 7fb28a7fc640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:27:00.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.905+0000 7fb29badf640 1 -- 172.21.15.63:0/3027087705 --> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7fb2940a37d0 con 0x7fb278035f10 2024-04-26T04:27:00.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.909+0000 7fb28a7fc640 1 -- 172.21.15.63:0/3027087705 <== mgr.4105 v2:172.21.15.186:6824/2583706639 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fb290007360 con 0x7fb278035f10 2024-04-26T04:27:00.910 INFO:teuthology.orchestra.run.smithi063.stderr:instructed osd(s) 3 to deep-scrub 2024-04-26T04:27:00.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.909+0000 7fb29badf640 1 -- 172.21.15.63:0/3027087705 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fb278035f10 msgr2=0x7fb2780383d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:27:00.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.909+0000 7fb29badf640 1 --2- 172.21.15.63:0/3027087705 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fb278035f10 0x7fb2780383d0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fb290007ca0 tx=0x7fb290007bf0 comp rx=0 tx=0).stop 2024-04-26T04:27:00.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.909+0000 7fb29badf640 10 monclient: shutdown 2024-04-26T04:27:00.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.909+0000 7fb29badf640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:27:00.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.909+0000 7fb29badf640 1 -- 172.21.15.63:0/3027087705 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb2940a30b0 msgr2=0x7fb2941cf740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:27:00.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.909+0000 7fb29badf640 1 --2- 172.21.15.63:0/3027087705 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb2940a30b0 0x7fb2941cf740 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb284019fa0 tx=0x7fb28401abb0 comp rx=0 tx=0).stop 2024-04-26T04:27:00.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.909+0000 7fb29badf640 1 -- 172.21.15.63:0/3027087705 shutdown_connections 2024-04-26T04:27:00.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.909+0000 7fb29badf640 1 --2- 172.21.15.63:0/3027087705 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fb278035f10 0x7fb2780383d0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:00.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.909+0000 7fb29badf640 1 --2- 172.21.15.63:0/3027087705 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb29413c540 0x7fb29406f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:00.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.909+0000 7fb29badf640 1 --2- 172.21.15.63:0/3027087705 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb2940a30b0 0x7fb2941cf740 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:00.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.909+0000 7fb29badf640 1 --2- 172.21.15.63:0/3027087705 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fb2940a2cd0 0x7fb2941cef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:00.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.909+0000 7fb29badf640 1 -- 172.21.15.63:0/3027087705 >> 172.21.15.63:0/3027087705 conn(0x7fb2940a5f40 msgr2=0x7fb2940a1a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:27:00.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.909+0000 7fb29badf640 1 -- 172.21.15.63:0/3027087705 shutdown_connections 2024-04-26T04:27:00.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:00.909+0000 7fb29badf640 1 -- 172.21.15.63:0/3027087705 wait complete. 2024-04-26T04:27:00.927 INFO:tasks.ceph:Scrubbing osd.4 2024-04-26T04:27:00.927 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-04-26T04:27:01.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.053+0000 7fdbc3f7b640 10 monclient: get_monmap_and_config 2024-04-26T04:27:01.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.053+0000 7fdbc3f7b640 10 monclient: build_initial_monmap 2024-04-26T04:27:01.055 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.053+0000 7fdbc3f7b640 10 monclient: monmap: 2024-04-26T04:27:01.056 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:27:01.056 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:27:01.056 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:27:01.055422+0000 2024-04-26T04:27:01.056 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:27:01.055422+0000 2024-04-26T04:27:01.056 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:27:01.056 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:27:01.056 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:27:01.056 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:27:01.057 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:27:01.057 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:27:01.057 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:27:01.057 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:01.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.053+0000 7fdbc3f7b640 1 Processor -- start 2024-04-26T04:27:01.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.053+0000 7fdbc3f7b640 1 -- start start 2024-04-26T04:27:01.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.053+0000 7fdbc3f7b640 10 monclient: init 2024-04-26T04:27:01.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.053+0000 7fdbc3f7b640 10 monclient: _reopen_session rank -1 2024-04-26T04:27:01.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.053+0000 7fdbc3f7b640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-04-26T04:27:01.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.053+0000 7fdbc3f7b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdbbc0a0bb0 addr v1:172.21.15.63:6789/0 2024-04-26T04:27:01.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.053+0000 7fdbc3f7b640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fdbbc073280 0x7fdbbc09dec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:01.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.053+0000 7fdbc3f7b640 10 monclient(hunting): picked mon.noname-c con 0x7fdbbc073280 addr v2:172.21.15.186:3300/0 2024-04-26T04:27:01.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.053+0000 7fdbc3f7b640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7fdbbc09e400 0x7fdbbc13b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:01.059 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.053+0000 7fdbc3f7b640 10 monclient(hunting): picked mon.noname-a con 0x7fdbbc09e400 addr v2:172.21.15.63:3300/0 2024-04-26T04:27:01.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.053+0000 7fdbc3f7b640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbbc09bb80 con 0x7fdbbc0a0bb0 2024-04-26T04:27:01.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.053+0000 7fdbc3f7b640 10 monclient(hunting): start opening mon connection 2024-04-26T04:27:01.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.053+0000 7fdbc3f7b640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7fdbbc09e810 con 0x7fdbbc09e400 2024-04-26T04:27:01.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.053+0000 7fdbc3f7b640 10 monclient(hunting): start opening mon connection 2024-04-26T04:27:01.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.053+0000 7fdbc3f7b640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fdbbc0a0f90 con 0x7fdbbc073280 2024-04-26T04:27:01.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.053+0000 7fdbc3f7b640 10 monclient(hunting): _renew_subs 2024-04-26T04:27:01.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.053+0000 7fdbc3f7b640 10 monclient(hunting): authenticate will time out at 1452.763600s 2024-04-26T04:27:01.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.053+0000 7fdbc2518640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7fdbbc09e400 0x7fdbbc13b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:01.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.053+0000 7fdbc1d17640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7fdbbc0a0bb0 0x7fdbbc09d7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:37570/0 (socket says 172.21.15.63:37570) 2024-04-26T04:27:01.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.053+0000 7fdbc1d17640 1 -- 172.21.15.63:0/3795240636 learned_addr learned my addr 172.21.15.63:0/3795240636 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-04-26T04:27:01.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.053+0000 7fdbc2518640 10 monclient(hunting): get_auth_request con 0x7fdbbc09e400 auth_method 0 2024-04-26T04:27:01.060 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.053+0000 7fdbc2518640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:27:01.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.053+0000 7fdbc2518640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:27:01.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.053+0000 7fdbc2518640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:27:01.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.053+0000 7fdbc2518640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:27:01.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.053+0000 7fdbc2518640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:27:01.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.053+0000 7fdbc2518640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:27:01.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.057+0000 7fdbc1516640 1 --2- 172.21.15.63:0/3795240636 >> v2:172.21.15.186:3300/0 conn(0x7fdbbc073280 0x7fdbbc09dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:01.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.057+0000 7fdbc1516640 10 monclient(hunting): get_auth_request con 0x7fdbbc073280 auth_method 0 2024-04-26T04:27:01.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.057+0000 7fdbc1516640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:27:01.061 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.057+0000 7fdbc1516640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:27:01.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.057+0000 7fdbc1516640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:27:01.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.057+0000 7fdbc2518640 10 monclient(hunting): handle_auth_done global_id 44199 payload 274 2024-04-26T04:27:01.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.057+0000 7fdbc2518640 1 -- 172.21.15.63:0/3795240636 >> v2:172.21.15.186:3300/0 conn(0x7fdbbc073280 msgr2=0x7fdbbc09dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:27:01.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.057+0000 7fdbc2518640 1 --2- 172.21.15.63:0/3795240636 >> v2:172.21.15.186:3300/0 conn(0x7fdbbc073280 0x7fdbbc09dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:01.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.057+0000 7fdbc2518640 1 -- 172.21.15.63:0/3795240636 >> v1:172.21.15.63:6789/0 conn(0x7fdbbc0a0bb0 legacy=0x7fdbbc09d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:27:01.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.057+0000 7fdbc2518640 10 monclient: _finish_hunting 0 2024-04-26T04:27:01.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.057+0000 7fdbc2518640 1 monclient: found mon.noname-a 2024-04-26T04:27:01.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.057+0000 7fdbc2518640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:27:01.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.057+0000 7fdbc2518640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:27:01.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.057+0000 7fdbc2518640 1 -- 172.21.15.63:0/3795240636 --> v2:172.21.15.63:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbbc09f150 con 0x7fdbbc09e400 2024-04-26T04:27:01.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc2518640 1 -- 172.21.15.63:0/3795240636 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdbbc09e400 msgr2=0x7fdbbc13b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:27:01.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc2518640 1 --2- 172.21.15.63:0/3795240636 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdbbc09e400 0x7fdbbc13b340 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc06d500 tx=0x7fdbb401c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4a9cfd49e72976e server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:27:01.066 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc0d15640 1 -- 172.21.15.63:0/3795240636 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbb40b2070 con 0x7fdbbc09e400 2024-04-26T04:27:01.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc0d15640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:27:01.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc0d15640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:27:01.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc0d15640 10 monclient: dump: 2024-04-26T04:27:01.067 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:27:01.067 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:27:01.067 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:01.067 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:01.067 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:27:01.067 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:27:01.067 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:27:01.067 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:27:01.067 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:27:01.068 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:01.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc0d15640 10 monclient: _finish_auth 0 2024-04-26T04:27:01.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc0d15640 10 monclient: _check_auth_tickets 2024-04-26T04:27:01.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc0d15640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:27:01.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc0d15640 1 -- 172.21.15.63:0/3795240636 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdbb40a5810 con 0x7fdbbc09e400 2024-04-26T04:27:01.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc0d15640 10 monclient: handle_config config(0 keys) 2024-04-26T04:27:01.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc0d15640 1 -- 172.21.15.63:0/3795240636 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbb40a5be0 con 0x7fdbbc09e400 2024-04-26T04:27:01.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc3f7b640 5 monclient: authenticate success, global_id 44199 2024-04-26T04:27:01.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc3f7b640 10 monclient: get_monmap_and_config success 2024-04-26T04:27:01.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc3f7b640 4 set_mon_vals no callback set 2024-04-26T04:27:01.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc0d15640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:27:01.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc0d15640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:27:01.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc0d15640 10 monclient: dump: 2024-04-26T04:27:01.069 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:27:01.069 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:27:01.069 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:01.069 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:01.069 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:27:01.069 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:27:01.069 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:27:01.069 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:27:01.069 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:27:01.069 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:01.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc3f7b640 10 monclient: shutdown 2024-04-26T04:27:01.069 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc3f7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:27:01.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc3f7b640 1 -- 172.21.15.63:0/3795240636 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdbbc09e400 msgr2=0x7fdbbc13b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:27:01.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc3f7b640 1 --2- 172.21.15.63:0/3795240636 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdbbc09e400 0x7fdbbc13b340 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc06d500 tx=0x7fdbb401c840 comp rx=0 tx=0).stop 2024-04-26T04:27:01.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc3f7b640 1 -- 172.21.15.63:0/3795240636 shutdown_connections 2024-04-26T04:27:01.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc3f7b640 1 --2- 172.21.15.63:0/3795240636 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdbbc09e400 0x7fdbbc13b340 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:01.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc3f7b640 1 --2- 172.21.15.63:0/3795240636 >> v2:172.21.15.186:3300/0 conn(0x7fdbbc073280 0x7fdbbc09dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:01.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc3f7b640 1 -- 172.21.15.63:0/3795240636 >> 172.21.15.63:0/3795240636 conn(0x7fdbbc0a1ca0 msgr2=0x7fdbbc0a20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:27:01.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc3f7b640 1 -- 172.21.15.63:0/3795240636 shutdown_connections 2024-04-26T04:27:01.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc3f7b640 1 -- 172.21.15.63:0/3795240636 wait complete. 2024-04-26T04:27:01.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc3f7b640 10 monclient: build_initial_monmap 2024-04-26T04:27:01.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc3f7b640 10 monclient: monmap: 2024-04-26T04:27:01.070 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:27:01.070 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:27:01.071 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:27:01.071 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:27:01.071 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:27:01.071 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:27:01.071 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:27:01.071 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:27:01.071 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:27:01.071 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:01.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc3f7b640 1 Processor -- start 2024-04-26T04:27:01.071 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc3f7b640 1 -- start start 2024-04-26T04:27:01.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc3f7b640 10 monclient: init 2024-04-26T04:27:01.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc3f7b640 10 monclient: _reopen_session rank -1 2024-04-26T04:27:01.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc3f7b640 10 monclient: _add_conns ranks=[0,1,2] 2024-04-26T04:27:01.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc3f7b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdbbc073280 0x7fdbbc09c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:01.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc3f7b640 10 monclient(hunting): picked mon.noname-a con 0x7fdbbc073280 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:27:01.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc3f7b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdbbc09e400 0x7fdbbc0984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:01.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc3f7b640 10 monclient(hunting): picked mon.noname-b con 0x7fdbbc09e400 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:27:01.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc3f7b640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fdbbc0a0bb0 0x7fdbbc098a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:01.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc3f7b640 10 monclient(hunting): picked mon.noname-c con 0x7fdbbc0a0bb0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:27:01.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc3f7b640 10 monclient(hunting): start opening mon connection 2024-04-26T04:27:01.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc3f7b640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fdbbc13e430 con 0x7fdbbc073280 2024-04-26T04:27:01.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc3f7b640 10 monclient(hunting): start opening mon connection 2024-04-26T04:27:01.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc3f7b640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fdbbc13e2b0 con 0x7fdbbc09e400 2024-04-26T04:27:01.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc3f7b640 10 monclient(hunting): start opening mon connection 2024-04-26T04:27:01.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc3f7b640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7fdbbc13e5b0 con 0x7fdbbc0a0bb0 2024-04-26T04:27:01.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc3f7b640 10 monclient(hunting): _renew_subs 2024-04-26T04:27:01.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc3f7b640 10 monclient(hunting): authenticate will time out at 1452.771010s 2024-04-26T04:27:01.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc1d17640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdbbc073280 0x7fdbbc09c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:01.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc2518640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fdbbc0a0bb0 0x7fdbbc098a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:01.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc1d17640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdbbc073280 0x7fdbbc09c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:47594/0 (socket says 172.21.15.63:47594) 2024-04-26T04:27:01.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc1d17640 1 -- 172.21.15.63:0/1036757145 learned_addr learned my addr 172.21.15.63:0/1036757145 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:27:01.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc1d17640 10 monclient(hunting): get_auth_request con 0x7fdbbc073280 auth_method 0 2024-04-26T04:27:01.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc1d17640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:27:01.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc1d17640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:27:01.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc1d17640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:27:01.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc2518640 10 monclient(hunting): get_auth_request con 0x7fdbbc0a0bb0 auth_method 0 2024-04-26T04:27:01.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc2518640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:27:01.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc2518640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:27:01.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc2518640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:27:01.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc1d17640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:27:01.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc1d17640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:27:01.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc1d17640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:27:01.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc2518640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:27:01.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc2518640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:27:01.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc2518640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:27:01.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc1516640 1 --2- 172.21.15.63:0/1036757145 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdbbc09e400 0x7fdbbc0984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:01.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc1516640 10 monclient(hunting): get_auth_request con 0x7fdbbc09e400 auth_method 0 2024-04-26T04:27:01.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc1516640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:27:01.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc1516640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:27:01.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc1516640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:27:01.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc1d17640 10 monclient(hunting): handle_auth_done global_id 44202 payload 824 2024-04-26T04:27:01.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc1d17640 1 -- 172.21.15.63:0/1036757145 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fdbbc0a0bb0 msgr2=0x7fdbbc098a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:27:01.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc1d17640 1 --2- 172.21.15.63:0/1036757145 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fdbbc0a0bb0 0x7fdbbc098a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:01.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc1d17640 1 -- 172.21.15.63:0/1036757145 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdbbc09e400 msgr2=0x7fdbbc0984e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:27:01.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc1d17640 1 --2- 172.21.15.63:0/1036757145 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdbbc09e400 0x7fdbbc0984e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:01.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc1d17640 10 monclient: _finish_hunting 0 2024-04-26T04:27:01.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc1d17640 1 monclient: found mon.noname-a 2024-04-26T04:27:01.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc1d17640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:27:01.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc1d17640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:27:01.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc1d17640 1 -- 172.21.15.63:0/1036757145 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbbc1ccc10 con 0x7fdbbc073280 2024-04-26T04:27:01.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc1516640 1 --2- 172.21.15.63:0/1036757145 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdbbc09e400 0x7fdbbc0984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-26T04:27:01.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.061+0000 7fdbc2518640 1 --2- 172.21.15.63:0/1036757145 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fdbbc0a0bb0 0x7fdbbc098a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T04:27:01.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.065+0000 7fdbc1d17640 1 -- 172.21.15.63:0/1036757145 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdbbc073280 msgr2=0x7fdbbc09c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:27:01.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.065+0000 7fdbc1d17640 1 --2- 172.21.15.63:0/1036757145 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdbbc073280 0x7fdbbc09c6a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fdbb800c470 tx=0x7fdbb800c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:27:01.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.065+0000 7fdbb27fc640 1 -- 172.21.15.63:0/1036757145 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbb8007070 con 0x7fdbbc073280 2024-04-26T04:27:01.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.065+0000 7fdbb27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:27:01.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.065+0000 7fdbb27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:27:01.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.065+0000 7fdbb27fc640 10 monclient: dump: 2024-04-26T04:27:01.079 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:27:01.080 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:27:01.080 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:01.080 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:01.080 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:27:01.080 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:27:01.080 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:27:01.080 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:27:01.080 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:27:01.080 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:01.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.065+0000 7fdbb27fc640 10 monclient: _finish_auth 0 2024-04-26T04:27:01.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.065+0000 7fdbb27fc640 10 monclient: _check_auth_tickets 2024-04-26T04:27:01.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.065+0000 7fdbb27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:27:01.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.065+0000 7fdbc3f7b640 5 monclient: authenticate success, global_id 44202 2024-04-26T04:27:01.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.065+0000 7fdbb27fc640 1 -- 172.21.15.63:0/1036757145 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdbb8007210 con 0x7fdbbc073280 2024-04-26T04:27:01.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.065+0000 7fdbb27fc640 10 monclient: handle_config config(0 keys) 2024-04-26T04:27:01.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.065+0000 7fdbb27fc640 1 -- 172.21.15.63:0/1036757145 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbb8007600 con 0x7fdbbc073280 2024-04-26T04:27:01.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.065+0000 7fdbb27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:27:01.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.065+0000 7fdbb27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:27:01.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.065+0000 7fdbb27fc640 10 monclient: dump: 2024-04-26T04:27:01.081 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:27:01.081 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:27:01.081 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:01.081 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:01.081 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:27:01.081 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:27:01.081 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:27:01.082 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:27:01.082 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:27:01.082 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:01.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.065+0000 7fdbb3fff640 4 set_mon_vals no callback set 2024-04-26T04:27:01.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.065+0000 7fdbc3f7b640 10 monclient: _renew_subs 2024-04-26T04:27:01.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.065+0000 7fdbc3f7b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:27:01.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.065+0000 7fdbc3f7b640 1 -- 172.21.15.63:0/1036757145 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbbc0a3b00 con 0x7fdbbc073280 2024-04-26T04:27:01.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.065+0000 7fdbc3f7b640 10 monclient: _renew_subs 2024-04-26T04:27:01.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.065+0000 7fdbc3f7b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:27:01.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.065+0000 7fdbc3f7b640 1 -- 172.21.15.63:0/1036757145 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbbc091d30 con 0x7fdbbc073280 2024-04-26T04:27:01.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.065+0000 7fdbc3f7b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:27:01.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.065+0000 7fdbc3f7b640 1 -- 172.21.15.63:0/1036757145 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fdbbc067fb0 con 0x7fdbbc073280 2024-04-26T04:27:01.083 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.065+0000 7fdbb27fc640 1 -- 172.21.15.63:0/1036757145 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fdbb801b400 con 0x7fdbbc073280 2024-04-26T04:27:01.083 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.065+0000 7fdbb27fc640 1 --2- 172.21.15.63:0/1036757145 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fdba0035f10 0x7fdba00383d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:01.083 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.065+0000 7fdbb27fc640 1 -- 172.21.15.63:0/1036757145 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(323..323 src has 1..323) ==== 4346+0+0 (secure 0 0 0) 0x7fdbb80489c0 con 0x7fdbbc073280 2024-04-26T04:27:01.083 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.069+0000 7fdbb27fc640 1 --2- 172.21.15.63:0/1036757145 >> [v2:172.21.15.186:6800/3746743567,v1:172.21.15.186:6801/3746743567] conn(0x7fdba003bd90 0x7fdba003e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:01.083 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.069+0000 7fdbb27fc640 1 -- 172.21.15.63:0/1036757145 --> [v2:172.21.15.186:6800/3746743567,v1:172.21.15.186:6801/3746743567] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdbb8013e70 con 0x7fdba003bd90 2024-04-26T04:27:01.083 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.069+0000 7fdbb27fc640 1 -- 172.21.15.63:0/1036757145 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_get_version_reply(handle=1 version=323) ==== 24+0+0 (secure 0 0 0) 0x7fdbb804d2a0 con 0x7fdbbc073280 2024-04-26T04:27:01.083 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.069+0000 7fdbb27fc640 10 monclient: handle_get_version_reply finishing 1 version 323 2024-04-26T04:27:01.083 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.069+0000 7fdbc1516640 1 --2- 172.21.15.63:0/1036757145 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fdba0035f10 0x7fdba00383d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:01.083 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.069+0000 7fdbc1516640 10 monclient: get_auth_request con 0x7fdba0035f10 auth_method 0 2024-04-26T04:27:01.083 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.069+0000 7fdbc2518640 1 --2- 172.21.15.63:0/1036757145 >> [v2:172.21.15.186:6800/3746743567,v1:172.21.15.186:6801/3746743567] conn(0x7fdba003bd90 0x7fdba003e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:01.083 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.069+0000 7fdbc2518640 10 monclient: get_auth_request con 0x7fdba003bd90 auth_method 0 2024-04-26T04:27:01.083 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.069+0000 7fdbc1516640 1 --2- 172.21.15.63:0/1036757145 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fdba0035f10 0x7fdba00383d0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc13b830 tx=0x7fdbac008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:27:01.084 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.073+0000 7fdbc2518640 1 --2- 172.21.15.63:0/1036757145 >> [v2:172.21.15.186:6800/3746743567,v1:172.21.15.186:6801/3746743567] conn(0x7fdba003bd90 0x7fdba003e1b0 crc :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:27:01.084 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.073+0000 7fdbb27fc640 1 -- 172.21.15.63:0/1036757145 <== osd.4 v2:172.21.15.186:6800/3746743567 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdbb8013e70 con 0x7fdba003bd90 2024-04-26T04:27:01.090 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.089+0000 7fdbc3f7b640 1 -- 172.21.15.63:0/1036757145 --> [v2:172.21.15.186:6800/3746743567,v1:172.21.15.186:6801/3746743567] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fdbbc062e10 con 0x7fdba003bd90 2024-04-26T04:27:01.090 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.089+0000 7fdbb27fc640 1 -- 172.21.15.63:0/1036757145 <== osd.4 v2:172.21.15.186:6800/3746743567 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fdbbc062e10 con 0x7fdba003bd90 2024-04-26T04:27:01.091 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-26T04:27:01.091 INFO:teuthology.orchestra.run.smithi063.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-26T04:27:01.091 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-26T04:27:01.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.089+0000 7fdbc3f7b640 1 -- 172.21.15.63:0/1036757145 >> [v2:172.21.15.186:6800/3746743567,v1:172.21.15.186:6801/3746743567] conn(0x7fdba003bd90 msgr2=0x7fdba003e1b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:27:01.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.089+0000 7fdbc3f7b640 1 --2- 172.21.15.63:0/1036757145 >> [v2:172.21.15.186:6800/3746743567,v1:172.21.15.186:6801/3746743567] conn(0x7fdba003bd90 0x7fdba003e1b0 crc :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:01.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.089+0000 7fdbc3f7b640 1 -- 172.21.15.63:0/1036757145 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fdba0035f10 msgr2=0x7fdba00383d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:27:01.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.089+0000 7fdbc3f7b640 1 --2- 172.21.15.63:0/1036757145 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fdba0035f10 0x7fdba00383d0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc13b830 tx=0x7fdbac008040 comp rx=0 tx=0).stop 2024-04-26T04:27:01.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.089+0000 7fdbc3f7b640 10 monclient: shutdown 2024-04-26T04:27:01.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.089+0000 7fdbc3f7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:27:01.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.089+0000 7fdbc3f7b640 1 -- 172.21.15.63:0/1036757145 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdbbc073280 msgr2=0x7fdbbc09c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:27:01.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.089+0000 7fdbc3f7b640 1 --2- 172.21.15.63:0/1036757145 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdbbc073280 0x7fdbbc09c6a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fdbb800c470 tx=0x7fdbb800c930 comp rx=0 tx=0).stop 2024-04-26T04:27:01.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.089+0000 7fdbc2518640 1 -- 172.21.15.63:0/1036757145 reap_dead start 2024-04-26T04:27:01.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.089+0000 7fdbc3f7b640 1 -- 172.21.15.63:0/1036757145 shutdown_connections 2024-04-26T04:27:01.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.089+0000 7fdbc3f7b640 1 -- 172.21.15.63:0/1036757145 >> 172.21.15.63:0/1036757145 conn(0x7fdbbc0a1ca0 msgr2=0x7fdbbc099e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:27:01.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.089+0000 7fdbc3f7b640 1 -- 172.21.15.63:0/1036757145 shutdown_connections 2024-04-26T04:27:01.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.089+0000 7fdbc3f7b640 1 -- 172.21.15.63:0/1036757145 wait complete. 2024-04-26T04:27:01.107 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-04-26T04:27:01.214 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.213+0000 7ff7cf067640 10 monclient: get_monmap_and_config 2024-04-26T04:27:01.214 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.213+0000 7ff7cf067640 10 monclient: build_initial_monmap 2024-04-26T04:27:01.215 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.213+0000 7ff7cf067640 10 monclient: monmap: 2024-04-26T04:27:01.215 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:27:01.215 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:27:01.215 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:27:01.214802+0000 2024-04-26T04:27:01.215 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:27:01.214802+0000 2024-04-26T04:27:01.215 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:27:01.215 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:27:01.215 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:27:01.215 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:27:01.215 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:27:01.215 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:27:01.215 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:27:01.215 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:01.218 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.213+0000 7ff7cf067640 1 Processor -- start 2024-04-26T04:27:01.218 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.213+0000 7ff7cf067640 1 -- start start 2024-04-26T04:27:01.218 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.213+0000 7ff7cf067640 10 monclient: init 2024-04-26T04:27:01.218 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.213+0000 7ff7cf067640 10 monclient: _reopen_session rank -1 2024-04-26T04:27:01.218 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.213+0000 7ff7cf067640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-04-26T04:27:01.218 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.213+0000 7ff7cf067640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff7c809b880 0x7ff7c809bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:01.218 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.213+0000 7ff7cf067640 10 monclient(hunting): picked mon.noname-b con 0x7ff7c809b880 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:27:01.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.213+0000 7ff7cf067640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7ff7c8133c00 0x7ff7c8071f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:01.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.213+0000 7ff7cf067640 10 monclient(hunting): picked mon.noname-a con 0x7ff7c8133c00 addr v2:172.21.15.63:3300/0 2024-04-26T04:27:01.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.213+0000 7ff7cf067640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff7c8133fe0 0x7ff7c8072470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:01.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.213+0000 7ff7cf067640 10 monclient(hunting): picked mon.noname-c con 0x7ff7c8133fe0 addr v2:172.21.15.186:3300/0 2024-04-26T04:27:01.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.213+0000 7ff7cf067640 10 monclient(hunting): start opening mon connection 2024-04-26T04:27:01.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.213+0000 7ff7cf067640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7ff7c809c1a0 con 0x7ff7c8133c00 2024-04-26T04:27:01.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.213+0000 7ff7cf067640 10 monclient(hunting): start opening mon connection 2024-04-26T04:27:01.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.213+0000 7ff7cf067640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7ff7c8073730 con 0x7ff7c8133fe0 2024-04-26T04:27:01.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.213+0000 7ff7cf067640 10 monclient(hunting): start opening mon connection 2024-04-26T04:27:01.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.213+0000 7ff7cf067640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7ff7c8099e40 con 0x7ff7c809b880 2024-04-26T04:27:01.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.213+0000 7ff7cf067640 10 monclient(hunting): _renew_subs 2024-04-26T04:27:01.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.213+0000 7ff7cf067640 10 monclient(hunting): authenticate will time out at 1452.922270s 2024-04-26T04:27:01.219 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.213+0000 7ff7c7fff640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7ff7c8133c00 0x7ff7c8071f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:01.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.213+0000 7ff7c7fff640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7ff7c8133c00 0x7ff7c8071f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:47600/0 (socket says 172.21.15.63:47600) 2024-04-26T04:27:01.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.213+0000 7ff7c7fff640 1 -- 172.21.15.63:0/2320595598 learned_addr learned my addr 172.21.15.63:0/2320595598 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:27:01.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.213+0000 7ff7cce03640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff7c809b880 0x7ff7c809bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:01.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.213+0000 7ff7c7fff640 10 monclient(hunting): get_auth_request con 0x7ff7c8133c00 auth_method 0 2024-04-26T04:27:01.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.213+0000 7ff7c7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:27:01.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.213+0000 7ff7c7fff640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:27:01.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.213+0000 7ff7c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:27:01.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.213+0000 7ff7cce03640 10 monclient(hunting): get_auth_request con 0x7ff7c809b880 auth_method 0 2024-04-26T04:27:01.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.213+0000 7ff7cce03640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:27:01.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.213+0000 7ff7cce03640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:27:01.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.213+0000 7ff7cce03640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:27:01.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.213+0000 7ff7c7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:27:01.220 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.213+0000 7ff7c7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:27:01.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.213+0000 7ff7c7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:27:01.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.213+0000 7ff7cce03640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:27:01.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.213+0000 7ff7cce03640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:27:01.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.213+0000 7ff7cce03640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:27:01.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.213+0000 7ff7cd604640 1 --2- 172.21.15.63:0/2320595598 >> v2:172.21.15.186:3300/0 conn(0x7ff7c8133fe0 0x7ff7c8072470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:01.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.213+0000 7ff7c7fff640 10 monclient(hunting): handle_auth_done global_id 44205 payload 274 2024-04-26T04:27:01.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.213+0000 7ff7c7fff640 1 -- 172.21.15.63:0/2320595598 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff7c809b880 msgr2=0x7ff7c809bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:27:01.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.213+0000 7ff7c7fff640 1 --2- 172.21.15.63:0/2320595598 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff7c809b880 0x7ff7c809bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:01.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.213+0000 7ff7c7fff640 1 -- 172.21.15.63:0/2320595598 >> v2:172.21.15.186:3300/0 conn(0x7ff7c8133fe0 msgr2=0x7ff7c8072470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:27:01.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.213+0000 7ff7c7fff640 1 --2- 172.21.15.63:0/2320595598 >> v2:172.21.15.186:3300/0 conn(0x7ff7c8133fe0 0x7ff7c8072470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:01.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.213+0000 7ff7c7fff640 10 monclient: _finish_hunting 0 2024-04-26T04:27:01.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.213+0000 7ff7c7fff640 1 monclient: found mon.noname-a 2024-04-26T04:27:01.221 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.213+0000 7ff7c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:27:01.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.213+0000 7ff7c7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:27:01.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.213+0000 7ff7c7fff640 1 -- 172.21.15.63:0/2320595598 --> v2:172.21.15.63:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7c80998f0 con 0x7ff7c8133c00 2024-04-26T04:27:01.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.213+0000 7ff7cd604640 10 monclient: get_auth_request con 0x7ff7c8133fe0 auth_method 0 2024-04-26T04:27:01.222 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.213+0000 7ff7cd604640 1 --2- 172.21.15.63:0/2320595598 >> v2:172.21.15.186:3300/0 conn(0x7ff7c8133fe0 0x7ff7c8072470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T04:27:01.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.217+0000 7ff7c7fff640 1 -- 172.21.15.63:0/2320595598 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff7c8133c00 msgr2=0x7ff7c8071f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:27:01.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.217+0000 7ff7c7fff640 1 --2- 172.21.15.63:0/2320595598 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff7c8133c00 0x7ff7c8071f30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff7c806bef0 tx=0x7ff7bc01c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fcede0b0d6d7ab3 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:27:01.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.217+0000 7ff7c77fe640 1 -- 172.21.15.63:0/2320595598 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7bc0b0070 con 0x7ff7c8133c00 2024-04-26T04:27:01.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.217+0000 7ff7c77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:27:01.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.217+0000 7ff7c77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:27:01.227 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.217+0000 7ff7c77fe640 10 monclient: dump: 2024-04-26T04:27:01.227 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:27:01.227 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:27:01.227 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:01.227 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:01.228 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:27:01.228 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:27:01.228 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:27:01.228 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:27:01.228 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:27:01.228 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:01.228 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.217+0000 7ff7c77fe640 10 monclient: _finish_auth 0 2024-04-26T04:27:01.228 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.217+0000 7ff7c77fe640 10 monclient: _check_auth_tickets 2024-04-26T04:27:01.228 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.217+0000 7ff7c77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:27:01.228 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.217+0000 7ff7c77fe640 1 -- 172.21.15.63:0/2320595598 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7bc0ab600 con 0x7ff7c8133c00 2024-04-26T04:27:01.228 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.217+0000 7ff7c77fe640 10 monclient: handle_config config(0 keys) 2024-04-26T04:27:01.228 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.217+0000 7ff7c77fe640 1 -- 172.21.15.63:0/2320595598 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7bc0ab9d0 con 0x7ff7c8133c00 2024-04-26T04:27:01.228 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.217+0000 7ff7cf067640 5 monclient: authenticate success, global_id 44205 2024-04-26T04:27:01.229 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.217+0000 7ff7cf067640 10 monclient: get_monmap_and_config success 2024-04-26T04:27:01.229 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.217+0000 7ff7cf067640 4 set_mon_vals no callback set 2024-04-26T04:27:01.229 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.217+0000 7ff7c77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:27:01.229 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.217+0000 7ff7c77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:27:01.229 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.217+0000 7ff7c77fe640 10 monclient: dump: 2024-04-26T04:27:01.229 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:27:01.229 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:27:01.229 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:01.229 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:01.229 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:27:01.229 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:27:01.229 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:27:01.229 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:27:01.229 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:27:01.229 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:01.230 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.217+0000 7ff7cf067640 10 monclient: shutdown 2024-04-26T04:27:01.230 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.217+0000 7ff7cf067640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:27:01.230 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.217+0000 7ff7cf067640 1 -- 172.21.15.63:0/2320595598 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff7c8133c00 msgr2=0x7ff7c8071f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:27:01.230 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.217+0000 7ff7cf067640 1 --2- 172.21.15.63:0/2320595598 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff7c8133c00 0x7ff7c8071f30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff7c806bef0 tx=0x7ff7bc01c3b0 comp rx=0 tx=0).stop 2024-04-26T04:27:01.230 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.217+0000 7ff7cf067640 1 -- 172.21.15.63:0/2320595598 shutdown_connections 2024-04-26T04:27:01.230 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.217+0000 7ff7cf067640 1 --2- 172.21.15.63:0/2320595598 >> v2:172.21.15.186:3300/0 conn(0x7ff7c8133fe0 0x7ff7c8072470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:01.230 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.217+0000 7ff7cf067640 1 --2- 172.21.15.63:0/2320595598 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff7c8133c00 0x7ff7c8071f30 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:01.230 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.217+0000 7ff7cf067640 1 --2- 172.21.15.63:0/2320595598 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff7c809b880 0x7ff7c809bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:01.230 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.217+0000 7ff7cf067640 1 -- 172.21.15.63:0/2320595598 >> 172.21.15.63:0/2320595598 conn(0x7ff7c80a1e70 msgr2=0x7ff7c80a2280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:27:01.230 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.217+0000 7ff7cf067640 1 -- 172.21.15.63:0/2320595598 shutdown_connections 2024-04-26T04:27:01.230 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.217+0000 7ff7cf067640 1 -- 172.21.15.63:0/2320595598 wait complete. 2024-04-26T04:27:01.230 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.221+0000 7ff7cf067640 10 monclient: build_initial_monmap 2024-04-26T04:27:01.230 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.221+0000 7ff7cf067640 10 monclient: monmap: 2024-04-26T04:27:01.230 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:27:01.231 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:27:01.231 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:27:01.231 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:27:01.231 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:27:01.231 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:27:01.231 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:27:01.231 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:27:01.231 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:27:01.231 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:01.231 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.221+0000 7ff7cf067640 1 Processor -- start 2024-04-26T04:27:01.231 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.221+0000 7ff7cf067640 1 -- start start 2024-04-26T04:27:01.231 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.221+0000 7ff7cf067640 10 monclient: init 2024-04-26T04:27:01.231 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.221+0000 7ff7cf067640 10 monclient: _reopen_session rank -1 2024-04-26T04:27:01.231 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.221+0000 7ff7cf067640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T04:27:01.231 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.221+0000 7ff7cf067640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff7c809b880 0x7ff7c813d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:01.232 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.221+0000 7ff7cf067640 10 monclient(hunting): picked mon.noname-c con 0x7ff7c809b880 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:27:01.232 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.221+0000 7ff7cf067640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff7c8133c00 0x7ff7c81cedc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:01.232 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.221+0000 7ff7cf067640 10 monclient(hunting): picked mon.noname-b con 0x7ff7c8133c00 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:27:01.232 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.221+0000 7ff7cf067640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff7c8133fe0 0x7ff7c81cf300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:01.232 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.221+0000 7ff7cf067640 10 monclient(hunting): picked mon.noname-a con 0x7ff7c8133fe0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:27:01.232 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.221+0000 7ff7cf067640 10 monclient(hunting): start opening mon connection 2024-04-26T04:27:01.232 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.221+0000 7ff7cf067640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff7c8140980 con 0x7ff7c8133fe0 2024-04-26T04:27:01.232 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.221+0000 7ff7cf067640 10 monclient(hunting): start opening mon connection 2024-04-26T04:27:01.232 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.221+0000 7ff7cf067640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff7c8140800 con 0x7ff7c8133c00 2024-04-26T04:27:01.232 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.221+0000 7ff7cf067640 10 monclient(hunting): start opening mon connection 2024-04-26T04:27:01.232 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.221+0000 7ff7cf067640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7ff7c8140b00 con 0x7ff7c809b880 2024-04-26T04:27:01.232 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.221+0000 7ff7cf067640 10 monclient(hunting): _renew_subs 2024-04-26T04:27:01.232 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.221+0000 7ff7cf067640 10 monclient(hunting): authenticate will time out at 1452.929231s 2024-04-26T04:27:01.232 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.221+0000 7ff7cce03640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff7c809b880 0x7ff7c813d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:01.232 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.221+0000 7ff7cd604640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff7c8133fe0 0x7ff7c81cf300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:01.233 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.221+0000 7ff7cce03640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff7c809b880 0x7ff7c813d810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:39924/0 (socket says 172.21.15.63:39924) 2024-04-26T04:27:01.233 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.221+0000 7ff7cce03640 1 -- 172.21.15.63:0/2527772885 learned_addr learned my addr 172.21.15.63:0/2527772885 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:27:01.233 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.221+0000 7ff7cce03640 10 monclient(hunting): get_auth_request con 0x7ff7c809b880 auth_method 0 2024-04-26T04:27:01.233 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.221+0000 7ff7cd604640 1 --2- 172.21.15.63:0/2527772885 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff7c8133fe0 0x7ff7c81cf300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:47614/0 (socket says 172.21.15.63:47614) 2024-04-26T04:27:01.233 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.221+0000 7ff7cce03640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:27:01.233 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.221+0000 7ff7cce03640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:27:01.233 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.221+0000 7ff7cce03640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:27:01.233 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.221+0000 7ff7cd604640 10 monclient(hunting): get_auth_request con 0x7ff7c8133fe0 auth_method 0 2024-04-26T04:27:01.233 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.221+0000 7ff7cd604640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:27:01.233 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.221+0000 7ff7cd604640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:27:01.233 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.221+0000 7ff7cd604640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:27:01.233 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.221+0000 7ff7cd604640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:27:01.233 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.221+0000 7ff7cd604640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:27:01.233 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.221+0000 7ff7cd604640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:27:01.233 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.221+0000 7ff7cce03640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:27:01.234 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.221+0000 7ff7cce03640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:27:01.234 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.221+0000 7ff7cce03640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:27:01.234 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.221+0000 7ff7c7fff640 1 --2- 172.21.15.63:0/2527772885 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff7c8133c00 0x7ff7c81cedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:01.234 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.221+0000 7ff7cd604640 10 monclient(hunting): handle_auth_done global_id 44208 payload 824 2024-04-26T04:27:01.234 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.221+0000 7ff7cd604640 1 -- 172.21.15.63:0/2527772885 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff7c809b880 msgr2=0x7ff7c813d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:27:01.234 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.221+0000 7ff7cd604640 1 --2- 172.21.15.63:0/2527772885 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff7c809b880 0x7ff7c813d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:01.234 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.221+0000 7ff7cd604640 1 -- 172.21.15.63:0/2527772885 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff7c8133c00 msgr2=0x7ff7c81cedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:27:01.234 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.221+0000 7ff7cd604640 1 --2- 172.21.15.63:0/2527772885 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff7c8133c00 0x7ff7c81cedc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:01.234 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.221+0000 7ff7cd604640 10 monclient: _finish_hunting 0 2024-04-26T04:27:01.234 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.221+0000 7ff7cd604640 1 monclient: found mon.noname-a 2024-04-26T04:27:01.234 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.221+0000 7ff7cd604640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:27:01.234 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.221+0000 7ff7cd604640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:27:01.234 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.221+0000 7ff7cd604640 1 -- 172.21.15.63:0/2527772885 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7c813c9e0 con 0x7ff7c8133fe0 2024-04-26T04:27:01.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.221+0000 7ff7c7fff640 10 monclient: get_auth_request con 0x7ff7c8133c00 auth_method 0 2024-04-26T04:27:01.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.221+0000 7ff7c7fff640 1 --2- 172.21.15.63:0/2527772885 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff7c8133c00 0x7ff7c81cedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T04:27:01.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.221+0000 7ff7cd604640 1 -- 172.21.15.63:0/2527772885 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff7c8133fe0 msgr2=0x7ff7c81cf300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:27:01.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.221+0000 7ff7cd604640 1 --2- 172.21.15.63:0/2527772885 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff7c8133fe0 0x7ff7c81cf300 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff7c000cd30 tx=0x7ff7c000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:27:01.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.225+0000 7ff7c57fa640 1 -- 172.21.15.63:0/2527772885 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c0003ba0 con 0x7ff7c8133fe0 2024-04-26T04:27:01.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.225+0000 7ff7c57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:27:01.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.225+0000 7ff7c57fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:27:01.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.225+0000 7ff7c57fa640 10 monclient: dump: 2024-04-26T04:27:01.235 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:27:01.235 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:27:01.235 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:01.235 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:01.235 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:27:01.235 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:27:01.235 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:27:01.235 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:27:01.236 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:27:01.236 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:01.236 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.225+0000 7ff7c57fa640 10 monclient: _finish_auth 0 2024-04-26T04:27:01.236 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.225+0000 7ff7c57fa640 10 monclient: _check_auth_tickets 2024-04-26T04:27:01.236 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.225+0000 7ff7c57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:27:01.236 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.225+0000 7ff7c57fa640 1 -- 172.21.15.63:0/2527772885 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7c0003d60 con 0x7ff7c8133fe0 2024-04-26T04:27:01.236 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.225+0000 7ff7c57fa640 10 monclient: handle_config config(0 keys) 2024-04-26T04:27:01.236 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.225+0000 7ff7c57fa640 1 -- 172.21.15.63:0/2527772885 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c000da90 con 0x7ff7c8133fe0 2024-04-26T04:27:01.236 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.225+0000 7ff7cf067640 5 monclient: authenticate success, global_id 44208 2024-04-26T04:27:01.236 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.225+0000 7ff7cf067640 10 monclient: _renew_subs 2024-04-26T04:27:01.236 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.225+0000 7ff7cf067640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:27:01.236 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.225+0000 7ff7cf067640 1 -- 172.21.15.63:0/2527772885 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7c813cc50 con 0x7ff7c8133fe0 2024-04-26T04:27:01.236 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.225+0000 7ff7c57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:27:01.236 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.225+0000 7ff7c57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:27:01.237 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.225+0000 7ff7c57fa640 10 monclient: dump: 2024-04-26T04:27:01.237 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:27:01.237 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:27:01.237 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:01.237 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:01.237 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:27:01.237 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:27:01.237 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:27:01.237 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:27:01.237 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:27:01.237 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:01.237 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.225+0000 7ff7c6ffd640 4 set_mon_vals no callback set 2024-04-26T04:27:01.237 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.225+0000 7ff7cf067640 10 monclient: _renew_subs 2024-04-26T04:27:01.237 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.225+0000 7ff7cf067640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:27:01.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.225+0000 7ff7cf067640 1 -- 172.21.15.63:0/2527772885 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7c8139270 con 0x7ff7c8133fe0 2024-04-26T04:27:01.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.225+0000 7ff7cf067640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:27:01.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.225+0000 7ff7cf067640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:27:01.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.225+0000 7ff7cf067640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:27:01.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.225+0000 7ff7c57fa640 1 -- 172.21.15.63:0/2527772885 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7ff7c000dcf0 con 0x7ff7c8133fe0 2024-04-26T04:27:01.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.225+0000 7ff7cf067640 1 -- 172.21.15.63:0/2527772885 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7c8062e10 con 0x7ff7c8133fe0 2024-04-26T04:27:01.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.225+0000 7ff7c57fa640 1 --2- 172.21.15.63:0/2527772885 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7ff7b4035f10 0x7ff7b40383d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:01.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.225+0000 7ff7c57fa640 1 -- 172.21.15.63:0/2527772885 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(323..323 src has 1..323) ==== 4346+0+0 (secure 0 0 0) 0x7ff7c0049600 con 0x7ff7c8133fe0 2024-04-26T04:27:01.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.225+0000 7ff7cce03640 1 --2- 172.21.15.63:0/2527772885 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7ff7b4035f10 0x7ff7b40383d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:01.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.225+0000 7ff7cce03640 10 monclient: get_auth_request con 0x7ff7b4035f10 auth_method 0 2024-04-26T04:27:01.239 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.229+0000 7ff7cce03640 1 --2- 172.21.15.63:0/2527772885 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7ff7b4035f10 0x7ff7b40383d0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7ff7b8009c20 tx=0x7ff7b8009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:27:01.239 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.229+0000 7ff7c57fa640 1 -- 172.21.15.63:0/2527772885 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7ff7c000ae20 con 0x7ff7c8133fe0 2024-04-26T04:27:01.239 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.229+0000 7ff7c57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:27:01.239 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.229+0000 7ff7c57fa640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:27:01.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.385+0000 7ff7cf067640 1 -- 172.21.15.63:0/2527772885 --> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7ff7c8133450 con 0x7ff7b4035f10 2024-04-26T04:27:01.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.385+0000 7ff7c57fa640 1 -- 172.21.15.63:0/2527772885 <== mgr.4105 v2:172.21.15.186:6824/2583706639 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7ff7c8133450 con 0x7ff7b4035f10 2024-04-26T04:27:01.388 INFO:teuthology.orchestra.run.smithi063.stderr:instructed osd(s) 4 to deep-scrub 2024-04-26T04:27:01.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.389+0000 7ff7cf067640 1 -- 172.21.15.63:0/2527772885 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7ff7b4035f10 msgr2=0x7ff7b40383d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:27:01.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.389+0000 7ff7cf067640 1 --2- 172.21.15.63:0/2527772885 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7ff7b4035f10 0x7ff7b40383d0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7ff7b8009c20 tx=0x7ff7b8009190 comp rx=0 tx=0).stop 2024-04-26T04:27:01.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.389+0000 7ff7cf067640 10 monclient: shutdown 2024-04-26T04:27:01.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.389+0000 7ff7cf067640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:27:01.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.389+0000 7ff7cf067640 1 -- 172.21.15.63:0/2527772885 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff7c8133fe0 msgr2=0x7ff7c81cf300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:27:01.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.389+0000 7ff7cf067640 1 --2- 172.21.15.63:0/2527772885 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff7c8133fe0 0x7ff7c81cf300 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff7c000cd30 tx=0x7ff7c000a710 comp rx=0 tx=0).stop 2024-04-26T04:27:01.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.389+0000 7ff7cf067640 1 -- 172.21.15.63:0/2527772885 shutdown_connections 2024-04-26T04:27:01.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.389+0000 7ff7cf067640 1 --2- 172.21.15.63:0/2527772885 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7ff7b4035f10 0x7ff7b40383d0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:01.391 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.389+0000 7ff7cf067640 1 --2- 172.21.15.63:0/2527772885 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff7c8133fe0 0x7ff7c81cf300 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:01.391 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.389+0000 7ff7cf067640 1 --2- 172.21.15.63:0/2527772885 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff7c8133c00 0x7ff7c81cedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:01.391 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.389+0000 7ff7cf067640 1 --2- 172.21.15.63:0/2527772885 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7ff7c809b880 0x7ff7c813d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:01.391 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.389+0000 7ff7cf067640 1 -- 172.21.15.63:0/2527772885 >> 172.21.15.63:0/2527772885 conn(0x7ff7c80a1e70 msgr2=0x7ff7c809e5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:27:01.391 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.389+0000 7ff7cf067640 1 -- 172.21.15.63:0/2527772885 shutdown_connections 2024-04-26T04:27:01.391 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.389+0000 7ff7cf067640 1 -- 172.21.15.63:0/2527772885 wait complete. 2024-04-26T04:27:01.405 INFO:tasks.ceph:Scrubbing osd.5 2024-04-26T04:27:01.405 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-04-26T04:27:01.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.529+0000 7f44b3420640 10 monclient: get_monmap_and_config 2024-04-26T04:27:01.533 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.529+0000 7f44b3420640 10 monclient: build_initial_monmap 2024-04-26T04:27:01.533 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.529+0000 7f44b3420640 10 monclient: monmap: 2024-04-26T04:27:01.533 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:27:01.533 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:27:01.533 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:27:01.533014+0000 2024-04-26T04:27:01.533 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:27:01.533014+0000 2024-04-26T04:27:01.533 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:27:01.533 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:27:01.533 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:27:01.533 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:27:01.533 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:27:01.533 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:27:01.533 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:27:01.534 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:01.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.533+0000 7f44b3420640 1 Processor -- start 2024-04-26T04:27:01.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.533+0000 7f44b3420640 1 -- start start 2024-04-26T04:27:01.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.533+0000 7f44b3420640 10 monclient: init 2024-04-26T04:27:01.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.533+0000 7f44b3420640 10 monclient: _reopen_session rank -1 2024-04-26T04:27:01.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.533+0000 7f44b3420640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-04-26T04:27:01.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.533+0000 7f44b3420640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f44ac0a2980 0x7f44ac0a2d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:01.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.533+0000 7f44b3420640 10 monclient(hunting): picked mon.noname-b con 0x7f44ac0a2980 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:27:01.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.533+0000 7f44b3420640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f44ac0a1f30 0x7f44ac0a3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:01.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.533+0000 7f44b3420640 10 monclient(hunting): picked mon.noname-a con 0x7f44ac0a1f30 addr v2:172.21.15.63:3300/0 2024-04-26T04:27:01.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.533+0000 7f44b3420640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f44ac0a2310 0x7f44ac0a43f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:01.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.533+0000 7f44b3420640 10 monclient(hunting): picked mon.noname-c con 0x7f44ac0a2310 addr v2:172.21.15.186:3300/0 2024-04-26T04:27:01.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.533+0000 7f44b3420640 10 monclient(hunting): start opening mon connection 2024-04-26T04:27:01.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.533+0000 7f44b3420640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7f44ac0a0cd0 con 0x7f44ac0a1f30 2024-04-26T04:27:01.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.533+0000 7f44b3420640 10 monclient(hunting): start opening mon connection 2024-04-26T04:27:01.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.533+0000 7f44b3420640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f44ac0a0290 con 0x7f44ac0a2310 2024-04-26T04:27:01.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.533+0000 7f44b3420640 10 monclient(hunting): start opening mon connection 2024-04-26T04:27:01.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.533+0000 7f44b3420640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f44ac0a0fb0 con 0x7f44ac0a2980 2024-04-26T04:27:01.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.533+0000 7f44b3420640 10 monclient(hunting): _renew_subs 2024-04-26T04:27:01.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.533+0000 7f44b3420640 10 monclient(hunting): authenticate will time out at 1453.241155s 2024-04-26T04:27:01.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.533+0000 7f44b09bb640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f44ac0a1f30 0x7f44ac0a3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:01.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.533+0000 7f44b11bc640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f44ac0a2980 0x7f44ac0a2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:01.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.533+0000 7f44b09bb640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f44ac0a1f30 0x7f44ac0a3eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:47624/0 (socket says 172.21.15.63:47624) 2024-04-26T04:27:01.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.533+0000 7f44b11bc640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f44ac0a2980 0x7f44ac0a2d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:39930/0 (socket says 172.21.15.63:39930) 2024-04-26T04:27:01.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.533+0000 7f44b09bb640 1 -- 172.21.15.63:0/72710417 learned_addr learned my addr 172.21.15.63:0/72710417 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:27:01.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.533+0000 7f44b09bb640 10 monclient(hunting): get_auth_request con 0x7f44ac0a1f30 auth_method 0 2024-04-26T04:27:01.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.533+0000 7f44b09bb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:27:01.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.533+0000 7f44b09bb640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:27:01.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.533+0000 7f44b09bb640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:27:01.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.533+0000 7f44b11bc640 10 monclient(hunting): get_auth_request con 0x7f44ac0a2980 auth_method 0 2024-04-26T04:27:01.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.533+0000 7f44b11bc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:27:01.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.533+0000 7f44b11bc640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:27:01.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.533+0000 7f44b11bc640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:27:01.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.533+0000 7f44b09bb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:27:01.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.533+0000 7f44b09bb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:27:01.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.533+0000 7f44b09bb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:27:01.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.533+0000 7f44b11bc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:27:01.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.533+0000 7f44b11bc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:27:01.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.533+0000 7f44b11bc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:27:01.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.533+0000 7f44b09bb640 10 monclient(hunting): handle_auth_done global_id 44211 payload 274 2024-04-26T04:27:01.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.533+0000 7f44b19bd640 1 --2- 172.21.15.63:0/72710417 >> v2:172.21.15.186:3300/0 conn(0x7f44ac0a2310 0x7f44ac0a43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:01.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.533+0000 7f44b09bb640 1 -- 172.21.15.63:0/72710417 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f44ac0a2980 msgr2=0x7f44ac0a2d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:27:01.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.533+0000 7f44b09bb640 1 --2- 172.21.15.63:0/72710417 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f44ac0a2980 0x7f44ac0a2d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:01.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.533+0000 7f44b09bb640 1 -- 172.21.15.63:0/72710417 >> v2:172.21.15.186:3300/0 conn(0x7f44ac0a2310 msgr2=0x7f44ac0a43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:27:01.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.533+0000 7f44b09bb640 1 --2- 172.21.15.63:0/72710417 >> v2:172.21.15.186:3300/0 conn(0x7f44ac0a2310 0x7f44ac0a43f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:01.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.533+0000 7f44b09bb640 10 monclient: _finish_hunting 0 2024-04-26T04:27:01.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.533+0000 7f44b09bb640 1 monclient: found mon.noname-a 2024-04-26T04:27:01.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.533+0000 7f44b09bb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:27:01.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.533+0000 7f44b09bb640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:27:01.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.533+0000 7f44b09bb640 1 -- 172.21.15.63:0/72710417 --> v2:172.21.15.63:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44ac0a6d30 con 0x7f44ac0a1f30 2024-04-26T04:27:01.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.533+0000 7f44b19bd640 10 monclient: get_auth_request con 0x7f44ac0a2310 auth_method 0 2024-04-26T04:27:01.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.533+0000 7f44b19bd640 1 --2- 172.21.15.63:0/72710417 >> v2:172.21.15.186:3300/0 conn(0x7f44ac0a2310 0x7f44ac0a43f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T04:27:01.543 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.537+0000 7f44b09bb640 1 -- 172.21.15.63:0/72710417 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f44ac0a1f30 msgr2=0x7f44ac0a3eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:27:01.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.537+0000 7f44b09bb640 1 --2- 172.21.15.63:0/72710417 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f44ac0a1f30 0x7f44ac0a3eb0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f44ac0a1990 tx=0x7f449c01c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67e901c0c541b5f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:27:01.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.537+0000 7f44a3fff640 1 -- 172.21.15.63:0/72710417 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f449c0b0070 con 0x7f44ac0a1f30 2024-04-26T04:27:01.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.537+0000 7f44a3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:27:01.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.537+0000 7f44a3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:27:01.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.537+0000 7f44a3fff640 10 monclient: dump: 2024-04-26T04:27:01.544 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:27:01.544 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:27:01.544 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:01.544 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:01.544 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:27:01.544 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:27:01.544 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:27:01.545 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:27:01.545 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:27:01.545 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:01.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.537+0000 7f44a3fff640 10 monclient: _finish_auth 0 2024-04-26T04:27:01.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.537+0000 7f44a3fff640 10 monclient: _check_auth_tickets 2024-04-26T04:27:01.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.537+0000 7f44a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:27:01.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.537+0000 7f44a3fff640 1 -- 172.21.15.63:0/72710417 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f449c0a9710 con 0x7f44ac0a1f30 2024-04-26T04:27:01.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.537+0000 7f44b3420640 5 monclient: authenticate success, global_id 44211 2024-04-26T04:27:01.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.537+0000 7f44b3420640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T04:27:01.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.537+0000 7f44a3fff640 10 monclient: handle_config config(0 keys) 2024-04-26T04:27:01.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.537+0000 7f44b3420640 10 monclient: get_monmap_and_config success 2024-04-26T04:27:01.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.537+0000 7f44b3420640 4 set_mon_vals no callback set 2024-04-26T04:27:01.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.537+0000 7f44a3fff640 1 -- 172.21.15.63:0/72710417 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f449c0a9ae0 con 0x7f44ac0a1f30 2024-04-26T04:27:01.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.537+0000 7f44a3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:27:01.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.537+0000 7f44a3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:27:01.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.537+0000 7f44a3fff640 10 monclient: dump: 2024-04-26T04:27:01.546 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:27:01.546 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:27:01.546 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:01.546 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:01.546 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:27:01.546 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:27:01.546 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:27:01.546 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:27:01.546 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:27:01.546 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:01.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.537+0000 7f44b3420640 10 monclient: shutdown 2024-04-26T04:27:01.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.537+0000 7f44b3420640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:27:01.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.537+0000 7f44b3420640 1 -- 172.21.15.63:0/72710417 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f44ac0a1f30 msgr2=0x7f44ac0a3eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:27:01.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.537+0000 7f44b3420640 1 --2- 172.21.15.63:0/72710417 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f44ac0a1f30 0x7f44ac0a3eb0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f44ac0a1990 tx=0x7f449c01c3e0 comp rx=0 tx=0).stop 2024-04-26T04:27:01.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.537+0000 7f44b3420640 1 -- 172.21.15.63:0/72710417 shutdown_connections 2024-04-26T04:27:01.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.537+0000 7f44b3420640 1 --2- 172.21.15.63:0/72710417 >> v2:172.21.15.186:3300/0 conn(0x7f44ac0a2310 0x7f44ac0a43f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:01.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.537+0000 7f44b3420640 1 --2- 172.21.15.63:0/72710417 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f44ac0a1f30 0x7f44ac0a3eb0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:01.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.537+0000 7f44b3420640 1 --2- 172.21.15.63:0/72710417 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f44ac0a2980 0x7f44ac0a2d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:01.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.537+0000 7f44b3420640 1 -- 172.21.15.63:0/72710417 >> 172.21.15.63:0/72710417 conn(0x7f44ac0a5fb0 msgr2=0x7f44ac0a63c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:27:01.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.537+0000 7f44b3420640 1 -- 172.21.15.63:0/72710417 shutdown_connections 2024-04-26T04:27:01.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.537+0000 7f44b3420640 1 -- 172.21.15.63:0/72710417 wait complete. 2024-04-26T04:27:01.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.537+0000 7f44b3420640 10 monclient: build_initial_monmap 2024-04-26T04:27:01.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.537+0000 7f44b3420640 10 monclient: monmap: 2024-04-26T04:27:01.547 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:27:01.547 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:27:01.547 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:27:01.548 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:27:01.548 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:27:01.548 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:27:01.548 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:27:01.548 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:27:01.548 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:27:01.548 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:01.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.537+0000 7f44b3420640 1 Processor -- start 2024-04-26T04:27:01.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.537+0000 7f44b3420640 1 -- start start 2024-04-26T04:27:01.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.537+0000 7f44b3420640 10 monclient: init 2024-04-26T04:27:01.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44b3420640 10 monclient: _reopen_session rank -1 2024-04-26T04:27:01.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44b3420640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T04:27:01.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44b3420640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f44ac0a1f30 0x7f44ac13cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:01.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44b3420640 10 monclient(hunting): picked mon.noname-b con 0x7f44ac0a1f30 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:27:01.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44b3420640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f44ac0a2310 0x7f44ac136fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:01.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44b3420640 10 monclient(hunting): picked mon.noname-c con 0x7f44ac0a2310 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:27:01.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44b3420640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f44ac0a2980 0x7f44ac1cf020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:01.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44b3420640 10 monclient(hunting): picked mon.noname-a con 0x7f44ac0a2980 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:27:01.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44b3420640 10 monclient(hunting): start opening mon connection 2024-04-26T04:27:01.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44b3420640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f44ac140b90 con 0x7f44ac0a2980 2024-04-26T04:27:01.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44b3420640 10 monclient(hunting): start opening mon connection 2024-04-26T04:27:01.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44b3420640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f44ac140a10 con 0x7f44ac0a1f30 2024-04-26T04:27:01.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44b3420640 10 monclient(hunting): start opening mon connection 2024-04-26T04:27:01.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44b3420640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f44ac140d10 con 0x7f44ac0a2310 2024-04-26T04:27:01.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44b3420640 10 monclient(hunting): _renew_subs 2024-04-26T04:27:01.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44b3420640 10 monclient(hunting): authenticate will time out at 1453.248302s 2024-04-26T04:27:01.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44b09bb640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f44ac0a2310 0x7f44ac136fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:01.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44b19bd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f44ac0a2980 0x7f44ac1cf020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:01.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44b09bb640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f44ac0a2310 0x7f44ac136fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:39944/0 (socket says 172.21.15.63:39944) 2024-04-26T04:27:01.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44b09bb640 1 -- 172.21.15.63:0/1091280335 learned_addr learned my addr 172.21.15.63:0/1091280335 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:27:01.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44b09bb640 10 monclient(hunting): get_auth_request con 0x7f44ac0a2310 auth_method 0 2024-04-26T04:27:01.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44b09bb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:27:01.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44b09bb640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:27:01.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44b09bb640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:27:01.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44b19bd640 10 monclient(hunting): get_auth_request con 0x7f44ac0a2980 auth_method 0 2024-04-26T04:27:01.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44b19bd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:27:01.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44b19bd640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:27:01.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44b19bd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:27:01.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44b19bd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:27:01.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44b19bd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:27:01.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44b19bd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:27:01.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44b09bb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:27:01.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44b09bb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:27:01.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44b09bb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:27:01.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44b11bc640 1 --2- 172.21.15.63:0/1091280335 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f44ac0a1f30 0x7f44ac13cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:01.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44b19bd640 10 monclient(hunting): handle_auth_done global_id 44214 payload 824 2024-04-26T04:27:01.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44b19bd640 1 -- 172.21.15.63:0/1091280335 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f44ac0a2310 msgr2=0x7f44ac136fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:27:01.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44b19bd640 1 --2- 172.21.15.63:0/1091280335 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f44ac0a2310 0x7f44ac136fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:01.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44b19bd640 1 -- 172.21.15.63:0/1091280335 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f44ac0a1f30 msgr2=0x7f44ac13cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:27:01.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44b19bd640 1 --2- 172.21.15.63:0/1091280335 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f44ac0a1f30 0x7f44ac13cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:01.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44b19bd640 10 monclient: _finish_hunting 0 2024-04-26T04:27:01.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44b19bd640 1 monclient: found mon.noname-a 2024-04-26T04:27:01.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44b19bd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:27:01.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44b19bd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:27:01.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44b19bd640 1 -- 172.21.15.63:0/1091280335 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44ac0a3380 con 0x7f44ac0a2980 2024-04-26T04:27:01.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44b11bc640 10 monclient: get_auth_request con 0x7f44ac0a1f30 auth_method 0 2024-04-26T04:27:01.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44b11bc640 1 --2- 172.21.15.63:0/1091280335 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f44ac0a1f30 0x7f44ac13cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-26T04:27:01.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44b19bd640 1 -- 172.21.15.63:0/1091280335 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f44ac0a2980 msgr2=0x7f44ac1cf020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:27:01.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44b19bd640 1 --2- 172.21.15.63:0/1091280335 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f44ac0a2980 0x7f44ac1cf020 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f44ac0a2720 tx=0x7f44a400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:27:01.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44a1ffb640 1 -- 172.21.15.63:0/1091280335 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a401a070 con 0x7f44ac0a2980 2024-04-26T04:27:01.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44a1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:27:01.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44a1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:27:01.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44a1ffb640 10 monclient: dump: 2024-04-26T04:27:01.554 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:27:01.554 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:27:01.555 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:01.555 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:01.555 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:27:01.555 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:27:01.555 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:27:01.555 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:27:01.555 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:27:01.555 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:01.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44a1ffb640 10 monclient: _finish_auth 0 2024-04-26T04:27:01.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44a1ffb640 10 monclient: _check_auth_tickets 2024-04-26T04:27:01.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:27:01.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44a1ffb640 1 -- 172.21.15.63:0/1091280335 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44a400da50 con 0x7f44ac0a2980 2024-04-26T04:27:01.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44a1ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T04:27:01.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44a1ffb640 1 -- 172.21.15.63:0/1091280335 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a400de40 con 0x7f44ac0a2980 2024-04-26T04:27:01.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44a1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:27:01.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44a1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:27:01.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44a1ffb640 10 monclient: dump: 2024-04-26T04:27:01.556 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:27:01.556 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:27:01.556 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:01.556 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:01.556 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:27:01.556 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:27:01.556 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:27:01.556 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:27:01.556 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:27:01.556 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:01.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44b3420640 5 monclient: authenticate success, global_id 44214 2024-04-26T04:27:01.556 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44b3420640 10 monclient: _renew_subs 2024-04-26T04:27:01.557 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44a37fe640 4 set_mon_vals no callback set 2024-04-26T04:27:01.557 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44b3420640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:27:01.557 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44b3420640 1 -- 172.21.15.63:0/1091280335 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44ac072c20 con 0x7f44ac0a2980 2024-04-26T04:27:01.557 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44b3420640 10 monclient: _renew_subs 2024-04-26T04:27:01.557 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44b3420640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:27:01.557 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44b3420640 1 -- 172.21.15.63:0/1091280335 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44ac122940 con 0x7f44ac0a2980 2024-04-26T04:27:01.557 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44b3420640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:27:01.557 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.541+0000 7f44b3420640 1 -- 172.21.15.63:0/1091280335 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f44ac067fb0 con 0x7f44ac0a2980 2024-04-26T04:27:01.557 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.545+0000 7f44a1ffb640 1 -- 172.21.15.63:0/1091280335 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f44a401d070 con 0x7f44ac0a2980 2024-04-26T04:27:01.557 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.545+0000 7f44a1ffb640 1 --2- 172.21.15.63:0/1091280335 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f4490035ec0 0x7f4490038380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:01.557 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.545+0000 7f44a1ffb640 1 -- 172.21.15.63:0/1091280335 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(323..323 src has 1..323) ==== 4346+0+0 (secure 0 0 0) 0x7f44a4049460 con 0x7f44ac0a2980 2024-04-26T04:27:01.557 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.545+0000 7f44a1ffb640 1 --2- 172.21.15.63:0/1091280335 >> [v2:172.21.15.186:6816/3109931567,v1:172.21.15.186:6817/3109931567] conn(0x7f449003bd30 0x7f449003e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:01.557 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.545+0000 7f44a1ffb640 1 -- 172.21.15.63:0/1091280335 --> [v2:172.21.15.186:6816/3109931567,v1:172.21.15.186:6817/3109931567] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f449003e800 con 0x7f449003bd30 2024-04-26T04:27:01.557 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.545+0000 7f44a1ffb640 1 -- 172.21.15.63:0/1091280335 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_get_version_reply(handle=1 version=323) ==== 24+0+0 (secure 0 0 0) 0x7f44a4050050 con 0x7f44ac0a2980 2024-04-26T04:27:01.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.545+0000 7f44a1ffb640 10 monclient: handle_get_version_reply finishing 1 version 323 2024-04-26T04:27:01.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.545+0000 7f44b11bc640 1 --2- 172.21.15.63:0/1091280335 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f4490035ec0 0x7f4490038380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:01.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.545+0000 7f44b09bb640 1 --2- 172.21.15.63:0/1091280335 >> [v2:172.21.15.186:6816/3109931567,v1:172.21.15.186:6817/3109931567] conn(0x7f449003bd30 0x7f449003e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:01.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.545+0000 7f44b09bb640 10 monclient: get_auth_request con 0x7f449003bd30 auth_method 0 2024-04-26T04:27:01.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.545+0000 7f44b11bc640 10 monclient: get_auth_request con 0x7f4490035ec0 auth_method 0 2024-04-26T04:27:01.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.545+0000 7f44b11bc640 1 --2- 172.21.15.63:0/1091280335 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f4490035ec0 0x7f4490038380 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f44a8009a60 tx=0x7f44a8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:27:01.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.549+0000 7f44b09bb640 1 --2- 172.21.15.63:0/1091280335 >> [v2:172.21.15.186:6816/3109931567,v1:172.21.15.186:6817/3109931567] conn(0x7f449003bd30 0x7f449003e150 crc :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:27:01.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.549+0000 7f44a1ffb640 1 -- 172.21.15.63:0/1091280335 <== osd.5 v2:172.21.15.186:6816/3109931567 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f449003e800 con 0x7f449003bd30 2024-04-26T04:27:01.566 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.565+0000 7f44b3420640 1 -- 172.21.15.63:0/1091280335 --> [v2:172.21.15.186:6816/3109931567,v1:172.21.15.186:6817/3109931567] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f44ac062e10 con 0x7f449003bd30 2024-04-26T04:27:01.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.565+0000 7f44a1ffb640 1 -- 172.21.15.63:0/1091280335 <== osd.5 v2:172.21.15.186:6816/3109931567 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f44ac062e10 con 0x7f449003bd30 2024-04-26T04:27:01.567 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-26T04:27:01.567 INFO:teuthology.orchestra.run.smithi063.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-26T04:27:01.567 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-26T04:27:01.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.565+0000 7f44b3420640 1 -- 172.21.15.63:0/1091280335 >> [v2:172.21.15.186:6816/3109931567,v1:172.21.15.186:6817/3109931567] conn(0x7f449003bd30 msgr2=0x7f449003e150 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:27:01.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.565+0000 7f44b3420640 1 --2- 172.21.15.63:0/1091280335 >> [v2:172.21.15.186:6816/3109931567,v1:172.21.15.186:6817/3109931567] conn(0x7f449003bd30 0x7f449003e150 crc :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:01.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.565+0000 7f44b3420640 1 -- 172.21.15.63:0/1091280335 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f4490035ec0 msgr2=0x7f4490038380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:27:01.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.565+0000 7f44b3420640 1 --2- 172.21.15.63:0/1091280335 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f4490035ec0 0x7f4490038380 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f44a8009a60 tx=0x7f44a8008040 comp rx=0 tx=0).stop 2024-04-26T04:27:01.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.565+0000 7f44b3420640 10 monclient: shutdown 2024-04-26T04:27:01.569 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.565+0000 7f44b3420640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:27:01.569 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.565+0000 7f44b3420640 1 -- 172.21.15.63:0/1091280335 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f44ac0a2980 msgr2=0x7f44ac1cf020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:27:01.569 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.565+0000 7f44b3420640 1 --2- 172.21.15.63:0/1091280335 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f44ac0a2980 0x7f44ac1cf020 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f44ac0a2720 tx=0x7f44a400a710 comp rx=0 tx=0).stop 2024-04-26T04:27:01.569 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.565+0000 7f44b19bd640 1 -- 172.21.15.63:0/1091280335 reap_dead start 2024-04-26T04:27:01.569 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.565+0000 7f44b3420640 1 -- 172.21.15.63:0/1091280335 shutdown_connections 2024-04-26T04:27:01.569 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.565+0000 7f44b3420640 1 -- 172.21.15.63:0/1091280335 >> 172.21.15.63:0/1091280335 conn(0x7f44ac0a5fb0 msgr2=0x7f44ac071210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:27:01.569 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.565+0000 7f44b3420640 1 -- 172.21.15.63:0/1091280335 shutdown_connections 2024-04-26T04:27:01.569 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.565+0000 7f44b3420640 1 -- 172.21.15.63:0/1091280335 wait complete. 2024-04-26T04:27:01.583 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-04-26T04:27:01.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.693+0000 7fc181364640 10 monclient: get_monmap_and_config 2024-04-26T04:27:01.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.693+0000 7fc181364640 10 monclient: build_initial_monmap 2024-04-26T04:27:01.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.693+0000 7fc181364640 10 monclient: monmap: 2024-04-26T04:27:01.696 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:27:01.696 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:27:01.696 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:27:01.695600+0000 2024-04-26T04:27:01.696 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:27:01.695600+0000 2024-04-26T04:27:01.696 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:27:01.696 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:27:01.696 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:27:01.696 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:27:01.696 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:27:01.696 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:27:01.696 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:27:01.696 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:01.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.693+0000 7fc181364640 1 Processor -- start 2024-04-26T04:27:01.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.693+0000 7fc181364640 1 -- start start 2024-04-26T04:27:01.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.693+0000 7fc181364640 10 monclient: init 2024-04-26T04:27:01.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.693+0000 7fc181364640 10 monclient: _reopen_session rank -1 2024-04-26T04:27:01.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.693+0000 7fc181364640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-04-26T04:27:01.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.693+0000 7fc181364640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc17c0a0bb0 addr v1:172.21.15.63:6789/0 2024-04-26T04:27:01.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.693+0000 7fc181364640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc17c136310 addr v1:172.21.15.186:6789/0 2024-04-26T04:27:01.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.693+0000 7fc181364640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fc17c139e20 0x7fc17c13c210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:01.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.693+0000 7fc181364640 10 monclient(hunting): picked mon.noname-b con 0x7fc17c139e20 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:27:01.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.693+0000 7fc181364640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc17c09f270 con 0x7fc17c0a0bb0 2024-04-26T04:27:01.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.693+0000 7fc181364640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc17c0a0f90 con 0x7fc17c136310 2024-04-26T04:27:01.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.693+0000 7fc181364640 10 monclient(hunting): start opening mon connection 2024-04-26T04:27:01.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.693+0000 7fc181364640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7fc17c099240 con 0x7fc17c139e20 2024-04-26T04:27:01.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.693+0000 7fc181364640 10 monclient(hunting): _renew_subs 2024-04-26T04:27:01.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.693+0000 7fc181364640 10 monclient(hunting): authenticate will time out at 1453.403238s 2024-04-26T04:27:01.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.693+0000 7fc17b7fe640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fc17c139e20 0x7fc17c13c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:01.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.693+0000 7fc17affd640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7fc17c0a0bb0 0x7fc17c09eb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:37582/0 (socket says 172.21.15.63:37582) 2024-04-26T04:27:01.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.693+0000 7fc17affd640 1 -- 172.21.15.63:0/3653146758 learned_addr learned my addr 172.21.15.63:0/3653146758 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-04-26T04:27:01.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.693+0000 7fc17b7fe640 10 monclient(hunting): get_auth_request con 0x7fc17c139e20 auth_method 0 2024-04-26T04:27:01.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.693+0000 7fc17b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:27:01.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.693+0000 7fc17b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:27:01.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.693+0000 7fc17b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:27:01.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.693+0000 7fc17b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:27:01.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.693+0000 7fc17b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:27:01.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.693+0000 7fc17b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:27:01.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.693+0000 7fc17b7fe640 10 monclient(hunting): handle_auth_done global_id 24767 payload 274 2024-04-26T04:27:01.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.693+0000 7fc17b7fe640 1 -- 172.21.15.63:0/3653146758 >> v1:172.21.15.186:6789/0 conn(0x7fc17c136310 legacy=0x7fc17c138700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:27:01.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.693+0000 7fc17b7fe640 1 -- 172.21.15.63:0/3653146758 >> v1:172.21.15.63:6789/0 conn(0x7fc17c0a0bb0 legacy=0x7fc17c09eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:27:01.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.693+0000 7fc17b7fe640 10 monclient: _finish_hunting 0 2024-04-26T04:27:01.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.693+0000 7fc17b7fe640 1 monclient: found mon.noname-b 2024-04-26T04:27:01.702 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.693+0000 7fc17b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:27:01.702 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.693+0000 7fc17b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.63:3301/0 2024-04-26T04:27:01.702 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.693+0000 7fc17b7fe640 1 -- 172.21.15.63:0/3653146758 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc17c09b820 con 0x7fc17c139e20 2024-04-26T04:27:01.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.697+0000 7fc17b7fe640 1 -- 172.21.15.63:0/3653146758 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fc17c139e20 msgr2=0x7fc17c13c210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:27:01.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.697+0000 7fc17b7fe640 1 --2- 172.21.15.63:0/3653146758 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fc17c139e20 0x7fc17c13c210 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc174009820 tx=0x7fc17401c0f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=478e0b1ddec09f05 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:27:01.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc179ffb640 1 -- 172.21.15.63:0/3653146758 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1740b0070 con 0x7fc17c139e20 2024-04-26T04:27:01.708 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc179ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:27:01.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc179ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T04:27:01.709 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc179ffb640 10 monclient: dump: 2024-04-26T04:27:01.709 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:27:01.709 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:27:01.709 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:01.709 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:01.709 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:27:01.709 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:27:01.709 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:27:01.709 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:27:01.709 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:27:01.709 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:01.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc179ffb640 10 monclient: _finish_auth 0 2024-04-26T04:27:01.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc179ffb640 10 monclient: _check_auth_tickets 2024-04-26T04:27:01.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc179ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:27:01.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc179ffb640 1 -- 172.21.15.63:0/3653146758 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1740ab600 con 0x7fc17c139e20 2024-04-26T04:27:01.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc179ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T04:27:01.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc179ffb640 1 -- 172.21.15.63:0/3653146758 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1740ab9d0 con 0x7fc17c139e20 2024-04-26T04:27:01.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc181364640 5 monclient: authenticate success, global_id 24767 2024-04-26T04:27:01.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc181364640 10 monclient: get_monmap_and_config success 2024-04-26T04:27:01.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc181364640 4 set_mon_vals no callback set 2024-04-26T04:27:01.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc179ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:27:01.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc179ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:27:01.710 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc179ffb640 10 monclient: dump: 2024-04-26T04:27:01.711 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:27:01.711 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:27:01.711 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:01.715 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:01.716 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:27:01.716 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:27:01.716 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:27:01.716 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:27:01.716 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:27:01.716 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:01.716 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc181364640 10 monclient: shutdown 2024-04-26T04:27:01.716 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc181364640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:27:01.716 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc181364640 1 -- 172.21.15.63:0/3653146758 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fc17c139e20 msgr2=0x7fc17c13c210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:27:01.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc181364640 1 --2- 172.21.15.63:0/3653146758 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fc17c139e20 0x7fc17c13c210 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc174009820 tx=0x7fc17401c0f0 comp rx=0 tx=0).stop 2024-04-26T04:27:01.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc181364640 1 -- 172.21.15.63:0/3653146758 shutdown_connections 2024-04-26T04:27:01.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc181364640 1 --2- 172.21.15.63:0/3653146758 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fc17c139e20 0x7fc17c13c210 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:01.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc181364640 1 -- 172.21.15.63:0/3653146758 >> 172.21.15.63:0/3653146758 conn(0x7fc17c0a1e70 msgr2=0x7fc17c0a2280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:27:01.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc181364640 1 -- 172.21.15.63:0/3653146758 shutdown_connections 2024-04-26T04:27:01.717 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc181364640 1 -- 172.21.15.63:0/3653146758 wait complete. 2024-04-26T04:27:01.718 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc181364640 10 monclient: build_initial_monmap 2024-04-26T04:27:01.718 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc181364640 10 monclient: monmap: 2024-04-26T04:27:01.718 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:27:01.718 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:27:01.718 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:27:01.718 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:27:01.718 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:27:01.718 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:27:01.718 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:27:01.718 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:27:01.718 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:27:01.718 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:01.718 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc181364640 1 Processor -- start 2024-04-26T04:27:01.718 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc181364640 1 -- start start 2024-04-26T04:27:01.718 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc181364640 10 monclient: init 2024-04-26T04:27:01.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc181364640 10 monclient: _reopen_session rank -1 2024-04-26T04:27:01.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc181364640 10 monclient: _add_conns ranks=[2,1,0] 2024-04-26T04:27:01.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc181364640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fc17c0a0bb0 0x7fc17c12a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:01.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc181364640 10 monclient(hunting): picked mon.noname-c con 0x7fc17c0a0bb0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:27:01.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc181364640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc17c136310 0x7fc17c135630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:01.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc181364640 10 monclient(hunting): picked mon.noname-b con 0x7fc17c136310 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:27:01.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc181364640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc17c139e20 0x7fc17c1cc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:01.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc181364640 10 monclient(hunting): picked mon.noname-a con 0x7fc17c139e20 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:27:01.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc181364640 10 monclient(hunting): start opening mon connection 2024-04-26T04:27:01.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc181364640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc17c13f3d0 con 0x7fc17c139e20 2024-04-26T04:27:01.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc181364640 10 monclient(hunting): start opening mon connection 2024-04-26T04:27:01.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc181364640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc17c13f250 con 0x7fc17c136310 2024-04-26T04:27:01.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc181364640 10 monclient(hunting): start opening mon connection 2024-04-26T04:27:01.719 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc181364640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7fc17c13f550 con 0x7fc17c0a0bb0 2024-04-26T04:27:01.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc181364640 10 monclient(hunting): _renew_subs 2024-04-26T04:27:01.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc181364640 10 monclient(hunting): authenticate will time out at 1453.410302s 2024-04-26T04:27:01.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc17affd640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fc17c0a0bb0 0x7fc17c12a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:01.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc17b7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc17c139e20 0x7fc17c1cc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:01.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc17affd640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fc17c0a0bb0 0x7fc17c12a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:39966/0 (socket says 172.21.15.63:39966) 2024-04-26T04:27:01.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc17affd640 1 -- 172.21.15.63:0/3801585602 learned_addr learned my addr 172.21.15.63:0/3801585602 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:27:01.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc17affd640 10 monclient(hunting): get_auth_request con 0x7fc17c0a0bb0 auth_method 0 2024-04-26T04:27:01.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc17affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:27:01.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc17affd640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:27:01.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc17affd640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:27:01.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc17b7fe640 10 monclient(hunting): get_auth_request con 0x7fc17c139e20 auth_method 0 2024-04-26T04:27:01.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc17b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:27:01.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc17b7fe640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:27:01.720 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc17b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:27:01.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc17affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:27:01.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc17affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:27:01.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc17affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:27:01.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc17b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:27:01.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc17b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:27:01.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc17b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:27:01.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc17affd640 10 monclient(hunting): handle_auth_done global_id 24770 payload 824 2024-04-26T04:27:01.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc17a7fc640 1 --2- 172.21.15.63:0/3801585602 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc17c136310 0x7fc17c135630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:01.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc17affd640 1 -- 172.21.15.63:0/3801585602 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc17c136310 msgr2=0x7fc17c135630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:27:01.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc17affd640 1 --2- 172.21.15.63:0/3801585602 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc17c136310 0x7fc17c135630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:01.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc17affd640 1 -- 172.21.15.63:0/3801585602 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc17c139e20 msgr2=0x7fc17c1cc7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:27:01.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc17affd640 1 --2- 172.21.15.63:0/3801585602 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc17c139e20 0x7fc17c1cc7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:01.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc17affd640 10 monclient: _finish_hunting 0 2024-04-26T04:27:01.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc17affd640 1 monclient: found mon.noname-c 2024-04-26T04:27:01.721 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc17affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:27:01.722 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc17affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.63:3301/0 2024-04-26T04:27:01.722 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc17affd640 1 -- 172.21.15.63:0/3801585602 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc17c12c8c0 con 0x7fc17c0a0bb0 2024-04-26T04:27:01.722 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.701+0000 7fc17b7fe640 1 --2- 172.21.15.63:0/3801585602 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc17c139e20 0x7fc17c1cc7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T04:27:01.722 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.705+0000 7fc17affd640 1 -- 172.21.15.63:0/3801585602 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fc17c0a0bb0 msgr2=0x7fc17c12a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:27:01.722 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.705+0000 7fc17affd640 1 --2- 172.21.15.63:0/3801585602 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fc17c0a0bb0 0x7fc17c12a740 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fc16400cc10 tx=0x7fc16400a430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:27:01.722 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.705+0000 7fc1737fe640 1 -- 172.21.15.63:0/3801585602 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc164012400 con 0x7fc17c0a0bb0 2024-04-26T04:27:01.722 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.705+0000 7fc1737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:27:01.722 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.705+0000 7fc1737fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:27:01.722 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.705+0000 7fc1737fe640 10 monclient: dump: 2024-04-26T04:27:01.722 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:27:01.722 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:27:01.722 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:01.722 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:01.722 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:27:01.722 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:27:01.723 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:27:01.723 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:27:01.723 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:27:01.723 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:01.723 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.705+0000 7fc1737fe640 10 monclient: _finish_auth 0 2024-04-26T04:27:01.723 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.705+0000 7fc1737fe640 10 monclient: _check_auth_tickets 2024-04-26T04:27:01.723 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.705+0000 7fc1737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:27:01.723 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.705+0000 7fc1737fe640 1 -- 172.21.15.63:0/3801585602 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1640125e0 con 0x7fc17c0a0bb0 2024-04-26T04:27:01.723 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.705+0000 7fc181364640 5 monclient: authenticate success, global_id 24770 2024-04-26T04:27:01.723 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.705+0000 7fc181364640 10 monclient: _renew_subs 2024-04-26T04:27:01.723 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.705+0000 7fc181364640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:27:01.723 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.705+0000 7fc181364640 1 -- 172.21.15.63:0/3801585602 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc17c12cba0 con 0x7fc17c0a0bb0 2024-04-26T04:27:01.723 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.705+0000 7fc1737fe640 10 monclient: handle_config config(0 keys) 2024-04-26T04:27:01.723 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.705+0000 7fc1737fe640 1 -- 172.21.15.63:0/3801585602 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1640129d0 con 0x7fc17c0a0bb0 2024-04-26T04:27:01.723 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.705+0000 7fc1737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:27:01.724 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.705+0000 7fc1737fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:27:01.724 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.705+0000 7fc1737fe640 10 monclient: dump: 2024-04-26T04:27:01.724 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:27:01.724 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:27:01.724 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:01.724 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:01.724 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:27:01.724 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:27:01.724 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:27:01.724 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:27:01.724 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:27:01.724 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:01.724 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.705+0000 7fc181364640 10 monclient: _renew_subs 2024-04-26T04:27:01.724 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.705+0000 7fc1797fa640 4 set_mon_vals no callback set 2024-04-26T04:27:01.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.705+0000 7fc181364640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:27:01.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.705+0000 7fc181364640 1 -- 172.21.15.63:0/3801585602 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc17c072700 con 0x7fc17c0a0bb0 2024-04-26T04:27:01.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.705+0000 7fc181364640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:27:01.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.705+0000 7fc181364640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:27:01.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.705+0000 7fc181364640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:27:01.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.705+0000 7fc181364640 1 -- 172.21.15.63:0/3801585602 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc17c062e10 con 0x7fc17c0a0bb0 2024-04-26T04:27:01.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.705+0000 7fc1737fe640 1 -- 172.21.15.63:0/3801585602 <== mon.2 v2:172.21.15.63:3301/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fc164012bf0 con 0x7fc17c0a0bb0 2024-04-26T04:27:01.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.705+0000 7fc1737fe640 1 --2- 172.21.15.63:0/3801585602 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fc160035f10 0x7fc1600383d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:01.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.705+0000 7fc1737fe640 1 -- 172.21.15.63:0/3801585602 <== mon.2 v2:172.21.15.63:3301/0 5 ==== osd_map(323..323 src has 1..323) ==== 4346+0+0 (secure 0 0 0) 0x7fc1640493f0 con 0x7fc17c0a0bb0 2024-04-26T04:27:01.725 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.705+0000 7fc17a7fc640 1 --2- 172.21.15.63:0/3801585602 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fc160035f10 0x7fc1600383d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:01.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.705+0000 7fc17a7fc640 10 monclient: get_auth_request con 0x7fc160035f10 auth_method 0 2024-04-26T04:27:01.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.709+0000 7fc17a7fc640 1 --2- 172.21.15.63:0/3801585602 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fc160035f10 0x7fc1600383d0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fc17c071ce0 tx=0x7fc16c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:27:01.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.709+0000 7fc1737fe640 1 -- 172.21.15.63:0/3801585602 <== mon.2 v2:172.21.15.63:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fc16401a360 con 0x7fc17c0a0bb0 2024-04-26T04:27:01.726 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.709+0000 7fc1737fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:27:01.727 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.709+0000 7fc1737fe640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:27:01.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.865+0000 7fc181364640 1 -- 172.21.15.63:0/3801585602 --> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7fc17c12d1d0 con 0x7fc160035f10 2024-04-26T04:27:01.867 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.865+0000 7fc1737fe640 1 -- 172.21.15.63:0/3801585602 <== mgr.4105 v2:172.21.15.186:6824/2583706639 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fc17c12d1d0 con 0x7fc160035f10 2024-04-26T04:27:01.867 INFO:teuthology.orchestra.run.smithi063.stderr:instructed osd(s) 5 to deep-scrub 2024-04-26T04:27:01.869 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.865+0000 7fc181364640 1 -- 172.21.15.63:0/3801585602 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fc160035f10 msgr2=0x7fc1600383d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:27:01.869 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.865+0000 7fc181364640 1 --2- 172.21.15.63:0/3801585602 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fc160035f10 0x7fc1600383d0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fc17c071ce0 tx=0x7fc16c009040 comp rx=0 tx=0).stop 2024-04-26T04:27:01.869 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.865+0000 7fc181364640 10 monclient: shutdown 2024-04-26T04:27:01.869 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.865+0000 7fc181364640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:27:01.869 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.865+0000 7fc181364640 1 -- 172.21.15.63:0/3801585602 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fc17c0a0bb0 msgr2=0x7fc17c12a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:27:01.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.865+0000 7fc181364640 1 --2- 172.21.15.63:0/3801585602 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fc17c0a0bb0 0x7fc17c12a740 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fc16400cc10 tx=0x7fc16400a430 comp rx=0 tx=0).stop 2024-04-26T04:27:01.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.865+0000 7fc181364640 1 -- 172.21.15.63:0/3801585602 shutdown_connections 2024-04-26T04:27:01.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.865+0000 7fc181364640 1 --2- 172.21.15.63:0/3801585602 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fc160035f10 0x7fc1600383d0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:01.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.865+0000 7fc181364640 1 --2- 172.21.15.63:0/3801585602 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc17c139e20 0x7fc17c1cc7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:01.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.865+0000 7fc181364640 1 --2- 172.21.15.63:0/3801585602 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc17c136310 0x7fc17c135630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:01.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.865+0000 7fc181364640 1 --2- 172.21.15.63:0/3801585602 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fc17c0a0bb0 0x7fc17c12a740 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:01.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.865+0000 7fc181364640 1 -- 172.21.15.63:0/3801585602 >> 172.21.15.63:0/3801585602 conn(0x7fc17c0a1e70 msgr2=0x7fc17c0a2fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:27:01.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.865+0000 7fc181364640 1 -- 172.21.15.63:0/3801585602 shutdown_connections 2024-04-26T04:27:01.870 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:01.865+0000 7fc181364640 1 -- 172.21.15.63:0/3801585602 wait complete. 2024-04-26T04:27:01.884 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T04:27:02.001 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.001+0000 7f8fcf4c4640 10 monclient: get_monmap_and_config 2024-04-26T04:27:02.002 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.001+0000 7f8fcf4c4640 10 monclient: build_initial_monmap 2024-04-26T04:27:02.002 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.001+0000 7f8fcf4c4640 10 monclient: monmap: 2024-04-26T04:27:02.002 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:27:02.002 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:27:02.002 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:27:02.001887+0000 2024-04-26T04:27:02.002 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:27:02.001887+0000 2024-04-26T04:27:02.002 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:27:02.002 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:27:02.002 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:27:02.002 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:27:02.002 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:27:02.002 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:27:02.002 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:27:02.003 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:02.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.001+0000 7f8fcf4c4640 1 Processor -- start 2024-04-26T04:27:02.004 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.001+0000 7f8fcf4c4640 1 -- start start 2024-04-26T04:27:02.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.001+0000 7f8fcf4c4640 10 monclient: init 2024-04-26T04:27:02.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.001+0000 7f8fcf4c4640 10 monclient: _reopen_session rank -1 2024-04-26T04:27:02.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.001+0000 7f8fcf4c4640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-04-26T04:27:02.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.001+0000 7f8fcf4c4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8fc809fe60 addr v1:172.21.15.63:6789/0 2024-04-26T04:27:02.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.001+0000 7f8fcf4c4640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f8fc80a0240 0x7f8fc80a2a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:02.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.001+0000 7f8fcf4c4640 10 monclient(hunting): picked mon.noname-a con 0x7f8fc80a0240 addr v2:172.21.15.63:3300/0 2024-04-26T04:27:02.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.001+0000 7f8fcf4c4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8fc80a2f70 addr v1:172.21.15.186:6789/0 2024-04-26T04:27:02.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.001+0000 7f8fcf4c4640 1 -- --> v1:172.21.15.63:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fc80a0c60 con 0x7f8fc809fe60 2024-04-26T04:27:02.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.001+0000 7f8fcf4c4640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fc80a0f40 con 0x7f8fc80a2f70 2024-04-26T04:27:02.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.001+0000 7f8fcf4c4640 10 monclient(hunting): start opening mon connection 2024-04-26T04:27:02.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.001+0000 7f8fcf4c4640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7f8fc80a3730 con 0x7f8fc80a0240 2024-04-26T04:27:02.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.001+0000 7f8fcf4c4640 10 monclient(hunting): _renew_subs 2024-04-26T04:27:02.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.001+0000 7f8fcf4c4640 10 monclient(hunting): authenticate will time out at 1453.709505s 2024-04-26T04:27:02.005 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.001+0000 7f8fcca5f640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f8fc80a0240 0x7f8fc80a2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:02.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.001+0000 7f8fcd260640 1 --1- >> v1:172.21.15.63:6789/0 conn(0x7f8fc809fe60 0x7f8fc81372f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.63:6789/0 says I am v1:172.21.15.63:37590/0 (socket says 172.21.15.63:37590) 2024-04-26T04:27:02.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.001+0000 7f8fcd260640 1 -- 172.21.15.63:0/2702194634 learned_addr learned my addr 172.21.15.63:0/2702194634 (peer_addr_for_me v1:172.21.15.63:0/0) 2024-04-26T04:27:02.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.001+0000 7f8fcca5f640 10 monclient(hunting): get_auth_request con 0x7f8fc80a0240 auth_method 0 2024-04-26T04:27:02.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.001+0000 7f8fcca5f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:27:02.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.001+0000 7f8fcca5f640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:27:02.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.001+0000 7f8fcca5f640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:27:02.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.001+0000 7f8fcca5f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:27:02.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.001+0000 7f8fcca5f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:27:02.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.001+0000 7f8fcca5f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:27:02.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.001+0000 7f8fcca5f640 10 monclient(hunting): handle_auth_done global_id 44220 payload 274 2024-04-26T04:27:02.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.001+0000 7f8fcca5f640 1 -- 172.21.15.63:0/2702194634 >> v1:172.21.15.186:6789/0 conn(0x7f8fc80a2f70 legacy=0x7f8fc813e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:27:02.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.001+0000 7f8fcca5f640 1 -- 172.21.15.63:0/2702194634 >> v1:172.21.15.63:6789/0 conn(0x7f8fc809fe60 legacy=0x7f8fc81372f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:27:02.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.001+0000 7f8fcca5f640 10 monclient: _finish_hunting 0 2024-04-26T04:27:02.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.001+0000 7f8fcca5f640 1 monclient: found mon.noname-a 2024-04-26T04:27:02.006 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.001+0000 7f8fcca5f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:27:02.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.001+0000 7f8fcca5f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:27:02.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.001+0000 7f8fcca5f640 1 -- 172.21.15.63:0/2702194634 --> v2:172.21.15.63:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fc80a1220 con 0x7f8fc80a0240 2024-04-26T04:27:02.007 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.005+0000 7f8fcca5f640 1 -- 172.21.15.63:0/2702194634 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8fc80a0240 msgr2=0x7f8fc80a2a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:27:02.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.005+0000 7f8fcca5f640 1 --2- 172.21.15.63:0/2702194634 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8fc80a0240 0x7f8fc80a2a30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc0097c0 tx=0x7f8fbc01c2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f782b89ef0468afb server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:27:02.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.005+0000 7f8fbbfff640 1 -- 172.21.15.63:0/2702194634 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fbc0b0070 con 0x7f8fc80a0240 2024-04-26T04:27:02.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.005+0000 7f8fbbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:27:02.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.005+0000 7f8fbbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:27:02.008 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.005+0000 7f8fbbfff640 10 monclient: dump: 2024-04-26T04:27:02.008 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:27:02.008 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:27:02.008 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:02.008 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:02.008 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:27:02.009 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:27:02.009 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:27:02.009 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:27:02.009 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:27:02.009 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:02.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.005+0000 7f8fbbfff640 10 monclient: _finish_auth 0 2024-04-26T04:27:02.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.005+0000 7f8fbbfff640 10 monclient: _check_auth_tickets 2024-04-26T04:27:02.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.005+0000 7f8fbbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:27:02.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.005+0000 7f8fbbfff640 1 -- 172.21.15.63:0/2702194634 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fbc0a97f0 con 0x7f8fc80a0240 2024-04-26T04:27:02.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.005+0000 7f8fbbfff640 10 monclient: handle_config config(0 keys) 2024-04-26T04:27:02.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.005+0000 7f8fbbfff640 1 -- 172.21.15.63:0/2702194634 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fbc0a9bc0 con 0x7f8fc80a0240 2024-04-26T04:27:02.009 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.005+0000 7f8fcf4c4640 5 monclient: authenticate success, global_id 44220 2024-04-26T04:27:02.010 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.005+0000 7f8fcf4c4640 10 monclient: get_monmap_and_config success 2024-04-26T04:27:02.010 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.005+0000 7f8fcf4c4640 4 set_mon_vals no callback set 2024-04-26T04:27:02.010 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.005+0000 7f8fbbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:27:02.010 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.005+0000 7f8fbbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:27:02.010 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.005+0000 7f8fbbfff640 10 monclient: dump: 2024-04-26T04:27:02.010 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:27:02.010 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:27:02.010 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:02.010 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:02.010 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:27:02.010 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:27:02.010 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:27:02.010 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:27:02.010 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:27:02.010 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:02.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.005+0000 7f8fcf4c4640 10 monclient: shutdown 2024-04-26T04:27:02.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.005+0000 7f8fcf4c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:27:02.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.005+0000 7f8fcf4c4640 1 -- 172.21.15.63:0/2702194634 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8fc80a0240 msgr2=0x7f8fc80a2a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:27:02.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.005+0000 7f8fcf4c4640 1 --2- 172.21.15.63:0/2702194634 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8fc80a0240 0x7f8fc80a2a30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc0097c0 tx=0x7f8fbc01c2e0 comp rx=0 tx=0).stop 2024-04-26T04:27:02.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.005+0000 7f8fcf4c4640 1 -- 172.21.15.63:0/2702194634 shutdown_connections 2024-04-26T04:27:02.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.005+0000 7f8fcf4c4640 1 --2- 172.21.15.63:0/2702194634 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8fc80a0240 0x7f8fc80a2a30 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:02.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.005+0000 7f8fcf4c4640 1 -- 172.21.15.63:0/2702194634 >> 172.21.15.63:0/2702194634 conn(0x7f8fc80a5fb0 msgr2=0x7f8fc80a63c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:27:02.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.005+0000 7f8fcf4c4640 1 -- 172.21.15.63:0/2702194634 shutdown_connections 2024-04-26T04:27:02.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.005+0000 7f8fcf4c4640 1 -- 172.21.15.63:0/2702194634 wait complete. 2024-04-26T04:27:02.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.005+0000 7f8fcf4c4640 10 monclient: build_initial_monmap 2024-04-26T04:27:02.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.005+0000 7f8fcf4c4640 10 monclient: monmap: 2024-04-26T04:27:02.015 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:27:02.015 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:27:02.015 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:27:02.015 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:27:02.016 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:27:02.016 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:27:02.016 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:27:02.016 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:27:02.016 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:27:02.016 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:02.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fcf4c4640 1 Processor -- start 2024-04-26T04:27:02.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fcf4c4640 1 -- start start 2024-04-26T04:27:02.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fcf4c4640 10 monclient: init 2024-04-26T04:27:02.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fcf4c4640 10 monclient: _reopen_session rank -1 2024-04-26T04:27:02.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fcf4c4640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T04:27:02.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fcf4c4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8fc809fe60 0x7f8fc8137df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:02.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fcf4c4640 10 monclient(hunting): picked mon.noname-b con 0x7f8fc809fe60 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:27:02.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fcf4c4640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8fc80a0240 0x7f8fc8138330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:02.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fcf4c4640 10 monclient(hunting): picked mon.noname-a con 0x7f8fc80a0240 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:27:02.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fcf4c4640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f8fc80a2f70 0x7f8fc8138870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:02.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fcf4c4640 10 monclient(hunting): picked mon.noname-c con 0x7f8fc80a2f70 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:27:02.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fcf4c4640 10 monclient(hunting): start opening mon connection 2024-04-26T04:27:02.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fcf4c4640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f8fc81428e0 con 0x7f8fc80a0240 2024-04-26T04:27:02.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fcf4c4640 10 monclient(hunting): start opening mon connection 2024-04-26T04:27:02.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fcf4c4640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8fc8142760 con 0x7f8fc809fe60 2024-04-26T04:27:02.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fcf4c4640 10 monclient(hunting): start opening mon connection 2024-04-26T04:27:02.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fcf4c4640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f8fc8142a60 con 0x7f8fc80a2f70 2024-04-26T04:27:02.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fcf4c4640 10 monclient(hunting): _renew_subs 2024-04-26T04:27:02.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fcf4c4640 10 monclient(hunting): authenticate will time out at 1453.716638s 2024-04-26T04:27:02.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fcda61640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f8fc80a2f70 0x7f8fc8138870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:02.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fcca5f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8fc80a0240 0x7f8fc8138330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:02.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fcca5f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8fc80a0240 0x7f8fc8138330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:47666/0 (socket says 172.21.15.63:47666) 2024-04-26T04:27:02.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fcca5f640 1 -- 172.21.15.63:0/3320115475 learned_addr learned my addr 172.21.15.63:0/3320115475 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:27:02.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fcda61640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f8fc80a2f70 0x7f8fc8138870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:39974/0 (socket says 172.21.15.63:39974) 2024-04-26T04:27:02.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fcca5f640 10 monclient(hunting): get_auth_request con 0x7f8fc80a0240 auth_method 0 2024-04-26T04:27:02.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fcca5f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:27:02.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fcca5f640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:27:02.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fcca5f640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:27:02.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fcda61640 10 monclient(hunting): get_auth_request con 0x7f8fc80a2f70 auth_method 0 2024-04-26T04:27:02.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fcda61640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:27:02.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fcda61640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:27:02.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fcda61640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:27:02.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fcca5f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:27:02.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fcca5f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:27:02.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fcca5f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:27:02.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fcda61640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:27:02.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fcda61640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:27:02.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fcda61640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:27:02.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fcd260640 1 --2- 172.21.15.63:0/3320115475 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8fc809fe60 0x7f8fc8137df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:02.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fcca5f640 10 monclient(hunting): handle_auth_done global_id 44223 payload 824 2024-04-26T04:27:02.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fcca5f640 1 -- 172.21.15.63:0/3320115475 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f8fc80a2f70 msgr2=0x7f8fc8138870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:27:02.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fcca5f640 1 --2- 172.21.15.63:0/3320115475 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f8fc80a2f70 0x7f8fc8138870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:02.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fcca5f640 1 -- 172.21.15.63:0/3320115475 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8fc809fe60 msgr2=0x7f8fc8137df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:27:02.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fcca5f640 1 --2- 172.21.15.63:0/3320115475 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8fc809fe60 0x7f8fc8137df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:02.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fcca5f640 10 monclient: _finish_hunting 0 2024-04-26T04:27:02.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fcca5f640 1 monclient: found mon.noname-a 2024-04-26T04:27:02.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fcca5f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:27:02.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fcca5f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.63:3300/0 2024-04-26T04:27:02.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fcca5f640 1 -- 172.21.15.63:0/3320115475 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fc813f140 con 0x7f8fc80a0240 2024-04-26T04:27:02.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fcca5f640 1 -- 172.21.15.63:0/3320115475 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8fc80a0240 msgr2=0x7f8fc8138330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:27:02.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fcca5f640 1 --2- 172.21.15.63:0/3320115475 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8fc80a0240 0x7f8fc8138330 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc0ae040 tx=0x7f8fbc01ab20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:27:02.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fb9ffb640 1 -- 172.21.15.63:0/3320115475 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fbc0b0050 con 0x7f8fc80a0240 2024-04-26T04:27:02.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fb9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:27:02.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fb9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-04-26T04:27:02.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fb9ffb640 10 monclient: dump: 2024-04-26T04:27:02.020 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:27:02.020 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:27:02.020 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:02.020 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:02.020 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:27:02.020 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:27:02.020 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:27:02.020 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:27:02.020 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:27:02.020 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:02.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fb9ffb640 10 monclient: _finish_auth 0 2024-04-26T04:27:02.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fb9ffb640 10 monclient: _check_auth_tickets 2024-04-26T04:27:02.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fb9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:27:02.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fb9ffb640 1 -- 172.21.15.63:0/3320115475 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fbc0bc070 con 0x7f8fc80a0240 2024-04-26T04:27:02.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fb9ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T04:27:02.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fb9ffb640 1 -- 172.21.15.63:0/3320115475 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fbc0b7450 con 0x7f8fc80a0240 2024-04-26T04:27:02.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fcf4c4640 5 monclient: authenticate success, global_id 44223 2024-04-26T04:27:02.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fcf4c4640 10 monclient: _renew_subs 2024-04-26T04:27:02.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fcf4c4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:27:02.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fcf4c4640 1 -- 172.21.15.63:0/3320115475 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fc80a38f0 con 0x7f8fc80a0240 2024-04-26T04:27:02.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fb9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:27:02.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fb9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-04-26T04:27:02.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fb9ffb640 10 monclient: dump: 2024-04-26T04:27:02.021 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:27:02.021 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:27:02.021 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:02.021 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:02.021 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:27:02.022 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:27:02.022 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:27:02.022 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:27:02.022 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:27:02.022 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:02.022 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fbb7fe640 4 set_mon_vals no callback set 2024-04-26T04:27:02.022 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fcf4c4640 10 monclient: _renew_subs 2024-04-26T04:27:02.022 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fcf4c4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:27:02.022 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.009+0000 7f8fcf4c4640 1 -- 172.21.15.63:0/3320115475 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fc813a880 con 0x7f8fc80a0240 2024-04-26T04:27:02.022 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.013+0000 7f8fb9ffb640 1 -- 172.21.15.63:0/3320115475 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f8fbc0a96b0 con 0x7f8fc80a0240 2024-04-26T04:27:02.022 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.013+0000 7f8fcf4c4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:27:02.022 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.013+0000 7f8fcf4c4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:27:02.022 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.013+0000 7f8fcf4c4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.63:3300/0 2024-04-26T04:27:02.022 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.013+0000 7f8fcf4c4640 1 -- 172.21.15.63:0/3320115475 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fc8062e10 con 0x7f8fc80a0240 2024-04-26T04:27:02.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.013+0000 7f8fb9ffb640 1 --2- 172.21.15.63:0/3320115475 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f8fac035ec0 0x7f8fac038380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:02.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.013+0000 7f8fb9ffb640 1 -- 172.21.15.63:0/3320115475 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(323..323 src has 1..323) ==== 4346+0+0 (secure 0 0 0) 0x7f8fbc0e39b0 con 0x7f8fc80a0240 2024-04-26T04:27:02.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.013+0000 7f8fcd260640 1 --2- 172.21.15.63:0/3320115475 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f8fac035ec0 0x7f8fac038380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:02.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.013+0000 7f8fcd260640 10 monclient: get_auth_request con 0x7f8fac035ec0 auth_method 0 2024-04-26T04:27:02.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.013+0000 7f8fcd260640 1 --2- 172.21.15.63:0/3320115475 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f8fac035ec0 0x7f8fac038380 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f8fc4005060 tx=0x7f8fc4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:27:02.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.017+0000 7f8fb9ffb640 1 -- 172.21.15.63:0/3320115475 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f8fbc015ff0 con 0x7f8fc80a0240 2024-04-26T04:27:02.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.017+0000 7f8fb9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:27:02.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.017+0000 7f8fb9ffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:27:02.172 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.169+0000 7f8fcf4c4640 1 -- 172.21.15.63:0/3320115475 --> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8fc8138e70 con 0x7f8fac035ec0 2024-04-26T04:27:02.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.173+0000 7f8fb9ffb640 1 -- 172.21.15.63:0/3320115475 <== mgr.4105 v2:172.21.15.186:6824/2583706639 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33704 (secure 0 0 0) 0x7f8fc8138e70 con 0x7f8fac035ec0 2024-04-26T04:27:02.175 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-26T04:27:02.176 INFO:teuthology.orchestra.run.smithi063.stderr:dumped all 2024-04-26T04:27:02.177 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.173+0000 7f8fcf4c4640 1 -- 172.21.15.63:0/3320115475 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f8fac035ec0 msgr2=0x7f8fac038380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:27:02.177 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.173+0000 7f8fcf4c4640 1 --2- 172.21.15.63:0/3320115475 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f8fac035ec0 0x7f8fac038380 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f8fc4005060 tx=0x7f8fc4009040 comp rx=0 tx=0).stop 2024-04-26T04:27:02.177 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.173+0000 7f8fcf4c4640 10 monclient: shutdown 2024-04-26T04:27:02.177 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.173+0000 7f8fcf4c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:27:02.177 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.173+0000 7f8fcf4c4640 1 -- 172.21.15.63:0/3320115475 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8fc80a0240 msgr2=0x7f8fc8138330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:27:02.177 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.173+0000 7f8fcf4c4640 1 --2- 172.21.15.63:0/3320115475 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8fc80a0240 0x7f8fc8138330 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc0ae040 tx=0x7f8fbc01ab20 comp rx=0 tx=0).stop 2024-04-26T04:27:02.178 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.173+0000 7f8fcf4c4640 1 -- 172.21.15.63:0/3320115475 shutdown_connections 2024-04-26T04:27:02.178 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.173+0000 7f8fcf4c4640 1 --2- 172.21.15.63:0/3320115475 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f8fac035ec0 0x7f8fac038380 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:02.178 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.173+0000 7f8fcf4c4640 1 --2- 172.21.15.63:0/3320115475 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f8fc80a2f70 0x7f8fc8138870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:02.178 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.173+0000 7f8fcf4c4640 1 --2- 172.21.15.63:0/3320115475 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8fc80a0240 0x7f8fc8138330 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:02.178 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.173+0000 7f8fcf4c4640 1 --2- 172.21.15.63:0/3320115475 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8fc809fe60 0x7f8fc8137df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:02.178 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.173+0000 7f8fcf4c4640 1 -- 172.21.15.63:0/3320115475 >> 172.21.15.63:0/3320115475 conn(0x7f8fc80a5fb0 msgr2=0x7f8fc80a1a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:27:02.178 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.173+0000 7f8fcf4c4640 1 -- 172.21.15.63:0/3320115475 shutdown_connections 2024-04-26T04:27:02.178 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:02.173+0000 7f8fcf4c4640 1 -- 172.21.15.63:0/3320115475 wait complete. 2024-04-26T04:27:02.192 INFO:teuthology.orchestra.run.smithi063.stdout:{"pg_ready":true,"pg_map":{"version":567,"stamp":"2024-04-26T04:27:01.593625+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":205168,"kb_used_data":26408,"kb_used_omap":9,"kb_used_meta":178678,"kb_avail":566025872,"statfs":{"total":579820584960,"available":579610492928,"internally_reserved":0,"allocated":27041792,"data_stored":10728854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9567,"internal_metadata":182966945},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003192"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":617,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-04-26T04:26:38.082572+0000","last_change":"2024-04-26T04:18:47.131857+0000","last_active":"2024-04-26T04:26:38.082572+0000","last_peered":"2024-04-26T04:26:38.082572+0000","last_clean":"2024-04-26T04:26:38.082572+0000","last_became_active":"2024-04-26T04:18:47.131175+0000","last_became_peered":"2024-04-26T04:18:47.131175+0000","last_unstale":"2024-04-26T04:26:38.082572+0000","last_undegraded":"2024-04-26T04:26:38.082572+0000","last_fullsized":"2024-04-26T04:26:38.082572+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_clean_scrub_stamp":"2024-04-26T04:18:45.482773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:35:03.860574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":619,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-04-26T04:27:00.038214+0000","last_change":"2024-04-26T04:27:00.038234+0000","last_active":"2024-04-26T04:27:00.038214+0000","last_peered":"2024-04-26T04:27:00.038214+0000","last_clean":"2024-04-26T04:27:00.038214+0000","last_became_active":"2024-04-26T04:18:46.498983+0000","last_became_peered":"2024-04-26T04:18:46.498983+0000","last_unstale":"2024-04-26T04:27:00.038214+0000","last_undegraded":"2024-04-26T04:27:00.038214+0000","last_fullsized":"2024-04-26T04:27:00.038214+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:27:00.038137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:27:00.038137+0000","last_clean_scrub_stamp":"2024-04-26T04:27:00.038137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:16:48.146301+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":617,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-04-26T04:26:38.082586+0000","last_change":"2024-04-26T04:18:46.507904+0000","last_active":"2024-04-26T04:26:38.082586+0000","last_peered":"2024-04-26T04:26:38.082586+0000","last_clean":"2024-04-26T04:26:38.082586+0000","last_became_active":"2024-04-26T04:18:46.507335+0000","last_became_peered":"2024-04-26T04:18:46.507335+0000","last_unstale":"2024-04-26T04:26:38.082586+0000","last_undegraded":"2024-04-26T04:26:38.082586+0000","last_fullsized":"2024-04-26T04:26:38.082586+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_clean_scrub_stamp":"2024-04-26T04:18:45.482773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:52:50.029369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":612,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-04-26T04:26:37.276418+0000","last_change":"2024-04-26T04:18:46.506682+0000","last_active":"2024-04-26T04:26:37.276418+0000","last_peered":"2024-04-26T04:26:37.276418+0000","last_clean":"2024-04-26T04:26:37.276418+0000","last_became_active":"2024-04-26T04:18:46.506209+0000","last_became_peered":"2024-04-26T04:18:46.506209+0000","last_unstale":"2024-04-26T04:26:37.276418+0000","last_undegraded":"2024-04-26T04:26:37.276418+0000","last_fullsized":"2024-04-26T04:26:37.276418+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_clean_scrub_stamp":"2024-04-26T04:18:45.482773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":613,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-04-26T04:26:37.275301+0000","last_change":"2024-04-26T04:18:46.504685+0000","last_active":"2024-04-26T04:26:37.275301+0000","last_peered":"2024-04-26T04:26:37.275301+0000","last_clean":"2024-04-26T04:26:37.275301+0000","last_became_active":"2024-04-26T04:18:46.504292+0000","last_became_peered":"2024-04-26T04:18:46.504292+0000","last_unstale":"2024-04-26T04:26:37.275301+0000","last_undegraded":"2024-04-26T04:26:37.275301+0000","last_fullsized":"2024-04-26T04:26:37.275301+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_clean_scrub_stamp":"2024-04-26T04:18:45.482773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:42:29.110482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":606,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-04-26T04:26:38.261943+0000","last_change":"2024-04-26T04:18:46.504010+0000","last_active":"2024-04-26T04:26:38.261943+0000","last_peered":"2024-04-26T04:26:38.261943+0000","last_clean":"2024-04-26T04:26:38.261943+0000","last_became_active":"2024-04-26T04:18:46.503343+0000","last_became_peered":"2024-04-26T04:18:46.503343+0000","last_unstale":"2024-04-26T04:26:38.261943+0000","last_undegraded":"2024-04-26T04:26:38.261943+0000","last_fullsized":"2024-04-26T04:26:38.261943+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_clean_scrub_stamp":"2024-04-26T04:18:45.482773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":618,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-04-26T04:26:38.082607+0000","last_change":"2024-04-26T04:18:46.503233+0000","last_active":"2024-04-26T04:26:38.082607+0000","last_peered":"2024-04-26T04:26:38.082607+0000","last_clean":"2024-04-26T04:26:38.082607+0000","last_became_active":"2024-04-26T04:18:46.502682+0000","last_became_peered":"2024-04-26T04:18:46.502682+0000","last_unstale":"2024-04-26T04:26:38.082607+0000","last_undegraded":"2024-04-26T04:26:38.082607+0000","last_fullsized":"2024-04-26T04:26:38.082607+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_clean_scrub_stamp":"2024-04-26T04:18:45.482773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:26:15.760818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":613,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-04-26T04:26:37.275301+0000","last_change":"2024-04-26T04:18:46.504844+0000","last_active":"2024-04-26T04:26:37.275301+0000","last_peered":"2024-04-26T04:26:37.275301+0000","last_clean":"2024-04-26T04:26:37.275301+0000","last_became_active":"2024-04-26T04:18:46.504332+0000","last_became_peered":"2024-04-26T04:18:46.504332+0000","last_unstale":"2024-04-26T04:26:37.275301+0000","last_undegraded":"2024-04-26T04:26:37.275301+0000","last_fullsized":"2024-04-26T04:26:37.275301+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:45.482773+0000","last_clean_scrub_stamp":"2024-04-26T04:18:45.482773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:13:26.538440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":770,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-04-26T04:26:38.082546+0000","last_change":"2024-04-26T04:18:45.505818+0000","last_active":"2024-04-26T04:26:38.082546+0000","last_peered":"2024-04-26T04:26:38.082546+0000","last_clean":"2024-04-26T04:26:38.082546+0000","last_became_active":"2024-04-26T04:18:45.504887+0000","last_became_peered":"2024-04-26T04:18:45.504887+0000","last_unstale":"2024-04-26T04:26:38.082546+0000","last_undegraded":"2024-04-26T04:26:38.082546+0000","last_fullsized":"2024-04-26T04:26:38.082546+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:18:44.470061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:18:44.470061+0000","last_clean_scrub_stamp":"2024-04-26T04:18:44.470061+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T11:42:56.629937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":12,"seq":51539607657,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35928,"kb_used_data":4748,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335912,"statfs":{"total":96636764160,"available":96599973888,"internally_reserved":0,"allocated":4861952,"data_stored":2140761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":12,"seq":51539607655,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35408,"kb_used_data":4228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94336432,"statfs":{"total":96636764160,"available":96600506368,"internally_reserved":0,"allocated":4329472,"data_stored":1611833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607655,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35408,"kb_used_data":4228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94336432,"statfs":{"total":96636764160,"available":96600506368,"internally_reserved":0,"allocated":4329472,"data_stored":1611833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640359,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35408,"kb_used_data":4228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94336432,"statfs":{"total":96636764160,"available":96600506368,"internally_reserved":0,"allocated":4329472,"data_stored":1611833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640358,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31248,"kb_used_data":4228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340592,"statfs":{"total":96636764160,"available":96604766208,"internally_reserved":0,"allocated":4329472,"data_stored":1611833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640358,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31768,"kb_used_data":4748,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340072,"statfs":{"total":96636764160,"available":96604233728,"internally_reserved":0,"allocated":4861952,"data_stored":2140761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T04:27:02.196 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-04-26T04:18:45.482773+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=4, tm_min=18, tm_sec=45, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=4, tm_min=26, tm_sec=57, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T04:27:02.196 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-04-26T04:18:45.482773+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=4, tm_min=18, tm_sec=45, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=4, tm_min=26, tm_sec=57, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T04:27:02.196 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-04-26T04:18:45.482773+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=4, tm_min=18, tm_sec=45, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=4, tm_min=26, tm_sec=57, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T04:27:02.197 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-04-26T04:18:45.482773+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=4, tm_min=18, tm_sec=45, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=4, tm_min=26, tm_sec=57, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T04:27:02.197 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-04-26T04:18:45.482773+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=4, tm_min=18, tm_sec=45, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=4, tm_min=26, tm_sec=57, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T04:27:02.197 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-04-26T04:18:45.482773+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=4, tm_min=18, tm_sec=45, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=4, tm_min=26, tm_sec=57, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T04:27:02.197 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-04-26T04:18:45.482773+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=4, tm_min=18, tm_sec=45, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=4, tm_min=26, tm_sec=57, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T04:27:02.197 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-04-26T04:18:44.470061+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=4, tm_min=18, tm_sec=44, tm_wday=4, tm_yday=117, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=26, tm_hour=4, tm_min=26, tm_sec=57, tm_wday=4, tm_yday=117, tm_isdst=0) 2024-04-26T04:27:02.197 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-26T04:27:22.198 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-26T04:27:22.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.329+0000 7f30a38d8640 10 monclient: get_monmap_and_config 2024-04-26T04:27:22.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.329+0000 7f30a38d8640 10 monclient: build_initial_monmap 2024-04-26T04:27:22.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.329+0000 7f30a38d8640 10 monclient: monmap: 2024-04-26T04:27:22.332 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:27:22.333 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:27:22.333 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:27:22.332509+0000 2024-04-26T04:27:22.333 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:27:22.332509+0000 2024-04-26T04:27:22.333 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:27:22.333 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:27:22.333 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:27:22.333 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:27:22.333 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:27:22.333 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:27:22.333 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:27:22.333 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:22.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.329+0000 7f30a38d8640 1 Processor -- start 2024-04-26T04:27:22.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.329+0000 7f30a38d8640 1 -- start start 2024-04-26T04:27:22.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.329+0000 7f30a38d8640 10 monclient: init 2024-04-26T04:27:22.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.333+0000 7f30a38d8640 10 monclient: _reopen_session rank -1 2024-04-26T04:27:22.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.333+0000 7f30a38d8640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2024-04-26T04:27:22.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.333+0000 7f30a38d8640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f309c0a0bb0 0x7f309c09eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:22.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.333+0000 7f30a38d8640 10 monclient(hunting): picked mon.noname-b con 0x7f309c0a0bb0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:27:22.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.333+0000 7f30a38d8640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f309c09d7b0 0x7f309c09db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:22.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.333+0000 7f30a38d8640 10 monclient(hunting): picked mon.noname-c con 0x7f309c09d7b0 addr v2:172.21.15.186:3300/0 2024-04-26T04:27:22.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.333+0000 7f30a38d8640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f309c09e0d0 0x7f309c13a330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:22.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.333+0000 7f30a38d8640 10 monclient(hunting): picked mon.noname-a con 0x7f309c09e0d0 addr v2:172.21.15.63:3300/0 2024-04-26T04:27:22.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.333+0000 7f30a38d8640 10 monclient(hunting): start opening mon connection 2024-04-26T04:27:22.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.333+0000 7f30a38d8640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7f309c0a0f90 con 0x7f309c09e0d0 2024-04-26T04:27:22.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.333+0000 7f30a38d8640 10 monclient(hunting): start opening mon connection 2024-04-26T04:27:22.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.333+0000 7f30a38d8640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7f309c099240 con 0x7f309c09d7b0 2024-04-26T04:27:22.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.333+0000 7f30a38d8640 10 monclient(hunting): start opening mon connection 2024-04-26T04:27:22.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.333+0000 7f30a38d8640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f309c09b820 con 0x7f309c0a0bb0 2024-04-26T04:27:22.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.333+0000 7f30a38d8640 10 monclient(hunting): _renew_subs 2024-04-26T04:27:22.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.333+0000 7f30a38d8640 10 monclient(hunting): authenticate will time out at 1474.040176s 2024-04-26T04:27:22.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.333+0000 7f30a1674640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f309c0a0bb0 0x7f309c09eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:22.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.333+0000 7f30a1e75640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f309c09e0d0 0x7f309c13a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:22.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.333+0000 7f30a1e75640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7f309c09e0d0 0x7f309c13a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:33726/0 (socket says 172.21.15.63:33726) 2024-04-26T04:27:22.339 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.333+0000 7f30a1674640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f309c0a0bb0 0x7f309c09eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:43882/0 (socket says 172.21.15.63:43882) 2024-04-26T04:27:22.339 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.333+0000 7f30a1e75640 1 -- 172.21.15.63:0/3647002456 learned_addr learned my addr 172.21.15.63:0/3647002456 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:27:22.339 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.333+0000 7f30a1e75640 10 monclient(hunting): get_auth_request con 0x7f309c09e0d0 auth_method 0 2024-04-26T04:27:22.339 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.333+0000 7f30a1e75640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:27:22.339 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.333+0000 7f30a1e75640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:27:22.339 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.333+0000 7f30a1e75640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:27:22.339 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.333+0000 7f30a1674640 10 monclient(hunting): get_auth_request con 0x7f309c0a0bb0 auth_method 0 2024-04-26T04:27:22.339 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.333+0000 7f30a1674640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:27:22.339 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.333+0000 7f30a1674640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:27:22.339 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.333+0000 7f30a1674640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:27:22.339 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.333+0000 7f30a1e75640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:27:22.339 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.333+0000 7f30a1e75640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:27:22.339 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.333+0000 7f30a1e75640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:27:22.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.333+0000 7f30a1674640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:27:22.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.333+0000 7f30a1674640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:27:22.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.333+0000 7f30a1674640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:27:22.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.333+0000 7f30a0e73640 1 --2- 172.21.15.63:0/3647002456 >> v2:172.21.15.186:3300/0 conn(0x7f309c09d7b0 0x7f309c09db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:22.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.333+0000 7f30a0e73640 10 monclient(hunting): get_auth_request con 0x7f309c09d7b0 auth_method 0 2024-04-26T04:27:22.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.333+0000 7f30a0e73640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:27:22.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.333+0000 7f30a0e73640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:27:22.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.333+0000 7f30a0e73640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:27:22.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.333+0000 7f30a1674640 10 monclient(hunting): handle_auth_done global_id 24776 payload 274 2024-04-26T04:27:22.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.333+0000 7f30a1674640 1 -- 172.21.15.63:0/3647002456 >> v2:172.21.15.186:3300/0 conn(0x7f309c09d7b0 msgr2=0x7f309c09db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:27:22.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.333+0000 7f30a1674640 1 --2- 172.21.15.63:0/3647002456 >> v2:172.21.15.186:3300/0 conn(0x7f309c09d7b0 0x7f309c09db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:22.340 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.333+0000 7f30a1674640 1 -- 172.21.15.63:0/3647002456 >> v2:172.21.15.63:3300/0 conn(0x7f309c09e0d0 msgr2=0x7f309c13a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:27:22.341 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.333+0000 7f30a1674640 1 --2- 172.21.15.63:0/3647002456 >> v2:172.21.15.63:3300/0 conn(0x7f309c09e0d0 0x7f309c13a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:22.341 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.333+0000 7f30a1674640 10 monclient: _finish_hunting 0 2024-04-26T04:27:22.341 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.333+0000 7f30a1674640 1 monclient: found mon.noname-b 2024-04-26T04:27:22.341 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.333+0000 7f30a1674640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:27:22.341 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.333+0000 7f30a1674640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.63:3301/0 2024-04-26T04:27:22.341 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.333+0000 7f30a1674640 1 -- 172.21.15.63:0/3647002456 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f309c0994e0 con 0x7f309c0a0bb0 2024-04-26T04:27:22.341 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.333+0000 7f30a1e75640 1 --2- 172.21.15.63:0/3647002456 >> v2:172.21.15.63:3300/0 conn(0x7f309c09e0d0 0x7f309c13a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T04:27:22.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.337+0000 7f30a1674640 1 -- 172.21.15.63:0/3647002456 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f309c0a0bb0 msgr2=0x7f309c09eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:27:22.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.337+0000 7f30a1674640 1 --2- 172.21.15.63:0/3647002456 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f309c0a0bb0 0x7f309c09eb60 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f309c06bde0 tx=0x7f308c01e9e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=abb5466fe57469b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:27:22.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.337+0000 7f3093fff640 1 -- 172.21.15.63:0/3647002456 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f308c0b4070 con 0x7f309c0a0bb0 2024-04-26T04:27:22.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.337+0000 7f3093fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:27:22.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.337+0000 7f3093fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T04:27:22.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.337+0000 7f3093fff640 10 monclient: dump: 2024-04-26T04:27:22.347 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:27:22.347 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:27:22.347 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:22.347 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:22.347 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:27:22.348 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:27:22.348 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:27:22.348 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:27:22.348 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:27:22.348 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:22.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.337+0000 7f3093fff640 10 monclient: _finish_auth 0 2024-04-26T04:27:22.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.337+0000 7f3093fff640 10 monclient: _check_auth_tickets 2024-04-26T04:27:22.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.337+0000 7f3093fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:27:22.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.337+0000 7f3093fff640 1 -- 172.21.15.63:0/3647002456 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f308c0a7880 con 0x7f309c0a0bb0 2024-04-26T04:27:22.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.337+0000 7f3093fff640 10 monclient: handle_config config(0 keys) 2024-04-26T04:27:22.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.337+0000 7f3093fff640 1 -- 172.21.15.63:0/3647002456 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f308c0a7c50 con 0x7f309c0a0bb0 2024-04-26T04:27:22.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.337+0000 7f3093fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:27:22.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.337+0000 7f3093fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:27:22.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.337+0000 7f3093fff640 10 monclient: dump: 2024-04-26T04:27:22.349 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:27:22.349 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:27:22.349 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:22.349 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:22.349 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:27:22.349 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:27:22.349 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:27:22.349 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:27:22.349 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:27:22.349 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:22.349 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.337+0000 7f30a38d8640 5 monclient: authenticate success, global_id 24776 2024-04-26T04:27:22.349 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.337+0000 7f30a38d8640 10 monclient: get_monmap_and_config success 2024-04-26T04:27:22.349 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.337+0000 7f30a38d8640 4 set_mon_vals no callback set 2024-04-26T04:27:22.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.337+0000 7f30a38d8640 10 monclient: shutdown 2024-04-26T04:27:22.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.337+0000 7f30a38d8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:27:22.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.337+0000 7f30a38d8640 1 -- 172.21.15.63:0/3647002456 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f309c0a0bb0 msgr2=0x7f309c09eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:27:22.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.337+0000 7f30a38d8640 1 --2- 172.21.15.63:0/3647002456 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f309c0a0bb0 0x7f309c09eb60 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f309c06bde0 tx=0x7f308c01e9e0 comp rx=0 tx=0).stop 2024-04-26T04:27:22.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.337+0000 7f30a38d8640 1 -- 172.21.15.63:0/3647002456 shutdown_connections 2024-04-26T04:27:22.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.337+0000 7f30a38d8640 1 --2- 172.21.15.63:0/3647002456 >> v2:172.21.15.63:3300/0 conn(0x7f309c09e0d0 0x7f309c13a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:22.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.337+0000 7f30a38d8640 1 --2- 172.21.15.63:0/3647002456 >> v2:172.21.15.186:3300/0 conn(0x7f309c09d7b0 0x7f309c09db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:22.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.337+0000 7f30a38d8640 1 --2- 172.21.15.63:0/3647002456 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f309c0a0bb0 0x7f309c09eb60 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:22.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.337+0000 7f30a38d8640 1 -- 172.21.15.63:0/3647002456 >> 172.21.15.63:0/3647002456 conn(0x7f309c0a1e70 msgr2=0x7f309c0a2280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:27:22.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.337+0000 7f30a38d8640 1 -- 172.21.15.63:0/3647002456 shutdown_connections 2024-04-26T04:27:22.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.337+0000 7f30a38d8640 1 -- 172.21.15.63:0/3647002456 wait complete. 2024-04-26T04:27:22.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.337+0000 7f30a38d8640 10 monclient: build_initial_monmap 2024-04-26T04:27:22.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.337+0000 7f30a38d8640 10 monclient: monmap: 2024-04-26T04:27:22.351 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:27:22.351 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:27:22.351 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:27:22.351 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:27:22.351 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:27:22.351 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:27:22.351 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:27:22.351 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:27:22.351 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:27:22.351 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:22.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.337+0000 7f30a38d8640 1 Processor -- start 2024-04-26T04:27:22.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.337+0000 7f30a38d8640 1 -- start start 2024-04-26T04:27:22.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.337+0000 7f30a38d8640 10 monclient: init 2024-04-26T04:27:22.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.337+0000 7f30a38d8640 10 monclient: _reopen_session rank -1 2024-04-26T04:27:22.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.337+0000 7f30a38d8640 10 monclient: _add_conns ranks=[1,0,2] 2024-04-26T04:27:22.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.337+0000 7f30a38d8640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f309c09d7b0 0x7f309c1ce9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:22.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.337+0000 7f30a38d8640 10 monclient(hunting): picked mon.noname-b con 0x7f309c09d7b0 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:27:22.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.337+0000 7f30a38d8640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f309c09e0d0 0x7f309c097c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:22.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.337+0000 7f30a38d8640 10 monclient(hunting): picked mon.noname-a con 0x7f309c09e0d0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:27:22.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.337+0000 7f30a38d8640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f309c0a0bb0 0x7f309c098160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:22.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.337+0000 7f30a38d8640 10 monclient(hunting): picked mon.noname-c con 0x7f309c0a0bb0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:27:22.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.337+0000 7f30a38d8640 10 monclient(hunting): start opening mon connection 2024-04-26T04:27:22.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.337+0000 7f30a38d8640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f309c09d030 con 0x7f309c09e0d0 2024-04-26T04:27:22.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.337+0000 7f30a38d8640 10 monclient(hunting): start opening mon connection 2024-04-26T04:27:22.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.337+0000 7f30a38d8640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f309c09ceb0 con 0x7f309c09d7b0 2024-04-26T04:27:22.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.337+0000 7f30a38d8640 10 monclient(hunting): start opening mon connection 2024-04-26T04:27:22.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.337+0000 7f30a38d8640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7f309c09d1b0 con 0x7f309c0a0bb0 2024-04-26T04:27:22.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.337+0000 7f30a38d8640 10 monclient(hunting): _renew_subs 2024-04-26T04:27:22.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.337+0000 7f30a38d8640 10 monclient(hunting): authenticate will time out at 1474.047664s 2024-04-26T04:27:22.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.337+0000 7f30a1e75640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f309c0a0bb0 0x7f309c098160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:22.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.337+0000 7f30a0e73640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f309c09e0d0 0x7f309c097c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:22.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.337+0000 7f30a1e75640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f309c0a0bb0 0x7f309c098160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:43890/0 (socket says 172.21.15.63:43890) 2024-04-26T04:27:22.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.337+0000 7f30a1e75640 1 -- 172.21.15.63:0/3221867455 learned_addr learned my addr 172.21.15.63:0/3221867455 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:27:22.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.337+0000 7f30a1e75640 10 monclient(hunting): get_auth_request con 0x7f309c0a0bb0 auth_method 0 2024-04-26T04:27:22.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.337+0000 7f30a1e75640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:27:22.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.337+0000 7f30a1e75640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:27:22.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.337+0000 7f30a1e75640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:27:22.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.337+0000 7f30a0e73640 10 monclient(hunting): get_auth_request con 0x7f309c09e0d0 auth_method 0 2024-04-26T04:27:22.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.337+0000 7f30a0e73640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:27:22.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.337+0000 7f30a0e73640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:27:22.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.337+0000 7f30a0e73640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:27:22.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.337+0000 7f30a1e75640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:27:22.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.337+0000 7f30a1e75640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:27:22.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.337+0000 7f30a1e75640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:27:22.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.337+0000 7f30a0e73640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:27:22.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.337+0000 7f30a0e73640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:27:22.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.337+0000 7f30a0e73640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:27:22.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.341+0000 7f30a1674640 1 --2- 172.21.15.63:0/3221867455 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f309c09d7b0 0x7f309c1ce9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:22.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.341+0000 7f30a1674640 10 monclient(hunting): get_auth_request con 0x7f309c09d7b0 auth_method 0 2024-04-26T04:27:22.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.341+0000 7f30a1674640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:27:22.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.341+0000 7f30a1674640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:27:22.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.341+0000 7f30a1674640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:27:22.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.341+0000 7f30a1e75640 10 monclient(hunting): handle_auth_done global_id 24779 payload 824 2024-04-26T04:27:22.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.341+0000 7f30a1e75640 1 -- 172.21.15.63:0/3221867455 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f309c09d7b0 msgr2=0x7f309c1ce9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:27:22.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.341+0000 7f30a1e75640 1 --2- 172.21.15.63:0/3221867455 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f309c09d7b0 0x7f309c1ce9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:22.354 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.341+0000 7f30a1e75640 1 -- 172.21.15.63:0/3221867455 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f309c09e0d0 msgr2=0x7f309c097c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:27:22.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.341+0000 7f30a1e75640 1 --2- 172.21.15.63:0/3221867455 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f309c09e0d0 0x7f309c097c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:22.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.341+0000 7f30a1e75640 10 monclient: _finish_hunting 0 2024-04-26T04:27:22.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.341+0000 7f30a1e75640 1 monclient: found mon.noname-c 2024-04-26T04:27:22.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.341+0000 7f30a1e75640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:27:22.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.341+0000 7f30a1e75640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.63:3301/0 2024-04-26T04:27:22.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.341+0000 7f30a1e75640 1 -- 172.21.15.63:0/3221867455 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f309c092ae0 con 0x7f309c0a0bb0 2024-04-26T04:27:22.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.341+0000 7f30a0e73640 1 --2- 172.21.15.63:0/3221867455 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f309c09e0d0 0x7f309c097c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T04:27:22.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.341+0000 7f30a1e75640 1 -- 172.21.15.63:0/3221867455 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f309c0a0bb0 msgr2=0x7f309c098160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:27:22.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.341+0000 7f30a1e75640 1 --2- 172.21.15.63:0/3221867455 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f309c0a0bb0 0x7f309c098160 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f3094009e50 tx=0x7f309400c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:27:22.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.341+0000 7f3091ffb640 1 -- 172.21.15.63:0/3221867455 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f309400cd90 con 0x7f309c0a0bb0 2024-04-26T04:27:22.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.341+0000 7f3091ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:27:22.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.341+0000 7f3091ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:27:22.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.341+0000 7f3091ffb640 10 monclient: dump: 2024-04-26T04:27:22.355 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:27:22.355 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:27:22.356 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:22.356 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:22.356 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:27:22.356 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:27:22.356 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:27:22.356 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:27:22.356 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:27:22.356 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:22.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.341+0000 7f3091ffb640 10 monclient: _finish_auth 0 2024-04-26T04:27:22.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.341+0000 7f3091ffb640 10 monclient: _check_auth_tickets 2024-04-26T04:27:22.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.341+0000 7f3091ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:27:22.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.341+0000 7f3091ffb640 1 -- 172.21.15.63:0/3221867455 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30940039a0 con 0x7f309c0a0bb0 2024-04-26T04:27:22.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.341+0000 7f3091ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T04:27:22.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.341+0000 7f3091ffb640 1 -- 172.21.15.63:0/3221867455 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3094003d70 con 0x7f309c0a0bb0 2024-04-26T04:27:22.356 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.341+0000 7f30a38d8640 5 monclient: authenticate success, global_id 24779 2024-04-26T04:27:22.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.341+0000 7f3091ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:27:22.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.341+0000 7f3091ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:27:22.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.341+0000 7f3091ffb640 10 monclient: dump: 2024-04-26T04:27:22.357 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:27:22.357 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:27:22.357 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:22.357 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:22.357 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:27:22.357 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:27:22.357 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:27:22.357 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:27:22.357 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:27:22.357 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:22.357 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.341+0000 7f30a38d8640 10 monclient: _renew_subs 2024-04-26T04:27:22.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.341+0000 7f30937fe640 4 set_mon_vals no callback set 2024-04-26T04:27:22.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.341+0000 7f30a38d8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:27:22.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.341+0000 7f30a38d8640 1 -- 172.21.15.63:0/3221867455 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f309c1cef20 con 0x7f309c0a0bb0 2024-04-26T04:27:22.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.341+0000 7f30a38d8640 10 monclient: _renew_subs 2024-04-26T04:27:22.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.341+0000 7f30a38d8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:27:22.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.341+0000 7f30a38d8640 1 -- 172.21.15.63:0/3221867455 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f309c0946f0 con 0x7f309c0a0bb0 2024-04-26T04:27:22.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.341+0000 7f30a38d8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:27:22.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.341+0000 7f30a38d8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:27:22.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.341+0000 7f30a38d8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:27:22.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.341+0000 7f30a38d8640 1 -- 172.21.15.63:0/3221867455 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f309c062e10 con 0x7f309c0a0bb0 2024-04-26T04:27:22.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.341+0000 7f30a1e75640 1 --2- 172.21.15.63:0/3221867455 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f309c0a0bb0 0x7f309c098160 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f3094009e50 tx=0x7f309400c490 comp rx=0 tx=0).queue_received will delay after 1.71411e+09 on 0x7f3094048f40 osd_map(323..323 src has 1..323) 2024-04-26T04:27:22.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.341+0000 7f3091ffb640 1 -- 172.21.15.63:0/3221867455 <== mon.2 v2:172.21.15.63:3301/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7f309401b030 con 0x7f309c0a0bb0 2024-04-26T04:27:22.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.341+0000 7f3091ffb640 1 --2- 172.21.15.63:0/3221867455 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f3080035f10 0x7f30800383d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:22.359 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.345+0000 7f30a1674640 1 --2- 172.21.15.63:0/3221867455 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f3080035f10 0x7f30800383d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:22.359 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.345+0000 7f30a1674640 10 monclient: get_auth_request con 0x7f3080035f10 auth_method 0 2024-04-26T04:27:22.359 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.345+0000 7f30a1674640 1 --2- 172.21.15.63:0/3221867455 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f3080035f10 0x7f30800383d0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f309c099aa0 tx=0x7f308c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:27:22.359 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:22.349+0000 7f3091ffb640 1 -- 172.21.15.63:0/3221867455 <== mon.2 v2:172.21.15.63:3301/0 5 ==== osd_map(323..323 src has 1..323) ==== 4346+0+0 (secure 0 0 0) 0x7f3094048f40 con 0x7f309c0a0bb0 2024-04-26T04:27:23.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.169+0000 7f3091ffb640 1 -- 172.21.15.63:0/3221867455 <== mon.2 v2:172.21.15.63:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f3094022020 con 0x7f309c0a0bb0 2024-04-26T04:27:23.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.169+0000 7f3091ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:27:23.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.169+0000 7f3091ffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:27:23.346 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.345+0000 7f30a38d8640 1 -- 172.21.15.63:0/3221867455 --> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f309c0a3de0 con 0x7f3080035f10 2024-04-26T04:27:23.349 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.345+0000 7f3091ffb640 1 -- 172.21.15.63:0/3221867455 <== mgr.4105 v2:172.21.15.186:6824/2583706639 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+33787 (secure 0 0 0) 0x7f308c0af440 con 0x7f3080035f10 2024-04-26T04:27:23.350 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-26T04:27:23.350 INFO:teuthology.orchestra.run.smithi063.stderr:dumped all 2024-04-26T04:27:23.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.349+0000 7f30a38d8640 1 -- 172.21.15.63:0/3221867455 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f3080035f10 msgr2=0x7f30800383d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:27:23.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.349+0000 7f30a38d8640 1 --2- 172.21.15.63:0/3221867455 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f3080035f10 0x7f30800383d0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f309c099aa0 tx=0x7f308c008040 comp rx=0 tx=0).stop 2024-04-26T04:27:23.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.349+0000 7f30a38d8640 10 monclient: shutdown 2024-04-26T04:27:23.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.349+0000 7f30a38d8640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:27:23.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.349+0000 7f30a38d8640 1 -- 172.21.15.63:0/3221867455 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f309c0a0bb0 msgr2=0x7f309c098160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:27:23.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.349+0000 7f30a38d8640 1 --2- 172.21.15.63:0/3221867455 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f309c0a0bb0 0x7f309c098160 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f3094009e50 tx=0x7f309400c490 comp rx=0 tx=0).stop 2024-04-26T04:27:23.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.349+0000 7f30a38d8640 1 -- 172.21.15.63:0/3221867455 shutdown_connections 2024-04-26T04:27:23.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.349+0000 7f30a38d8640 1 --2- 172.21.15.63:0/3221867455 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7f3080035f10 0x7f30800383d0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:23.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.349+0000 7f30a38d8640 1 --2- 172.21.15.63:0/3221867455 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7f309c0a0bb0 0x7f309c098160 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:23.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.349+0000 7f30a38d8640 1 --2- 172.21.15.63:0/3221867455 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f309c09e0d0 0x7f309c097c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:23.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.349+0000 7f30a38d8640 1 --2- 172.21.15.63:0/3221867455 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f309c09d7b0 0x7f309c1ce9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:23.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.349+0000 7f30a38d8640 1 -- 172.21.15.63:0/3221867455 >> 172.21.15.63:0/3221867455 conn(0x7f309c0a1e70 msgr2=0x7f309c139680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:27:23.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.349+0000 7f30a38d8640 1 -- 172.21.15.63:0/3221867455 shutdown_connections 2024-04-26T04:27:23.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.349+0000 7f30a38d8640 1 -- 172.21.15.63:0/3221867455 wait complete. 2024-04-26T04:27:23.368 INFO:teuthology.orchestra.run.smithi063.stdout:{"pg_ready":true,"pg_map":{"version":577,"stamp":"2024-04-26T04:27:21.599533+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":205148,"kb_used_data":26408,"kb_used_omap":9,"kb_used_meta":178678,"kb_avail":566025892,"statfs":{"total":579820584960,"available":579610513408,"internally_reserved":0,"allocated":27041792,"data_stored":10728854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9567,"internal_metadata":182966945},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003463"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":626,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-04-26T04:27:03.959436+0000","last_change":"2024-04-26T04:27:03.959490+0000","last_active":"2024-04-26T04:27:03.959436+0000","last_peered":"2024-04-26T04:27:03.959436+0000","last_clean":"2024-04-26T04:27:03.959436+0000","last_became_active":"2024-04-26T04:18:47.131175+0000","last_became_peered":"2024-04-26T04:18:47.131175+0000","last_unstale":"2024-04-26T04:27:03.959436+0000","last_undegraded":"2024-04-26T04:27:03.959436+0000","last_fullsized":"2024-04-26T04:27:03.959436+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:27:03.959265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:27:03.959265+0000","last_clean_scrub_stamp":"2024-04-26T04:27:03.959265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:32:25.156705+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":619,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-04-26T04:27:00.038214+0000","last_change":"2024-04-26T04:27:00.038234+0000","last_active":"2024-04-26T04:27:00.038214+0000","last_peered":"2024-04-26T04:27:00.038214+0000","last_clean":"2024-04-26T04:27:00.038214+0000","last_became_active":"2024-04-26T04:18:46.498983+0000","last_became_peered":"2024-04-26T04:18:46.498983+0000","last_unstale":"2024-04-26T04:27:00.038214+0000","last_undegraded":"2024-04-26T04:27:00.038214+0000","last_fullsized":"2024-04-26T04:27:00.038214+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:27:00.038137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:27:00.038137+0000","last_clean_scrub_stamp":"2024-04-26T04:27:00.038137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:16:48.146301+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":627,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-04-26T04:27:03.001626+0000","last_change":"2024-04-26T04:27:03.001626+0000","last_active":"2024-04-26T04:27:03.001626+0000","last_peered":"2024-04-26T04:27:03.001626+0000","last_clean":"2024-04-26T04:27:03.001626+0000","last_became_active":"2024-04-26T04:18:46.507335+0000","last_became_peered":"2024-04-26T04:18:46.507335+0000","last_unstale":"2024-04-26T04:27:03.001626+0000","last_undegraded":"2024-04-26T04:27:03.001626+0000","last_fullsized":"2024-04-26T04:27:03.001626+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:27:03.001375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:27:03.001375+0000","last_clean_scrub_stamp":"2024-04-26T04:27:03.001375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:00:25.934032+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":621,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-04-26T04:27:01.031926+0000","last_change":"2024-04-26T04:27:01.031926+0000","last_active":"2024-04-26T04:27:01.031926+0000","last_peered":"2024-04-26T04:27:01.031926+0000","last_clean":"2024-04-26T04:27:01.031926+0000","last_became_active":"2024-04-26T04:18:46.506209+0000","last_became_peered":"2024-04-26T04:18:46.506209+0000","last_unstale":"2024-04-26T04:27:01.031926+0000","last_undegraded":"2024-04-26T04:27:01.031926+0000","last_fullsized":"2024-04-26T04:27:01.031926+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:27:01.031568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:27:01.031568+0000","last_clean_scrub_stamp":"2024-04-26T04:27:01.031568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T09:58:58.679850+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":623,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-04-26T04:27:02.608321+0000","last_change":"2024-04-26T04:27:02.608321+0000","last_active":"2024-04-26T04:27:02.608321+0000","last_peered":"2024-04-26T04:27:02.608321+0000","last_clean":"2024-04-26T04:27:02.608321+0000","last_became_active":"2024-04-26T04:18:46.504292+0000","last_became_peered":"2024-04-26T04:18:46.504292+0000","last_unstale":"2024-04-26T04:27:02.608321+0000","last_undegraded":"2024-04-26T04:27:02.608321+0000","last_fullsized":"2024-04-26T04:27:02.608321+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:27:02.608092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:27:02.608092+0000","last_clean_scrub_stamp":"2024-04-26T04:27:02.608092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T13:52:38.046993+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":615,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-04-26T04:27:01.130746+0000","last_change":"2024-04-26T04:27:01.130746+0000","last_active":"2024-04-26T04:27:01.130746+0000","last_peered":"2024-04-26T04:27:01.130746+0000","last_clean":"2024-04-26T04:27:01.130746+0000","last_became_active":"2024-04-26T04:18:46.503343+0000","last_became_peered":"2024-04-26T04:18:46.503343+0000","last_unstale":"2024-04-26T04:27:01.130746+0000","last_undegraded":"2024-04-26T04:27:01.130746+0000","last_fullsized":"2024-04-26T04:27:01.130746+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:27:01.130506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:27:01.130506+0000","last_clean_scrub_stamp":"2024-04-26T04:27:01.130506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:15:46.297840+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":627,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-04-26T04:27:02.024695+0000","last_change":"2024-04-26T04:27:02.024818+0000","last_active":"2024-04-26T04:27:02.024695+0000","last_peered":"2024-04-26T04:27:02.024695+0000","last_clean":"2024-04-26T04:27:02.024695+0000","last_became_active":"2024-04-26T04:18:46.502682+0000","last_became_peered":"2024-04-26T04:18:46.502682+0000","last_unstale":"2024-04-26T04:27:02.024695+0000","last_undegraded":"2024-04-26T04:27:02.024695+0000","last_fullsized":"2024-04-26T04:27:02.024695+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:27:02.024498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:27:02.024498+0000","last_clean_scrub_stamp":"2024-04-26T04:27:02.024498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T10:22:52.954284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":623,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-04-26T04:27:03.659005+0000","last_change":"2024-04-26T04:27:03.659005+0000","last_active":"2024-04-26T04:27:03.659005+0000","last_peered":"2024-04-26T04:27:03.659005+0000","last_clean":"2024-04-26T04:27:03.659005+0000","last_became_active":"2024-04-26T04:18:46.504332+0000","last_became_peered":"2024-04-26T04:18:46.504332+0000","last_unstale":"2024-04-26T04:27:03.659005+0000","last_undegraded":"2024-04-26T04:27:03.659005+0000","last_fullsized":"2024-04-26T04:27:03.659005+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-26T04:27:03.658779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-26T04:27:03.658779+0000","last_clean_scrub_stamp":"2024-04-26T04:27:03.658779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:14:59.423204+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"323'119","reported_seq":781,"reported_epoch":323,"state":"active+clean","last_fresh":"2024-04-26T04:27:01.072883+0000","last_change":"2024-04-26T04:27:01.073043+0000","last_active":"2024-04-26T04:27:01.072883+0000","last_peered":"2024-04-26T04:27:01.072883+0000","last_clean":"2024-04-26T04:27:01.072883+0000","last_became_active":"2024-04-26T04:18:45.504887+0000","last_became_peered":"2024-04-26T04:18:45.504887+0000","last_unstale":"2024-04-26T04:27:01.072883+0000","last_undegraded":"2024-04-26T04:27:01.072883+0000","last_fullsized":"2024-04-26T04:27:01.072883+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"323'119","last_scrub_stamp":"2024-04-26T04:27:01.072650+0000","last_deep_scrub":"323'119","last_deep_scrub_stamp":"2024-04-26T04:27:01.072650+0000","last_clean_scrub_stamp":"2024-04-26T04:27:01.072650+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T12:23:26.186978+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":1057856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":12,"seq":51539607661,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35924,"kb_used_data":4748,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94335916,"statfs":{"total":96636764160,"available":96599977984,"internally_reserved":0,"allocated":4861952,"data_stored":2140761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":12,"seq":51539607659,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35404,"kb_used_data":4228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94336436,"statfs":{"total":96636764160,"available":96600510464,"internally_reserved":0,"allocated":4329472,"data_stored":1611833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":12,"seq":51539607659,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35404,"kb_used_data":4228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94336436,"statfs":{"total":96636764160,"available":96600510464,"internally_reserved":0,"allocated":4329472,"data_stored":1611833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640363,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35404,"kb_used_data":4228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94336436,"statfs":{"total":96636764160,"available":96600510464,"internally_reserved":0,"allocated":4329472,"data_stored":1611833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640362,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31248,"kb_used_data":4228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340592,"statfs":{"total":96636764160,"available":96604766208,"internally_reserved":0,"allocated":4329472,"data_stored":1611833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640362,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31764,"kb_used_data":4748,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340076,"statfs":{"total":96636764160,"available":96604237824,"internally_reserved":0,"allocated":4861952,"data_stored":2140761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-26T04:27:23.370 DEBUG:teuthology.orchestra.run.smithi063:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-04-26T04:27:23.497 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.493+0000 7fe2d7619640 10 monclient: get_monmap_and_config 2024-04-26T04:27:23.497 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.493+0000 7fe2d7619640 10 monclient: build_initial_monmap 2024-04-26T04:27:23.497 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.493+0000 7fe2d7619640 10 monclient: monmap: 2024-04-26T04:27:23.497 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:27:23.497 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:27:23.497 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:27:23.497126+0000 2024-04-26T04:27:23.497 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:27:23.497126+0000 2024-04-26T04:27:23.497 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:27:23.497 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:27:23.497 INFO:teuthology.orchestra.run.smithi063.stderr:0: v1:172.21.15.63:6789/0 mon.noname-a-legacy 2024-04-26T04:27:23.497 INFO:teuthology.orchestra.run.smithi063.stderr:1: v1:172.21.15.186:6789/0 mon.noname-c-legacy 2024-04-26T04:27:23.497 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-b 2024-04-26T04:27:23.497 INFO:teuthology.orchestra.run.smithi063.stderr:3: v2:172.21.15.63:3300/0 mon.noname-a 2024-04-26T04:27:23.498 INFO:teuthology.orchestra.run.smithi063.stderr:4: v2:172.21.15.186:3300/0 mon.noname-c 2024-04-26T04:27:23.498 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:23.500 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.497+0000 7fe2d7619640 1 Processor -- start 2024-04-26T04:27:23.500 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.497+0000 7fe2d7619640 1 -- start start 2024-04-26T04:27:23.500 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.497+0000 7fe2d7619640 10 monclient: init 2024-04-26T04:27:23.500 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.497+0000 7fe2d7619640 10 monclient: _reopen_session rank -1 2024-04-26T04:27:23.500 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.497+0000 7fe2d7619640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-04-26T04:27:23.500 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.497+0000 7fe2d7619640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fe2d009c920 0x7fe2d009cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:23.500 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.497+0000 7fe2d7619640 10 monclient(hunting): picked mon.noname-b con 0x7fe2d009c920 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:27:23.501 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.497+0000 7fe2d7619640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7fe2d00a33d0 0x7fe2d00a0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:23.501 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.497+0000 7fe2d7619640 10 monclient(hunting): picked mon.noname-a con 0x7fe2d00a33d0 addr v2:172.21.15.63:3300/0 2024-04-26T04:27:23.501 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.497+0000 7fe2d7619640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe2d00a37b0 0x7fe2d00a0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:23.501 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.497+0000 7fe2d7619640 10 monclient(hunting): picked mon.noname-c con 0x7fe2d00a37b0 addr v2:172.21.15.186:3300/0 2024-04-26T04:27:23.501 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.497+0000 7fe2d7619640 10 monclient(hunting): start opening mon connection 2024-04-26T04:27:23.501 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.497+0000 7fe2d7619640 1 -- --> v2:172.21.15.63:3300/0 -- mon_getmap magic: 0 -- 0x7fe2d00a2a40 con 0x7fe2d00a33d0 2024-04-26T04:27:23.501 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.497+0000 7fe2d7619640 10 monclient(hunting): start opening mon connection 2024-04-26T04:27:23.501 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.497+0000 7fe2d7619640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 -- 0x7fe2d00a2760 con 0x7fe2d00a37b0 2024-04-26T04:27:23.501 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.497+0000 7fe2d7619640 10 monclient(hunting): start opening mon connection 2024-04-26T04:27:23.501 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.497+0000 7fe2d7619640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7fe2d00a56b0 con 0x7fe2d009c920 2024-04-26T04:27:23.501 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.497+0000 7fe2d7619640 10 monclient(hunting): _renew_subs 2024-04-26T04:27:23.501 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.497+0000 7fe2d7619640 10 monclient(hunting): authenticate will time out at 1475.204757s 2024-04-26T04:27:23.501 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.497+0000 7fe2d4bb4640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7fe2d00a33d0 0x7fe2d00a0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:23.501 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.497+0000 7fe2d53b5640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fe2d009c920 0x7fe2d009cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:23.502 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.497+0000 7fe2d53b5640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fe2d009c920 0x7fe2d009cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:43904/0 (socket says 172.21.15.63:43904) 2024-04-26T04:27:23.502 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.497+0000 7fe2d4bb4640 1 --2- >> v2:172.21.15.63:3300/0 conn(0x7fe2d00a33d0 0x7fe2d00a0ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:33754/0 (socket says 172.21.15.63:33754) 2024-04-26T04:27:23.502 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.497+0000 7fe2d53b5640 1 -- 172.21.15.63:0/1874584254 learned_addr learned my addr 172.21.15.63:0/1874584254 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:27:23.502 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.497+0000 7fe2d53b5640 10 monclient(hunting): get_auth_request con 0x7fe2d009c920 auth_method 0 2024-04-26T04:27:23.502 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.497+0000 7fe2d53b5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:27:23.502 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.497+0000 7fe2d53b5640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:27:23.502 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.497+0000 7fe2d53b5640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:27:23.502 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.497+0000 7fe2d4bb4640 10 monclient(hunting): get_auth_request con 0x7fe2d00a33d0 auth_method 0 2024-04-26T04:27:23.502 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.497+0000 7fe2d4bb4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:27:23.502 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.497+0000 7fe2d4bb4640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:27:23.502 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.497+0000 7fe2d4bb4640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:27:23.502 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.497+0000 7fe2d4bb4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:27:23.502 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.497+0000 7fe2d4bb4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:27:23.502 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.497+0000 7fe2d4bb4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:27:23.502 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.497+0000 7fe2d53b5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:27:23.503 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.497+0000 7fe2d53b5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:27:23.503 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.497+0000 7fe2d53b5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:27:23.503 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.497+0000 7fe2d5bb6640 1 --2- 172.21.15.63:0/1874584254 >> v2:172.21.15.186:3300/0 conn(0x7fe2d00a37b0 0x7fe2d00a0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:23.503 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.497+0000 7fe2d5bb6640 10 monclient(hunting): get_auth_request con 0x7fe2d00a37b0 auth_method 0 2024-04-26T04:27:23.503 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.497+0000 7fe2d5bb6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:27:23.503 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.497+0000 7fe2d5bb6640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:27:23.503 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.497+0000 7fe2d5bb6640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:27:23.503 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.497+0000 7fe2d53b5640 10 monclient(hunting): handle_auth_done global_id 24782 payload 274 2024-04-26T04:27:23.503 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.497+0000 7fe2d53b5640 1 -- 172.21.15.63:0/1874584254 >> v2:172.21.15.186:3300/0 conn(0x7fe2d00a37b0 msgr2=0x7fe2d00a0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:27:23.503 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.497+0000 7fe2d53b5640 1 --2- 172.21.15.63:0/1874584254 >> v2:172.21.15.186:3300/0 conn(0x7fe2d00a37b0 0x7fe2d00a0ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:23.503 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.497+0000 7fe2d53b5640 1 -- 172.21.15.63:0/1874584254 >> v2:172.21.15.63:3300/0 conn(0x7fe2d00a33d0 msgr2=0x7fe2d00a0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-26T04:27:23.503 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.497+0000 7fe2d53b5640 1 --2- 172.21.15.63:0/1874584254 >> v2:172.21.15.63:3300/0 conn(0x7fe2d00a33d0 0x7fe2d00a0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:23.503 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.497+0000 7fe2d53b5640 10 monclient: _finish_hunting 0 2024-04-26T04:27:23.503 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.497+0000 7fe2d53b5640 1 monclient: found mon.noname-b 2024-04-26T04:27:23.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.497+0000 7fe2d53b5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:27:23.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.497+0000 7fe2d53b5640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.63:3301/0 2024-04-26T04:27:23.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.497+0000 7fe2d53b5640 1 -- 172.21.15.63:0/1874584254 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2d00a2480 con 0x7fe2d009c920 2024-04-26T04:27:23.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.497+0000 7fe2d4bb4640 1 --2- 172.21.15.63:0/1874584254 >> v2:172.21.15.63:3300/0 conn(0x7fe2d00a33d0 0x7fe2d00a0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-26T04:27:23.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.501+0000 7fe2d53b5640 1 -- 172.21.15.63:0/1874584254 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fe2d009c920 msgr2=0x7fe2d009cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:27:23.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.501+0000 7fe2d53b5640 1 --2- 172.21.15.63:0/1874584254 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fe2d009c920 0x7fe2d009cd00 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe2d0136ca0 tx=0x7fe2cc01c380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=35b06d0b4ccd2232 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:27:23.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.501+0000 7fe2c7fff640 1 -- 172.21.15.63:0/1874584254 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2cc0b0070 con 0x7fe2d009c920 2024-04-26T04:27:23.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.501+0000 7fe2c7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:27:23.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.501+0000 7fe2c7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-04-26T04:27:23.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.501+0000 7fe2c7fff640 10 monclient: dump: 2024-04-26T04:27:23.508 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:27:23.508 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:27:23.508 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:23.508 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:23.508 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:27:23.508 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:27:23.509 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:27:23.509 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:27:23.509 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:27:23.509 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:23.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.501+0000 7fe2c7fff640 10 monclient: _finish_auth 0 2024-04-26T04:27:23.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.501+0000 7fe2c7fff640 10 monclient: _check_auth_tickets 2024-04-26T04:27:23.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.501+0000 7fe2c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:27:23.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.501+0000 7fe2c7fff640 1 -- 172.21.15.63:0/1874584254 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2cc0ab600 con 0x7fe2d009c920 2024-04-26T04:27:23.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.501+0000 7fe2d7619640 5 monclient: authenticate success, global_id 24782 2024-04-26T04:27:23.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.501+0000 7fe2d7619640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-04-26T04:27:23.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.501+0000 7fe2c7fff640 10 monclient: handle_config config(0 keys) 2024-04-26T04:27:23.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.501+0000 7fe2d7619640 10 monclient: get_monmap_and_config success 2024-04-26T04:27:23.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.501+0000 7fe2c7fff640 1 -- 172.21.15.63:0/1874584254 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2cc0ab9d0 con 0x7fe2d009c920 2024-04-26T04:27:23.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.501+0000 7fe2d7619640 4 set_mon_vals no callback set 2024-04-26T04:27:23.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.501+0000 7fe2c7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:27:23.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.501+0000 7fe2c7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:27:23.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.501+0000 7fe2c7fff640 10 monclient: dump: 2024-04-26T04:27:23.510 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:27:23.510 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:27:23.510 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:23.510 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:23.510 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:27:23.510 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:27:23.510 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:27:23.510 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:27:23.510 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:27:23.510 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:23.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.501+0000 7fe2d7619640 10 monclient: shutdown 2024-04-26T04:27:23.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.501+0000 7fe2d7619640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:27:23.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.501+0000 7fe2d7619640 1 -- 172.21.15.63:0/1874584254 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fe2d009c920 msgr2=0x7fe2d009cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:27:23.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.501+0000 7fe2d7619640 1 --2- 172.21.15.63:0/1874584254 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fe2d009c920 0x7fe2d009cd00 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe2d0136ca0 tx=0x7fe2cc01c380 comp rx=0 tx=0).stop 2024-04-26T04:27:23.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.501+0000 7fe2d7619640 1 -- 172.21.15.63:0/1874584254 shutdown_connections 2024-04-26T04:27:23.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.501+0000 7fe2d7619640 1 --2- 172.21.15.63:0/1874584254 >> v2:172.21.15.186:3300/0 conn(0x7fe2d00a37b0 0x7fe2d00a0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:23.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.501+0000 7fe2d7619640 1 --2- 172.21.15.63:0/1874584254 >> v2:172.21.15.63:3300/0 conn(0x7fe2d00a33d0 0x7fe2d00a0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:23.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.501+0000 7fe2d7619640 1 --2- 172.21.15.63:0/1874584254 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fe2d009c920 0x7fe2d009cd00 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:23.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.501+0000 7fe2d7619640 1 -- 172.21.15.63:0/1874584254 >> 172.21.15.63:0/1874584254 conn(0x7fe2d00a60b0 msgr2=0x7fe2d00a64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:27:23.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.501+0000 7fe2d7619640 1 -- 172.21.15.63:0/1874584254 shutdown_connections 2024-04-26T04:27:23.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.501+0000 7fe2d7619640 1 -- 172.21.15.63:0/1874584254 wait complete. 2024-04-26T04:27:23.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.501+0000 7fe2d7619640 10 monclient: build_initial_monmap 2024-04-26T04:27:23.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.501+0000 7fe2d7619640 10 monclient: monmap: 2024-04-26T04:27:23.511 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 0 2024-04-26T04:27:23.511 INFO:teuthology.orchestra.run.smithi063.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-04-26T04:27:23.512 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 0.000000 2024-04-26T04:27:23.512 INFO:teuthology.orchestra.run.smithi063.stderr:created 0.000000 2024-04-26T04:27:23.512 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 0 (unknown) 2024-04-26T04:27:23.512 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:27:23.512 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.noname-a 2024-04-26T04:27:23.512 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.noname-b 2024-04-26T04:27:23.512 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.noname-c 2024-04-26T04:27:23.512 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:23.512 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.501+0000 7fe2d7619640 1 Processor -- start 2024-04-26T04:27:23.512 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2d7619640 1 -- start start 2024-04-26T04:27:23.512 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2d7619640 10 monclient: init 2024-04-26T04:27:23.512 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2d7619640 10 monclient: _reopen_session rank -1 2024-04-26T04:27:23.512 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2d7619640 10 monclient: _add_conns ranks=[1,2,0] 2024-04-26T04:27:23.512 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2d7619640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe2d009c920 0x7fe2d01cf160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:23.512 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2d7619640 10 monclient(hunting): picked mon.noname-b con 0x7fe2d009c920 addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-26T04:27:23.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2d7619640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fe2d00a33d0 0x7fe2d013d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:23.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2d7619640 10 monclient(hunting): picked mon.noname-c con 0x7fe2d00a33d0 addr [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] 2024-04-26T04:27:23.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2d7619640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe2d00a37b0 0x7fe2d013d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:23.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2d7619640 10 monclient(hunting): picked mon.noname-a con 0x7fe2d00a37b0 addr [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-26T04:27:23.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2d7619640 10 monclient(hunting): start opening mon connection 2024-04-26T04:27:23.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2d7619640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe2d0140b80 con 0x7fe2d00a37b0 2024-04-26T04:27:23.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2d7619640 10 monclient(hunting): start opening mon connection 2024-04-26T04:27:23.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2d7619640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe2d0140a00 con 0x7fe2d009c920 2024-04-26T04:27:23.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2d7619640 10 monclient(hunting): start opening mon connection 2024-04-26T04:27:23.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2d7619640 1 -- --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_getmap magic: 0 -- 0x7fe2d0140d00 con 0x7fe2d00a33d0 2024-04-26T04:27:23.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2d7619640 10 monclient(hunting): _renew_subs 2024-04-26T04:27:23.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2d7619640 10 monclient(hunting): authenticate will time out at 1475.212770s 2024-04-26T04:27:23.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2d5bb6640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe2d00a37b0 0x7fe2d013d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:23.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2d4bb4640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fe2d00a33d0 0x7fe2d013d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:23.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2d4bb4640 1 --2- >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fe2d00a33d0 0x7fe2d013d070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3301/0 says I am v2:172.21.15.63:43906/0 (socket says 172.21.15.63:43906) 2024-04-26T04:27:23.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2d5bb6640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe2d00a37b0 0x7fe2d013d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:33764/0 (socket says 172.21.15.63:33764) 2024-04-26T04:27:23.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2d4bb4640 1 -- 172.21.15.63:0/3681206512 learned_addr learned my addr 172.21.15.63:0/3681206512 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-26T04:27:23.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2d4bb4640 10 monclient(hunting): get_auth_request con 0x7fe2d00a33d0 auth_method 0 2024-04-26T04:27:23.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2d4bb4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:27:23.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2d4bb4640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:27:23.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2d4bb4640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:27:23.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2d5bb6640 10 monclient(hunting): get_auth_request con 0x7fe2d00a37b0 auth_method 0 2024-04-26T04:27:23.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2d5bb6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:27:23.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2d5bb6640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:27:23.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2d5bb6640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:27:23.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2d4bb4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:27:23.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2d4bb4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:27:23.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2d4bb4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:27:23.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2d5bb6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-04-26T04:27:23.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2d5bb6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-04-26T04:27:23.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2d5bb6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-04-26T04:27:23.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2d53b5640 1 --2- 172.21.15.63:0/3681206512 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe2d009c920 0x7fe2d01cf160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:23.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2d53b5640 10 monclient(hunting): get_auth_request con 0x7fe2d009c920 auth_method 0 2024-04-26T04:27:23.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2d53b5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-04-26T04:27:23.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2d53b5640 10 monclient(hunting): _init_auth method 2 2024-04-26T04:27:23.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2d53b5640 10 monclient(hunting): _init_auth creating new auth 2024-04-26T04:27:23.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2d4bb4640 10 monclient(hunting): handle_auth_done global_id 24785 payload 824 2024-04-26T04:27:23.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2d4bb4640 1 -- 172.21.15.63:0/3681206512 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe2d009c920 msgr2=0x7fe2d01cf160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:27:23.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2d4bb4640 1 --2- 172.21.15.63:0/3681206512 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe2d009c920 0x7fe2d01cf160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:23.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2d4bb4640 1 -- 172.21.15.63:0/3681206512 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe2d00a37b0 msgr2=0x7fe2d013d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:27:23.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2d4bb4640 1 --2- 172.21.15.63:0/3681206512 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe2d00a37b0 0x7fe2d013d5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:23.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2d4bb4640 10 monclient: _finish_hunting 0 2024-04-26T04:27:23.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2d4bb4640 1 monclient: found mon.noname-c 2024-04-26T04:27:23.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2d4bb4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-04-26T04:27:23.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2d4bb4640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.63:3301/0 2024-04-26T04:27:23.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2d4bb4640 1 -- 172.21.15.63:0/3681206512 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2d0138f20 con 0x7fe2d00a33d0 2024-04-26T04:27:23.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2d4bb4640 1 -- 172.21.15.63:0/3681206512 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fe2d00a33d0 msgr2=0x7fe2d013d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-04-26T04:27:23.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2d4bb4640 1 --2- 172.21.15.63:0/3681206512 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fe2d00a33d0 0x7fe2d013d070 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe2c000ebe0 tx=0x7fe2c000c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:27:23.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2c5ffb640 1 -- 172.21.15.63:0/3681206512 <== mon.2 v2:172.21.15.63:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2c0017070 con 0x7fe2d00a33d0 2024-04-26T04:27:23.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2c5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:27:23.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2c5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-04-26T04:27:23.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2c5ffb640 10 monclient: dump: 2024-04-26T04:27:23.518 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:27:23.518 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:27:23.518 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:23.518 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:23.518 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:27:23.519 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:27:23.519 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:27:23.519 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:27:23.519 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:27:23.519 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:23.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2c5ffb640 10 monclient: _finish_auth 0 2024-04-26T04:27:23.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2c5ffb640 10 monclient: _check_auth_tickets 2024-04-26T04:27:23.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2c5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-04-26T04:27:23.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2c5ffb640 1 -- 172.21.15.63:0/3681206512 <== mon.2 v2:172.21.15.63:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2c000ce40 con 0x7fe2d00a33d0 2024-04-26T04:27:23.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2d7619640 5 monclient: authenticate success, global_id 24785 2024-04-26T04:27:23.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2d7619640 10 monclient: _renew_subs 2024-04-26T04:27:23.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2d7619640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:27:23.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2d7619640 1 -- 172.21.15.63:0/3681206512 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2d009c310 con 0x7fe2d00a33d0 2024-04-26T04:27:23.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2c5ffb640 10 monclient: handle_config config(0 keys) 2024-04-26T04:27:23.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2c5ffb640 1 -- 172.21.15.63:0/3681206512 <== mon.2 v2:172.21.15.63:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2c0013690 con 0x7fe2d00a33d0 2024-04-26T04:27:23.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2c5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-04-26T04:27:23.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2c5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-04-26T04:27:23.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2c5ffb640 10 monclient: dump: 2024-04-26T04:27:23.520 INFO:teuthology.orchestra.run.smithi063.stderr:epoch 1 2024-04-26T04:27:23.520 INFO:teuthology.orchestra.run.smithi063.stderr:fsid cd1c6a0c-4e4b-4014-89c0-2e535479ef39 2024-04-26T04:27:23.520 INFO:teuthology.orchestra.run.smithi063.stderr:last_changed 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:23.520 INFO:teuthology.orchestra.run.smithi063.stderr:created 2024-04-26T04:18:03.702339+0000 2024-04-26T04:27:23.520 INFO:teuthology.orchestra.run.smithi063.stderr:min_mon_release 19 (squid) 2024-04-26T04:27:23.520 INFO:teuthology.orchestra.run.smithi063.stderr:election_strategy: 1 2024-04-26T04:27:23.520 INFO:teuthology.orchestra.run.smithi063.stderr:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-04-26T04:27:23.520 INFO:teuthology.orchestra.run.smithi063.stderr:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-04-26T04:27:23.520 INFO:teuthology.orchestra.run.smithi063.stderr:2: [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] mon.c 2024-04-26T04:27:23.520 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:23.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2c77fe640 4 set_mon_vals no callback set 2024-04-26T04:27:23.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2d7619640 10 monclient: _renew_subs 2024-04-26T04:27:23.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2d7619640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:27:23.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.505+0000 7fe2d7619640 1 -- 172.21.15.63:0/3681206512 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2d0073370 con 0x7fe2d00a33d0 2024-04-26T04:27:23.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.509+0000 7fe2d7619640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-04-26T04:27:23.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.509+0000 7fe2d7619640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:27:23.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.509+0000 7fe2d7619640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:27:23.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.509+0000 7fe2c5ffb640 1 -- 172.21.15.63:0/3681206512 <== mon.2 v2:172.21.15.63:3301/0 4 ==== mgrmap(e 5) ==== 44053+0+0 (secure 0 0 0) 0x7fe2c001a070 con 0x7fe2d00a33d0 2024-04-26T04:27:23.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.509+0000 7fe2d7619640 1 -- 172.21.15.63:0/3681206512 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2d0062e10 con 0x7fe2d00a33d0 2024-04-26T04:27:23.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.509+0000 7fe2c5ffb640 1 --2- 172.21.15.63:0/3681206512 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fe2b4035ef0 0x7fe2b40383b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-26T04:27:23.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.509+0000 7fe2c5ffb640 1 -- 172.21.15.63:0/3681206512 <== mon.2 v2:172.21.15.63:3301/0 5 ==== osd_map(323..323 src has 1..323) ==== 4346+0+0 (secure 0 0 0) 0x7fe2c0048d10 con 0x7fe2d00a33d0 2024-04-26T04:27:23.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.509+0000 7fe2d53b5640 1 --2- 172.21.15.63:0/3681206512 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fe2b4035ef0 0x7fe2b40383b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-26T04:27:23.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.509+0000 7fe2d53b5640 10 monclient: get_auth_request con 0x7fe2b4035ef0 auth_method 0 2024-04-26T04:27:23.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.513+0000 7fe2d53b5640 1 --2- 172.21.15.63:0/3681206512 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fe2b4035ef0 0x7fe2b40383b0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fe2d004f740 tx=0x7fe2cc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-26T04:27:23.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.513+0000 7fe2c5ffb640 1 -- 172.21.15.63:0/3681206512 <== mon.2 v2:172.21.15.63:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fe2d009c310 con 0x7fe2d00a33d0 2024-04-26T04:27:23.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.513+0000 7fe2c5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-04-26T04:27:23.522 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.513+0000 7fe2c5ffb640 10 monclient: _finish_command 1 = system:0 2024-04-26T04:27:23.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.673+0000 7fe2d7619640 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-04-26T04:27:23.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.673+0000 7fe2d7619640 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-04-26T04:27:23.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.673+0000 7fe2d7619640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.63:3301/0 2024-04-26T04:27:23.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.673+0000 7fe2d7619640 1 -- 172.21.15.63:0/3681206512 --> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7fe2d0075990 con 0x7fe2d00a33d0 2024-04-26T04:27:23.685 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.681+0000 7fe2c5ffb640 1 -- 172.21.15.63:0/3681206512 <== mon.2 v2:172.21.15.63:3301/0 7 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fe2c0011510 con 0x7fe2d00a33d0 2024-04-26T04:27:23.685 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.681+0000 7fe2c5ffb640 10 monclient: handle_config config(1 keys) 2024-04-26T04:27:23.685 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.681+0000 7fe2c77fe640 4 set_mon_vals no callback set 2024-04-26T04:27:23.685 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.681+0000 7fe2c77fe640 10 set_mon_vals mon_health_to_clog = false 2024-04-26T04:27:23.685 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.681+0000 7fe2c5ffb640 1 -- 172.21.15.63:0/3681206512 <== mon.2 v2:172.21.15.63:3301/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v6)=0 v6) ==== 123+0+0 (secure 0 0 0) 0x7fe2c00478c0 con 0x7fe2d00a33d0 2024-04-26T04:27:23.685 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.681+0000 7fe2c5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-04-26T04:27:23.685 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.681+0000 7fe2c5ffb640 10 monclient: _finish_command 2 = system:0 2024-04-26T04:27:23.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.685+0000 7fe2d7619640 1 -- 172.21.15.63:0/3681206512 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fe2b4035ef0 msgr2=0x7fe2b40383b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:27:23.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.685+0000 7fe2d7619640 1 --2- 172.21.15.63:0/3681206512 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fe2b4035ef0 0x7fe2b40383b0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fe2d004f740 tx=0x7fe2cc008040 comp rx=0 tx=0).stop 2024-04-26T04:27:23.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.685+0000 7fe2d7619640 10 monclient: shutdown 2024-04-26T04:27:23.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.685+0000 7fe2d7619640 20 monclient: shutdown discarding 0 pending message(s) 2024-04-26T04:27:23.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.685+0000 7fe2d7619640 1 -- 172.21.15.63:0/3681206512 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fe2d00a33d0 msgr2=0x7fe2d013d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-26T04:27:23.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.685+0000 7fe2d7619640 1 --2- 172.21.15.63:0/3681206512 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fe2d00a33d0 0x7fe2d013d070 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe2c000ebe0 tx=0x7fe2c000c520 comp rx=0 tx=0).stop 2024-04-26T04:27:23.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.685+0000 7fe2d7619640 1 -- 172.21.15.63:0/3681206512 shutdown_connections 2024-04-26T04:27:23.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.685+0000 7fe2d7619640 1 --2- 172.21.15.63:0/3681206512 >> [v2:172.21.15.186:6824/2583706639,v1:172.21.15.186:6825/2583706639] conn(0x7fe2b4035ef0 0x7fe2b40383b0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:23.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.685+0000 7fe2d7619640 1 --2- 172.21.15.63:0/3681206512 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe2d00a37b0 0x7fe2d013d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:23.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.685+0000 7fe2d7619640 1 --2- 172.21.15.63:0/3681206512 >> [v2:172.21.15.63:3301/0,v1:172.21.15.63:6790/0] conn(0x7fe2d00a33d0 0x7fe2d013d070 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:23.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.685+0000 7fe2d7619640 1 --2- 172.21.15.63:0/3681206512 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe2d009c920 0x7fe2d01cf160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-26T04:27:23.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.685+0000 7fe2d7619640 1 -- 172.21.15.63:0/3681206512 >> 172.21.15.63:0/3681206512 conn(0x7fe2d00a60b0 msgr2=0x7fe2d009f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-26T04:27:23.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.685+0000 7fe2d7619640 1 -- 172.21.15.63:0/3681206512 shutdown_connections 2024-04-26T04:27:23.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-26T04:27:23.685+0000 7fe2d7619640 1 -- 172.21.15.63:0/3681206512 wait complete. 2024-04-26T04:27:23.706 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-26T04:27:23.707 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-26T04:27:23.707 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-26T04:27:23.707 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T04:27:23.805 INFO:tasks.ceph.osd.0:Stopped 2024-04-26T04:27:23.805 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-26T04:27:23.805 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T04:27:23.892 INFO:tasks.ceph.osd.1:Stopped 2024-04-26T04:27:23.892 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-26T04:27:23.892 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T04:27:23.986 INFO:tasks.ceph.osd.2:Stopped 2024-04-26T04:27:23.986 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-26T04:27:23.986 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T04:27:28.732 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-04-26T04:27:30.088 INFO:tasks.ceph.osd.3:Stopped 2024-04-26T04:27:30.088 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-04-26T04:27:30.088 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T04:27:30.150 INFO:tasks.ceph.osd.4:Stopped 2024-04-26T04:27:30.150 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-04-26T04:27:30.151 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T04:27:30.245 INFO:tasks.ceph.osd.5:Stopped 2024-04-26T04:27:30.246 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-26T04:27:30.246 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-26T04:27:30.246 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T04:27:30.276 INFO:tasks.ceph.mgr.x:Stopped 2024-04-26T04:27:30.277 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-26T04:27:30.277 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-26T04:27:30.277 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T04:27:30.295 INFO:tasks.ceph.mon.a:Stopped 2024-04-26T04:27:30.295 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-04-26T04:27:30.295 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T04:27:30.362 INFO:tasks.ceph.mon.c:Stopped 2024-04-26T04:27:30.362 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-04-26T04:27:30.362 INFO:teuthology.orchestra.run:waiting for 300 2024-04-26T04:27:30.387 INFO:tasks.ceph.mon.b:Stopped 2024-04-26T04:27:30.387 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-26T04:27:30.387 DEBUG:teuthology.orchestra.run.smithi063:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v ScrubResult | egrep -v 'scrub mismatch' | egrep -v 'overall HEALTH_' | egrep -v '\(MGR_DOWN\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | head -n 1 2024-04-26T04:27:30.425 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi063.front.sepia.ceph.com 2024-04-26T04:27:30.425 DEBUG:teuthology.orchestra.run.smithi063:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-26T04:27:30.722 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi063.front.sepia.ceph.com 2024-04-26T04:27:30.723 DEBUG:teuthology.orchestra.run.smithi063:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-26T04:27:30.935 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi063.front.sepia.ceph.com 2024-04-26T04:27:30.936 DEBUG:teuthology.orchestra.run.smithi063:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-26T04:27:31.103 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi186.front.sepia.ceph.com 2024-04-26T04:27:31.105 DEBUG:teuthology.orchestra.run.smithi186:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-04-26T04:27:31.311 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi186.front.sepia.ceph.com 2024-04-26T04:27:31.311 DEBUG:teuthology.orchestra.run.smithi186:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-04-26T04:27:31.452 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi186.front.sepia.ceph.com 2024-04-26T04:27:31.452 DEBUG:teuthology.orchestra.run.smithi186:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-04-26T04:27:31.576 INFO:tasks.ceph:Archiving mon data... 2024-04-26T04:27:31.577 DEBUG:teuthology.misc:Transferring archived files from smithi063:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2024-04-26_01:23:56-rados-wip-yuri2-testing-2024-04-24-0914-squid-distro-default-smithi/7674515/data/mon.a.tgz 2024-04-26T04:27:31.577 DEBUG:teuthology.orchestra.run.smithi063:> mktemp 2024-04-26T04:27:31.583 INFO:teuthology.orchestra.run.smithi063.stdout:/tmp/tmp.B5gIFHvyCF 2024-04-26T04:27:31.584 DEBUG:teuthology.orchestra.run.smithi063:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.B5gIFHvyCF 2024-04-26T04:27:31.678 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod 0666 /tmp/tmp.B5gIFHvyCF 2024-04-26T04:27:31.741 DEBUG:teuthology.orchestra.remote:smithi063:/tmp/tmp.B5gIFHvyCF is 189KB 2024-04-26T04:27:31.758 DEBUG:teuthology.orchestra.run.smithi063:> rm -fr /tmp/tmp.B5gIFHvyCF 2024-04-26T04:27:31.763 DEBUG:teuthology.misc:Transferring archived files from smithi063:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2024-04-26_01:23:56-rados-wip-yuri2-testing-2024-04-24-0914-squid-distro-default-smithi/7674515/data/mon.c.tgz 2024-04-26T04:27:31.763 DEBUG:teuthology.orchestra.run.smithi063:> mktemp 2024-04-26T04:27:31.809 INFO:teuthology.orchestra.run.smithi063.stdout:/tmp/tmp.qgYDbjIn12 2024-04-26T04:27:31.809 DEBUG:teuthology.orchestra.run.smithi063:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.qgYDbjIn12 2024-04-26T04:27:31.912 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod 0666 /tmp/tmp.qgYDbjIn12 2024-04-26T04:27:31.977 DEBUG:teuthology.orchestra.remote:smithi063:/tmp/tmp.qgYDbjIn12 is 194KB 2024-04-26T04:27:31.995 DEBUG:teuthology.orchestra.run.smithi063:> rm -fr /tmp/tmp.qgYDbjIn12 2024-04-26T04:27:32.000 DEBUG:teuthology.misc:Transferring archived files from smithi186:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2024-04-26_01:23:56-rados-wip-yuri2-testing-2024-04-24-0914-squid-distro-default-smithi/7674515/data/mon.b.tgz 2024-04-26T04:27:32.000 DEBUG:teuthology.orchestra.run.smithi186:> mktemp 2024-04-26T04:27:32.006 INFO:teuthology.orchestra.run.smithi186.stdout:/tmp/tmp.B5e7H1ZaML 2024-04-26T04:27:32.006 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.B5e7H1ZaML 2024-04-26T04:27:32.103 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 0666 /tmp/tmp.B5e7H1ZaML 2024-04-26T04:27:32.168 DEBUG:teuthology.orchestra.remote:smithi186:/tmp/tmp.B5e7H1ZaML is 193KB 2024-04-26T04:27:32.183 DEBUG:teuthology.orchestra.run.smithi186:> rm -fr /tmp/tmp.B5e7H1ZaML 2024-04-26T04:27:32.188 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-26T04:27:32.188 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-26T04:27:32.191 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-26T04:27:32.250 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-26T04:27:32.292 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-26T04:27:32.292 INFO:tasks.ceph:Archiving crash dumps... 2024-04-26T04:27:32.294 DEBUG:teuthology.misc:Transferring archived files from smithi063:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-04-26_01:23:56-rados-wip-yuri2-testing-2024-04-24-0914-squid-distro-default-smithi/7674515/remote/smithi063/crash 2024-04-26T04:27:32.294 DEBUG:teuthology.orchestra.run.smithi063:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-26T04:27:32.308 DEBUG:teuthology.misc:Transferring archived files from smithi186:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-04-26_01:23:56-rados-wip-yuri2-testing-2024-04-24-0914-squid-distro-default-smithi/7674515/remote/smithi186/crash 2024-04-26T04:27:32.309 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-26T04:27:32.347 INFO:tasks.ceph:Compressing logs... 2024-04-26T04:27:32.348 DEBUG:teuthology.orchestra.run.smithi063:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-26T04:27:32.355 DEBUG:teuthology.orchestra.run.smithi186:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-26T04:27:32.410 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31129.log 2024-04-26T04:27:32.410 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29979.log 2024-04-26T04:27:32.411 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.31129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27717.log 2024-04-26T04:27:32.422 INFO:teuthology.orchestra.run.smithi186.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.31129.log.gz 2024-04-26T04:27:32.422 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.29979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28534.log 2024-04-26T04:27:32.422 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.27717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29753.log 2024-04-26T04:27:32.422 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.28534.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.29979.log.gz 2024-04-26T04:27:32.422 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30188.log 2024-04-26T04:27:32.422 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.29753.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29841.log 2024-04-26T04:27:32.423 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.30188.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.28534.log.gz 2024-04-26T04:27:32.423 INFO:teuthology.orchestra.run.smithi186.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.27717.log.gz 2024-04-26T04:27:32.423 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29319.log 2024-04-26T04:27:32.423 INFO:teuthology.orchestra.run.smithi186.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.29753.log.gz 2024-04-26T04:27:32.423 INFO:teuthology.orchestra.run.smithi186.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.30188.log.gz 2024-04-26T04:27:32.423 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.29841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30763.log 2024-04-26T04:27:32.423 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.29319.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28414.log 2024-04-26T04:27:32.423 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.30763.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29491.log 2024-04-26T04:27:32.423 INFO:teuthology.orchestra.run.smithi186.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.29841.log.gz 2024-04-26T04:27:32.423 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.28414.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31583.log 2024-04-26T04:27:32.424 INFO:teuthology.orchestra.run.smithi186.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.29319.log.gz 2024-04-26T04:27:32.424 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.29491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27839.log 2024-04-26T04:27:32.424 INFO:teuthology.orchestra.run.smithi186.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.30763.log.gz 2024-04-26T04:27:32.424 INFO:teuthology.orchestra.run.smithi186.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.28414.log.gz 2024-04-26T04:27:32.424 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.31583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29231.log 2024-04-26T04:27:32.424 INFO:teuthology.orchestra.run.smithi186.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.29491.log.gz 2024-04-26T04:27:32.424 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.27839.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31767.log 2024-04-26T04:27:32.424 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.29231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30067.log 2024-04-26T04:27:32.424 INFO:teuthology.orchestra.run.smithi186.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.27839.log.gz 2024-04-26T04:27:32.424 INFO:teuthology.orchestra.run.smithi186.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.31583.log.gz 2024-04-26T04:27:32.424 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.31767.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.29231.log.gz 2024-04-26T04:27:32.424 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29370.log 2024-04-26T04:27:32.425 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.30067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28882.log 2024-04-26T04:27:32.425 INFO:teuthology.orchestra.run.smithi186.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.31767.log.gz 2024-04-26T04:27:32.425 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.29370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31311.log 2024-04-26T04:27:32.425 INFO:teuthology.orchestra.run.smithi186.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.30067.log.gz 2024-04-26T04:27:32.425 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.28882.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28971.log 2024-04-26T04:27:32.425 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.31311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29665.log 2024-04-26T04:27:32.425 INFO:teuthology.orchestra.run.smithi186.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.28882.log.gz 2024-04-26T04:27:32.425 INFO:teuthology.orchestra.run.smithi186.stderr: 88.4%/var/log/ceph/ceph-client.admin.28971.log: -- replaced with /var/log/ceph/ceph-client.admin.31311.log.gz 2024-04-26T04:27:32.425 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28969.log 2024-04-26T04:27:32.425 INFO:teuthology.orchestra.run.smithi186.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.29370.log.gz 2024-04-26T04:27:32.425 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.29665.log: /var/log/ceph/ceph-client.admin.31037.log 2024-04-26T04:27:32.425 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.28969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29752.log 2024-04-26T04:27:32.426 INFO:teuthology.orchestra.run.smithi186.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.28971.log.gz 2024-04-26T04:27:32.426 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.31037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30450.log 2024-04-26T04:27:32.426 INFO:teuthology.orchestra.run.smithi186.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.29665.log.gz 2024-04-26T04:27:32.426 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.29752.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.31037.log.gz 2024-04-26T04:27:32.426 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28066.log 2024-04-26T04:27:32.426 INFO:teuthology.orchestra.run.smithi186.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.28969.log.gz 2024-04-26T04:27:32.426 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.30450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31403.log 2024-04-26T04:27:32.426 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.28066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28501.log 2024-04-26T04:27:32.426 INFO:teuthology.orchestra.run.smithi186.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.30450.log.gz 2024-04-26T04:27:32.426 INFO:teuthology.orchestra.run.smithi186.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.29752.log.gz 2024-04-26T04:27:32.427 INFO:teuthology.orchestra.run.smithi186.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.28066.log.gz 2024-04-26T04:27:32.427 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.31403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28795.log 2024-04-26T04:27:32.450 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28834.log 2024-04-26T04:27:32.550 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28534.log 2024-04-26T04:27:32.550 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.28834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29639.log 2024-04-26T04:27:32.550 INFO:teuthology.orchestra.run.smithi063.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.28834.log.gz 2024-04-26T04:27:32.550 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.28534.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28358.log 2024-04-26T04:27:32.550 INFO:teuthology.orchestra.run.smithi063.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.28534.log.gz 2024-04-26T04:27:32.550 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29693.log 2024-04-26T04:27:32.550 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.29639.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30016.log 2024-04-26T04:27:32.550 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.28358.log: /var/log/ceph/ceph-client.admin.29693.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-04-26T04:27:32.551 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.30016.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.29693.log.gz 89.6% 2024-04-26T04:27:32.551 INFO:teuthology.orchestra.run.smithi063.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29639.log.gz 2024-04-26T04:27:32.551 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29771.log 2024-04-26T04:27:32.551 INFO:teuthology.orchestra.run.smithi063.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.28358.log.gz 2024-04-26T04:27:32.551 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31600.log 2024-04-26T04:27:32.551 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-osd.0.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.30016.log.gz 2024-04-26T04:27:32.551 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.29771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31418.log 2024-04-26T04:27:32.551 INFO:teuthology.orchestra.run.smithi063.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.29771.log.gz/var/log/ceph/ceph-client.admin.31600.log: 2024-04-26T04:27:32.551 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29116.log 2024-04-26T04:27:32.552 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.31418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29168.log 2024-04-26T04:27:32.552 INFO:teuthology.orchestra.run.smithi063.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.31600.log.gz 2024-04-26T04:27:32.552 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30588.log 2024-04-26T04:27:32.552 INFO:teuthology.orchestra.run.smithi063.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.31418.log.gz 2024-04-26T04:27:32.552 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.29116.log: /var/log/ceph/ceph-client.admin.29168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28531.log 2024-04-26T04:27:32.552 INFO:teuthology.orchestra.run.smithi063.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.29116.log.gz 2024-04-26T04:27:32.552 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.30588.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.29168.log.gz 2024-04-26T04:27:32.552 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31236.log 2024-04-26T04:27:32.552 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.28531.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30081.log 2024-04-26T04:27:32.552 INFO:teuthology.orchestra.run.smithi063.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.30588.log.gz 2024-04-26T04:27:32.552 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.31236.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.28531.log.gz 2024-04-26T04:27:32.552 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29142.log 2024-04-26T04:27:32.552 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.30081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31027.log 2024-04-26T04:27:32.553 INFO:teuthology.orchestra.run.smithi063.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.31236.log.gz 2024-04-26T04:27:32.553 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.29142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30562.log 2024-04-26T04:27:32.553 INFO:teuthology.orchestra.run.smithi063.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.30081.log.gz 2024-04-26T04:27:32.553 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.31027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31678.log 2024-04-26T04:27:32.553 INFO:teuthology.orchestra.run.smithi063.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.29142.log.gz 2024-04-26T04:27:32.553 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.30562.log: -- /var/log/ceph/ceph-client.admin.28692.log 2024-04-26T04:27:32.553 INFO:teuthology.orchestra.run.smithi063.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.31027.log.gz 2024-04-26T04:27:32.553 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.31678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29378.log 2024-04-26T04:27:32.553 INFO:teuthology.orchestra.run.smithi063.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.30562.log.gz 2024-04-26T04:27:32.553 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.28692.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-04-26T04:27:32.553 INFO:teuthology.orchestra.run.smithi063.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.31678.log.gz 2024-04-26T04:27:32.553 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.29378.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.28692.log.gz 2024-04-26T04:27:32.553 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30772.log 2024-04-26T04:27:32.554 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31001.log 2024-04-26T04:27:32.554 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.30772.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.29378.log.gz 2024-04-26T04:27:32.554 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28428.log 2024-04-26T04:27:32.554 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.31001.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.30772.log.gz 2024-04-26T04:27:32.554 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28333.log 2024-04-26T04:27:32.554 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30242.log 2024-04-26T04:27:32.554 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.28428.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.31001.log.gz 2024-04-26T04:27:32.554 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.28333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28384.log 2024-04-26T04:27:32.554 INFO:teuthology.orchestra.run.smithi063.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.28428.log.gz 2024-04-26T04:27:32.554 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.30242.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.28333.log.gz 2024-04-26T04:27:32.554 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28679.log 2024-04-26T04:27:32.554 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.28384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29719.log 2024-04-26T04:27:32.555 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.28679.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.30242.log.gz 2024-04-26T04:27:32.555 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-04-26T04:27:32.555 INFO:teuthology.orchestra.run.smithi063.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.28384.log.gz 2024-04-26T04:27:32.555 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.29719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30107.log 2024-04-26T04:27:32.555 INFO:teuthology.orchestra.run.smithi063.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.28679.log.gz 2024-04-26T04:27:32.555 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24079.log 2024-04-26T04:27:32.555 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.30107.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.29719.log.gz 2024-04-26T04:27:32.555 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31288.log 2024-04-26T04:27:32.555 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.24079.log: gzip 91.8% -- replaced with /var/log/ceph/ceph-client.admin.30107.log.gz 2024-04-26T04:27:32.555 INFO:teuthology.orchestra.run.smithi063.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.30826.log 2024-04-26T04:27:32.555 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.31288.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31652.log 2024-04-26T04:27:32.555 INFO:teuthology.orchestra.run.smithi063.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.24079.log.gz 2024-04-26T04:27:32.555 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.30826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31079.log 2024-04-26T04:27:32.556 INFO:teuthology.orchestra.run.smithi063.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.31288.log.gz 2024-04-26T04:27:32.556 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.31652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28533.log 2024-04-26T04:27:32.556 INFO:teuthology.orchestra.run.smithi063.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.30826.log.gz 2024-04-26T04:27:32.556 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.31079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31366.log 2024-04-26T04:27:32.556 INFO:teuthology.orchestra.run.smithi063.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.31652.log.gz 2024-04-26T04:27:32.556 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.28533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28694.log 2024-04-26T04:27:32.556 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.31366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28693.log 2024-04-26T04:27:32.556 INFO:teuthology.orchestra.run.smithi063.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.28533.log.gz 2024-04-26T04:27:32.556 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.28694.log: 92.6% -- replaced with /var/log/ceph/ceph-client.admin.31079.log.gz 2024-04-26T04:27:32.556 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30694.log 2024-04-26T04:27:32.556 INFO:teuthology.orchestra.run.smithi063.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.31366.log.gz 2024-04-26T04:27:32.556 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.28693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31522.log 2024-04-26T04:27:32.556 INFO:teuthology.orchestra.run.smithi063.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.28694.log.gz 2024-04-26T04:27:32.557 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.30694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30348.log 2024-04-26T04:27:32.557 INFO:teuthology.orchestra.run.smithi063.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.28693.log.gz 2024-04-26T04:27:32.557 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.31522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31132.log 2024-04-26T04:27:32.557 INFO:teuthology.orchestra.run.smithi063.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.30694.log.gz 2024-04-26T04:27:32.557 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.30348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28492.log 2024-04-26T04:27:32.557 INFO:teuthology.orchestra.run.smithi063.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.31522.log.gz 2024-04-26T04:27:32.557 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.31132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29274.log 2024-04-26T04:27:32.557 INFO:teuthology.orchestra.run.smithi063.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.30348.log.gz 2024-04-26T04:27:32.557 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.28492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29404.log 2024-04-26T04:27:32.557 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.29274.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.31132.log.gz 2024-04-26T04:27:32.557 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28890.log 2024-04-26T04:27:32.557 INFO:teuthology.orchestra.run.smithi063.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.28492.log.gz 2024-04-26T04:27:32.557 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.29404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30936.log 2024-04-26T04:27:32.558 INFO:teuthology.orchestra.run.smithi063.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.29274.log.gz 2024-04-26T04:27:32.558 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.28890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29326.log 2024-04-26T04:27:32.558 INFO:teuthology.orchestra.run.smithi063.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.29404.log.gz 2024-04-26T04:27:32.558 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.30936.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30374.log 2024-04-26T04:27:32.558 INFO:teuthology.orchestra.run.smithi063.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.28890.log.gz 2024-04-26T04:27:32.558 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.29326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29064.log 2024-04-26T04:27:32.558 INFO:teuthology.orchestra.run.smithi063.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.30936.log.gz 2024-04-26T04:27:32.558 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.30374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31574.log 2024-04-26T04:27:32.558 INFO:teuthology.orchestra.run.smithi063.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.29326.log.gz 2024-04-26T04:27:32.558 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28830.log 2024-04-26T04:27:32.558 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.29064.log: /var/log/ceph/ceph-client.admin.31574.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.30374.log.gz 2024-04-26T04:27:32.558 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29667.log 2024-04-26T04:27:32.558 INFO:teuthology.orchestra.run.smithi063.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.29064.log.gz 2024-04-26T04:27:32.559 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.28830.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-04-26T04:27:32.559 INFO:teuthology.orchestra.run.smithi063.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.31574.log.gz 2024-04-26T04:27:32.559 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.29667.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30322.log 2024-04-26T04:27:32.559 INFO:teuthology.orchestra.run.smithi063.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.28830.log.gz 2024-04-26T04:27:32.559 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30402.log 2024-04-26T04:27:32.559 INFO:teuthology.orchestra.run.smithi063.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.29667.log.gz 2024-04-26T04:27:32.559 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.30322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29352.log 2024-04-26T04:27:32.559 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.30402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28858.log 2024-04-26T04:27:32.559 INFO:teuthology.orchestra.run.smithi063.stderr: 89.0%/var/log/ceph/ceph-client.admin.29352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29038.log 2024-04-26T04:27:32.559 INFO:teuthology.orchestra.run.smithi063.stderr: -- replaced with /var/log/ceph/ceph-client.admin.30322.log.gz 2024-04-26T04:27:32.559 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.28858.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.30402.log.gz 2024-04-26T04:27:32.559 INFO:teuthology.orchestra.run.smithi063.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.29352.log.gz 2024-04-26T04:27:32.559 INFO:teuthology.orchestra.run.smithi063.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.28858.log.gz 2024-04-26T04:27:32.559 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28876.log 2024-04-26T04:27:32.560 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.29038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29248.log 2024-04-26T04:27:32.560 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5/var/log/ceph/ceph-client.admin.28876.log: --verbose -- /var/log/ceph/ceph-client.admin.31340.log 2024-04-26T04:27:32.560 INFO:teuthology.orchestra.run.smithi063.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.29038.log.gz 2024-04-26T04:27:32.560 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.29248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30216.log 2024-04-26T04:27:32.560 INFO:teuthology.orchestra.run.smithi063.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.28876.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30720.log 2024-04-26T04:27:32.560 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:32.560 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.31340.log: /var/log/ceph/ceph-client.admin.30216.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.29248.log.gz 2024-04-26T04:27:32.560 INFO:teuthology.orchestra.run.smithi063.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.31340.log.gz 2024-04-26T04:27:32.560 INFO:teuthology.orchestra.run.smithi063.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.30216.log.gz 2024-04-26T04:27:32.560 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30510.log 2024-04-26T04:27:32.560 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.30720.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30164.log 2024-04-26T04:27:32.560 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.30510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28685.log 2024-04-26T04:27:32.560 INFO:teuthology.orchestra.run.smithi063.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.30720.log.gz 2024-04-26T04:27:32.561 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.30164.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-26T04:27:32.561 INFO:teuthology.orchestra.run.smithi063.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.30510.log.gz 2024-04-26T04:27:32.561 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30616.log 2024-04-26T04:27:32.561 INFO:teuthology.orchestra.run.smithi063.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.30164.log.gz 2024-04-26T04:27:32.561 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.admin.28685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31106.log 2024-04-26T04:27:32.561 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.30616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29430.log 2024-04-26T04:27:32.561 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.31106.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.28685.log.gz 2024-04-26T04:27:32.561 INFO:teuthology.orchestra.run.smithi063.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.30616.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31444.log 2024-04-26T04:27:32.561 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:32.561 INFO:teuthology.orchestra.run.smithi063.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.31106.log.gz 2024-04-26T04:27:32.561 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.29430.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.29430.log.gz 2024-04-26T04:27:32.561 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28986.log 2024-04-26T04:27:32.561 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.31444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29300.log 2024-04-26T04:27:32.562 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.28986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28538.log 2024-04-26T04:27:32.562 INFO:teuthology.orchestra.run.smithi063.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.31444.log.gz 2024-04-26T04:27:32.562 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.29300.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.28986.log.gz 2024-04-26T04:27:32.562 INFO:teuthology.orchestra.run.smithi063.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.29300.log.gz 2024-04-26T04:27:32.562 INFO:teuthology.orchestra.run.smithi063.stderr: 91.5% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-26T04:27:32.562 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31470.log 2024-04-26T04:27:32.562 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.28538.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31496.log 2024-04-26T04:27:32.562 INFO:teuthology.orchestra.run.smithi063.stderr:gzip/var/log/ceph/ceph-client.admin.31470.log: -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-04-26T04:27:32.562 INFO:teuthology.orchestra.run.smithi063.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.28538.log.gz 2024-04-26T04:27:32.562 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.31496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31184.log 2024-04-26T04:27:32.562 INFO:teuthology.orchestra.run.smithi063.stderr: 88.9%/var/log/ceph/ceph-mon.c.log: -- replaced with /var/log/ceph/ceph-client.admin.31470.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28460.log 2024-04-26T04:27:32.562 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:32.562 INFO:teuthology.orchestra.run.smithi063.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.31496.log.gz 2024-04-26T04:27:32.562 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.31184.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.31184.log.gz 2024-04-26T04:27:32.563 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29194.log 2024-04-26T04:27:32.563 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.28460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31626.log 2024-04-26T04:27:32.563 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.29194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30190.log 2024-04-26T04:27:32.563 INFO:teuthology.orchestra.run.smithi063.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.28460.log.gz 2024-04-26T04:27:32.563 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.31626.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.29194.log.gz 2024-04-26T04:27:32.563 INFO:teuthology.orchestra.run.smithi063.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.31626.log.gz 2024-04-26T04:27:32.563 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23879.log 2024-04-26T04:27:32.563 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.30190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31392.log 2024-04-26T04:27:32.563 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.23879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30852.log 2024-04-26T04:27:32.563 INFO:teuthology.orchestra.run.smithi063.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.30190.log.gz 2024-04-26T04:27:32.563 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.31392.log: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.23879.log.gz 2024-04-26T04:27:32.563 INFO:teuthology.orchestra.run.smithi063.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.31392.log.gz 2024-04-26T04:27:32.563 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29090.log 2024-04-26T04:27:32.564 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.30852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29745.log 2024-04-26T04:27:32.564 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.29090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29823.log 2024-04-26T04:27:32.564 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.29745.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.30852.log.gz 2024-04-26T04:27:32.564 INFO:teuthology.orchestra.run.smithi063.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.29090.log.gz 2024-04-26T04:27:32.564 INFO:teuthology.orchestra.run.smithi063.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.29745.log.gz 2024-04-26T04:27:32.564 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30798.log 2024-04-26T04:27:32.564 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.29823.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30138.log 2024-04-26T04:27:32.564 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.30798.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-26T04:27:32.564 INFO:teuthology.orchestra.run.smithi063.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.29823.log.gz 2024-04-26T04:27:32.564 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.30138.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.30798.log.gz 2024-04-26T04:27:32.564 INFO:teuthology.orchestra.run.smithi063.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.30138.log.gz 2024-04-26T04:27:32.564 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29519.log 2024-04-26T04:27:32.564 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31702.log 2024-04-26T04:27:32.564 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.29519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30904.log 2024-04-26T04:27:32.565 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.31702.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.29519.log.gz 2024-04-26T04:27:32.565 INFO:teuthology.orchestra.run.smithi063.stderr: 93.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-26T04:27:32.565 INFO:teuthology.orchestra.run.smithi063.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.31702.log.gz 2024-04-26T04:27:32.565 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28689.log 2024-04-26T04:27:32.565 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29875.log 2024-04-26T04:27:32.565 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.30904.log: /var/log/ceph/ceph-client.admin.28689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29849.log 2024-04-26T04:27:32.565 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.29875.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.28689.log.gz 89.6% -- replaced with /var/log/ceph/ceph-client.admin.30904.log.gz 2024-04-26T04:27:32.565 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:32.565 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31314.log 2024-04-26T04:27:32.565 INFO:teuthology.orchestra.run.smithi063.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.29875.log.gz 2024-04-26T04:27:32.565 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.29849.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.29849.log.gz 2024-04-26T04:27:32.565 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30268.log 2024-04-26T04:27:32.565 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29929.log 2024-04-26T04:27:32.566 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.31314.log: /var/log/ceph/ceph-client.admin.30268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30746.log 2024-04-26T04:27:32.566 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.29929.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.31314.log.gz 2024-04-26T04:27:32.566 INFO:teuthology.orchestra.run.smithi063.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.30268.log.gz 2024-04-26T04:27:32.566 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30878.log 2024-04-26T04:27:32.566 INFO:teuthology.orchestra.run.smithi063.stderr: 90.5%/var/log/ceph/ceph-client.admin.30746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30668.log 2024-04-26T04:27:32.566 INFO:teuthology.orchestra.run.smithi063.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29929.log.gz 2024-04-26T04:27:32.566 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.30878.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.30746.log.gz 2024-04-26T04:27:32.566 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30428.log 2024-04-26T04:27:32.566 INFO:teuthology.orchestra.run.smithi063.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.30878.log.gz 2024-04-26T04:27:32.566 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.30668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30536.log 2024-04-26T04:27:32.566 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.30428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28536.log 2024-04-26T04:27:32.566 INFO:teuthology.orchestra.run.smithi063.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.30668.log.gz 2024-04-26T04:27:32.566 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.30536.log: gzip -5 --verbose 89.7% -- /var/log/ceph/ceph-client.admin.30457.log -- replaced with /var/log/ceph/ceph-client.admin.30428.log.gz 2024-04-26T04:27:32.567 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:32.567 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.28536.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.30536.log.gz 2024-04-26T04:27:32.567 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.20258.log 2024-04-26T04:27:32.567 INFO:teuthology.orchestra.run.smithi063.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.28536.log.gz 2024-04-26T04:27:32.567 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29610.log 2024-04-26T04:27:32.567 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.30457.log: /var/log/ceph/ceph.tmp-client.admin.20258.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.20258.log.gz 2024-04-26T04:27:32.567 INFO:teuthology.orchestra.run.smithi063.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.30457.log.gz 2024-04-26T04:27:32.567 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23996.log 2024-04-26T04:27:32.567 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30642.log 2024-04-26T04:27:32.567 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.29610.log: gzip -5 --verbose/var/log/ceph/ceph-client.admin.23996.log: -- /var/log/ceph/ceph-client.admin.31210.log 2024-04-26T04:27:32.567 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.30642.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.23996.log.gz 2024-04-26T04:27:32.567 INFO:teuthology.orchestra.run.smithi063.stderr: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.29610.log.gz 2024-04-26T04:27:32.567 INFO:teuthology.orchestra.run.smithi063.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.30642.log.gz 2024-04-26T04:27:32.567 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30295.log 2024-04-26T04:27:32.568 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29797.log 2024-04-26T04:27:32.568 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.31210.log: /var/log/ceph/ceph-client.admin.30295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28529.log 2024-04-26T04:27:32.568 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.29797.log: 88.4% 89.2% -- replaced with /var/log/ceph/ceph-client.admin.31210.log.gz 2024-04-26T04:27:32.568 INFO:teuthology.orchestra.run.smithi063.stderr: -- replaced with /var/log/ceph/ceph-client.admin.30295.log.gz 2024-04-26T04:27:32.568 INFO:teuthology.orchestra.run.smithi063.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.29797.log.gz 2024-04-26T04:27:32.568 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30483.log 2024-04-26T04:27:32.568 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29584.log 2024-04-26T04:27:32.568 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.28529.log: /var/log/ceph/ceph-client.admin.30483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31262.log 2024-04-26T04:27:32.568 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.29584.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.28529.log.gz 2024-04-26T04:27:32.568 INFO:teuthology.orchestra.run.smithi063.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.30483.log.gz 2024-04-26T04:27:32.568 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28865.log 2024-04-26T04:27:32.568 INFO:teuthology.orchestra.run.smithi063.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.29584.log.gz 2024-04-26T04:27:32.568 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31548.log 2024-04-26T04:27:32.568 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.31262.log: /var/log/ceph/ceph-client.admin.28865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29012.log 2024-04-26T04:27:32.569 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.31548.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.28865.log.gz 2024-04-26T04:27:32.569 INFO:teuthology.orchestra.run.smithi063.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.31262.log.gz 2024-04-26T04:27:32.569 INFO:teuthology.orchestra.run.smithi063.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.31548.log.gz 2024-04-26T04:27:32.569 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31053.log 2024-04-26T04:27:32.569 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29903.log 2024-04-26T04:27:32.569 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.29012.log: /var/log/ceph/ceph-client.admin.31053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29222.log 2024-04-26T04:27:32.569 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.29903.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.29012.log.gz 2024-04-26T04:27:32.569 INFO:teuthology.orchestra.run.smithi063.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.31053.log.gz 2024-04-26T04:27:32.569 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31158.log 2024-04-26T04:27:32.569 INFO:teuthology.orchestra.run.smithi063.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.29903.log.gz 2024-04-26T04:27:32.569 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.29222.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.29222.log.gz 2024-04-26T04:27:32.569 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.31158.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.31158.log.gz 2024-04-26T04:27:32.576 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.28501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31312.log 2024-04-26T04:27:32.576 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.28795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29232.log 2024-04-26T04:27:32.576 INFO:teuthology.orchestra.run.smithi186.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.28501.log.gz 2024-04-26T04:27:32.576 INFO:teuthology.orchestra.run.smithi186.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.31403.log.gz 2024-04-26T04:27:32.576 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.31312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30187.log 2024-04-26T04:27:32.576 INFO:teuthology.orchestra.run.smithi186.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.28795.log.gz 2024-04-26T04:27:32.576 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.29232.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24017.log 2024-04-26T04:27:32.576 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.30187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28708.log 2024-04-26T04:27:32.577 INFO:teuthology.orchestra.run.smithi186.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.31312.log.gz 2024-04-26T04:27:32.577 INFO:teuthology.orchestra.run.smithi186.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.29232.log.gz 2024-04-26T04:27:32.577 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.24017.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30415.log 2024-04-26T04:27:32.577 INFO:teuthology.orchestra.run.smithi186.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.24017.log.gz 2024-04-26T04:27:32.577 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29927.log 2024-04-26T04:27:32.577 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.28708.log: /var/log/ceph/ceph-client.admin.30415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29023.log 2024-04-26T04:27:32.577 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.29927.log: gzip -5 --verbose -- 89.2% /var/log/ceph/ceph-client.admin.29666.log 2024-04-26T04:27:32.577 INFO:teuthology.orchestra.run.smithi186.stderr: -- replaced with /var/log/ceph/ceph-client.admin.30187.log.gz 2024-04-26T04:27:32.577 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.29023.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.30415.log.gz 2024-04-26T04:27:32.577 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31000.log 2024-04-26T04:27:32.577 INFO:teuthology.orchestra.run.smithi186.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.29927.log.gz 2024-04-26T04:27:32.577 INFO:teuthology.orchestra.run.smithi186.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.28708.log.gz 2024-04-26T04:27:32.578 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.29666.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.29023.log.gz 2024-04-26T04:27:32.578 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27718.log 2024-04-26T04:27:32.578 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.31000.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.29666.log.gz 2024-04-26T04:27:32.578 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27978.log 2024-04-26T04:27:32.578 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.27718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31820.log 2024-04-26T04:27:32.578 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.27978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32039.log 2024-04-26T04:27:32.578 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.31820.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.27718.log.gz 2024-04-26T04:27:32.578 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29057.log 2024-04-26T04:27:32.578 INFO:teuthology.orchestra.run.smithi186.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.31000.log.gz 2024-04-26T04:27:32.578 INFO:teuthology.orchestra.run.smithi186.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.27978.log.gz 2024-04-26T04:27:32.578 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.32039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28761.log 2024-04-26T04:27:32.579 INFO:teuthology.orchestra.run.smithi186.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.31820.log.gz 2024-04-26T04:27:32.579 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.29057.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27752.log 2024-04-26T04:27:32.579 INFO:teuthology.orchestra.run.smithi186.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.32039.log.gz 2024-04-26T04:27:32.579 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.28761.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29406.log 2024-04-26T04:27:32.579 INFO:teuthology.orchestra.run.smithi186.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.29057.log.gz 2024-04-26T04:27:32.579 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.27752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31001.log 2024-04-26T04:27:32.579 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.29406.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.28761.log.gz 2024-04-26T04:27:32.579 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30764.log 2024-04-26T04:27:32.579 INFO:teuthology.orchestra.run.smithi186.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.27752.log.gz 2024-04-26T04:27:32.579 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.31001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31547.log 2024-04-26T04:27:32.579 INFO:teuthology.orchestra.run.smithi186.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.29406.log.gz 2024-04-26T04:27:32.579 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.30764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29110.log 2024-04-26T04:27:32.579 INFO:teuthology.orchestra.run.smithi186.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.31001.log.gz 2024-04-26T04:27:32.579 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.31547.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.30764.log.gz 2024-04-26T04:27:32.580 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30669.log 2024-04-26T04:27:32.580 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.29110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28848.log 2024-04-26T04:27:32.580 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.30669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29719.log 2024-04-26T04:27:32.580 INFO:teuthology.orchestra.run.smithi186.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.29110.log.gz 2024-04-26T04:27:32.580 INFO:teuthology.orchestra.run.smithi186.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.31547.log.gz 2024-04-26T04:27:32.580 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.28848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29893.log 2024-04-26T04:27:32.580 INFO:teuthology.orchestra.run.smithi186.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.30669.log.gz 2024-04-26T04:27:32.580 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.29719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31857.log 2024-04-26T04:27:32.580 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.29893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28448.log 2024-04-26T04:27:32.580 INFO:teuthology.orchestra.run.smithi186.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.29719.log.gz 2024-04-26T04:27:32.580 INFO:teuthology.orchestra.run.smithi186.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.28848.log.gz 2024-04-26T04:27:32.580 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.31857.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.29893.log.gz 2024-04-26T04:27:32.580 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31221.log 2024-04-26T04:27:32.581 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.28448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31674.log 2024-04-26T04:27:32.581 INFO:teuthology.orchestra.run.smithi186.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.31857.log.gz 2024-04-26T04:27:32.581 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.31221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28500.log 2024-04-26T04:27:32.581 INFO:teuthology.orchestra.run.smithi186.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.28448.log.gz 2024-04-26T04:27:32.581 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.31674.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29926.log 2024-04-26T04:27:32.581 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.28500.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.31221.log.gz 2024-04-26T04:27:32.581 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28014.log 2024-04-26T04:27:32.581 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.29926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29022.log 2024-04-26T04:27:32.581 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.28014.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.28500.log.gz 2024-04-26T04:27:32.581 INFO:teuthology.orchestra.run.smithi186.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.31674.log.gz 2024-04-26T04:27:32.581 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31494.log 2024-04-26T04:27:32.581 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23905.log 2024-04-26T04:27:32.581 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.29022.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.28014.log.gz 2024-04-26T04:27:32.582 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.31494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29109.log 2024-04-26T04:27:32.582 INFO:teuthology.orchestra.run.smithi186.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.29926.log.gz 2024-04-26T04:27:32.582 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.23905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29544.log 2024-04-26T04:27:32.582 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.29109.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.31494.log.gz 2024-04-26T04:27:32.582 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31274.log 2024-04-26T04:27:32.582 INFO:teuthology.orchestra.run.smithi186.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.29022.log.gz 2024-04-26T04:27:32.582 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.29544.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.23905.log.gz 2024-04-26T04:27:32.582 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31676.log 2024-04-26T04:27:32.582 INFO:teuthology.orchestra.run.smithi186.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.29109.log.gz 2024-04-26T04:27:32.582 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.31274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29317.log 2024-04-26T04:27:32.582 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.31676.log: /var/log/ceph/ceph-client.admin.30363.log 2024-04-26T04:27:32.582 INFO:teuthology.orchestra.run.smithi186.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.29544.log.gz 2024-04-26T04:27:32.582 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.29317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29805.log 2024-04-26T04:27:32.582 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.30363.log: gzip -5 89.5% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.31274.log.gz 2024-04-26T04:27:32.583 INFO:teuthology.orchestra.run.smithi186.stderr: /var/log/ceph/ceph-client.admin.29631.log 2024-04-26T04:27:32.583 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.29805.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.31676.log.gz 2024-04-26T04:27:32.583 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31456.log 2024-04-26T04:27:32.583 INFO:teuthology.orchestra.run.smithi186.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.29317.log.gz 2024-04-26T04:27:32.583 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.29631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30066.log 2024-04-26T04:27:32.583 INFO:teuthology.orchestra.run.smithi186.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.30363.log.gz 2024-04-26T04:27:32.583 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5/var/log/ceph/ceph-client.admin.31456.log: --verbose -- /var/log/ceph/ceph-client.admin.30274.log 2024-04-26T04:27:32.583 INFO:teuthology.orchestra.run.smithi186.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.29805.log.gz 2024-04-26T04:27:32.583 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.30066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28623.log 2024-04-26T04:27:32.583 INFO:teuthology.orchestra.run.smithi186.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.29631.log.gz 2024-04-26T04:27:32.583 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.30274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27892.log 2024-04-26T04:27:32.583 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.28623.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.31456.log.gz 2024-04-26T04:27:32.583 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28274.log 2024-04-26T04:27:32.583 INFO:teuthology.orchestra.run.smithi186.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.30066.log.gz 2024-04-26T04:27:32.584 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.27892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28100.log 2024-04-26T04:27:32.584 INFO:teuthology.orchestra.run.smithi186.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.30274.log.gz 2024-04-26T04:27:32.584 INFO:teuthology.orchestra.run.smithi186.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.28623.log.gz 2024-04-26T04:27:32.584 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.28274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27979.log 2024-04-26T04:27:32.584 INFO:teuthology.orchestra.run.smithi186.stderr: 89.4%/var/log/ceph/ceph-client.admin.28100.log: -- replaced with /var/log/ceph/ceph-client.admin.27892.log.gz 2024-04-26T04:27:32.584 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32038.log 2024-04-26T04:27:32.584 INFO:teuthology.orchestra.run.smithi186.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.28274.log.gz 2024-04-26T04:27:32.584 INFO:teuthology.orchestra.run.smithi186.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.28100.log.gz 2024-04-26T04:27:32.584 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.27979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30856.log 2024-04-26T04:27:32.584 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.32038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31366.log 2024-04-26T04:27:32.584 INFO:teuthology.orchestra.run.smithi186.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.27979.log.gz 2024-04-26T04:27:32.584 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.30856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31275.log 2024-04-26T04:27:32.584 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.31366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27753.log 2024-04-26T04:27:32.584 INFO:teuthology.orchestra.run.smithi186.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.30856.log.gz 2024-04-26T04:27:32.584 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.31275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31492.log 2024-04-26T04:27:32.585 INFO:teuthology.orchestra.run.smithi186.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.32038.log.gz 2024-04-26T04:27:32.585 INFO:teuthology.orchestra.run.smithi186.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.31366.log.gz 2024-04-26T04:27:32.585 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.27753.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.31275.log.gz 2024-04-26T04:27:32.585 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30327.log 2024-04-26T04:27:32.585 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.31492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28796.log 2024-04-26T04:27:32.585 INFO:teuthology.orchestra.run.smithi186.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.27753.log.gz 2024-04-26T04:27:32.585 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.30327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30414.log 2024-04-26T04:27:32.585 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.28796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27838.log 2024-04-26T04:27:32.585 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.30414.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.31492.log.gz 2024-04-26T04:27:32.585 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30501.log 2024-04-26T04:27:32.585 INFO:teuthology.orchestra.run.smithi186.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.28796.log.gz 2024-04-26T04:27:32.585 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.27838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31947.log 2024-04-26T04:27:32.586 INFO:teuthology.orchestra.run.smithi186.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.30327.log.gz 2024-04-26T04:27:32.586 INFO:teuthology.orchestra.run.smithi186.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.30414.log.gz 2024-04-26T04:27:32.586 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.30501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31310.log 2024-04-26T04:27:32.586 INFO:teuthology.orchestra.run.smithi186.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.27838.log.gz 2024-04-26T04:27:32.586 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30153.log 2024-04-26T04:27:32.586 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.31947.log: /var/log/ceph/ceph-client.admin.31310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28447.log 2024-04-26T04:27:32.586 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.30153.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.30501.log.gz 2024-04-26T04:27:32.586 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28361.log 2024-04-26T04:27:32.586 INFO:teuthology.orchestra.run.smithi186.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.31310.log.gz 2024-04-26T04:27:32.586 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.28447.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30189.log 2024-04-26T04:27:32.586 INFO:teuthology.orchestra.run.smithi186.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.31947.log.gz 2024-04-26T04:27:32.586 INFO:teuthology.orchestra.run.smithi186.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.30153.log.gz 2024-04-26T04:27:32.586 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.28361.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29284.log 2024-04-26T04:27:32.587 INFO:teuthology.orchestra.run.smithi186.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.28361.log.gz 2024-04-26T04:27:32.587 INFO:teuthology.orchestra.run.smithi186.stderr:gzip/var/log/ceph/ceph-client.admin.30189.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.29283.log 2024-04-26T04:27:32.587 INFO:teuthology.orchestra.run.smithi186.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.28447.log.gz 2024-04-26T04:27:32.587 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.29284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29578.log 2024-04-26T04:27:32.587 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.29283.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.29284.log.gz 2024-04-26T04:27:32.587 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28762.log 2024-04-26T04:27:32.587 INFO:teuthology.orchestra.run.smithi186.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.30189.log.gz 2024-04-26T04:27:32.587 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.29578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30015.log 2024-04-26T04:27:32.587 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.28762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31765.log 2024-04-26T04:27:32.587 INFO:teuthology.orchestra.run.smithi186.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.29283.log.gz 2024-04-26T04:27:32.587 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.30015.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.28762.log.gz 2024-04-26T04:27:32.587 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28884.log 2024-04-26T04:27:32.588 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.31765.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.29578.log.gz 2024-04-26T04:27:32.588 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29197.log 2024-04-26T04:27:32.588 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.28884.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-04-26T04:27:32.588 INFO:teuthology.orchestra.run.smithi186.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.30015.log.gz 2024-04-26T04:27:32.588 INFO:teuthology.orchestra.run.smithi186.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.31765.log.gz 2024-04-26T04:27:32.588 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.29197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28587.log 2024-04-26T04:27:32.588 INFO:teuthology.orchestra.run.smithi186.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.28884.log.gz 2024-04-26T04:27:32.588 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-04-26T04:27:32.588 INFO:teuthology.orchestra.run.smithi186.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.29197.log.gz 2024-04-26T04:27:32.588 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.28587.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30241.log 2024-04-26T04:27:32.588 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28273.log 2024-04-26T04:27:32.588 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.30241.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.28587.log.gz 2024-04-26T04:27:32.588 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29404.log 2024-04-26T04:27:32.589 INFO:teuthology.orchestra.run.smithi186.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.30241.log.gz 2024-04-26T04:27:32.589 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29144.log 2024-04-26T04:27:32.589 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.28273.log: /var/log/ceph/ceph-client.admin.29404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27751.log 2024-04-26T04:27:32.589 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.29144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29196.log 2024-04-26T04:27:32.589 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.27751.log: gzip -5 88.2% -- replaced with /var/log/ceph/ceph-client.admin.29144.log.gz 2024-04-26T04:27:32.589 INFO:teuthology.orchestra.run.smithi186.stderr: --verbose -- 89.5% -- replaced with /var/log/ceph/ceph-client.admin.29404.log.gz 2024-04-26T04:27:32.589 INFO:teuthology.orchestra.run.smithi186.stderr: /var/log/ceph/ceph-client.admin.30449.log 2024-04-26T04:27:32.589 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.29196.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.28273.log.gz 2024-04-26T04:27:32.589 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30240.log 2024-04-26T04:27:32.589 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.30449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28621.log 2024-04-26T04:27:32.589 INFO:teuthology.orchestra.run.smithi186.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.27751.log.gz 2024-04-26T04:27:32.589 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.30240.log: gzip -5 --verbose 89.1% -- -- replaced with /var/log/ceph/ceph-client.admin.30449.log.gz /var/log/ceph/ceph-client.admin.30101.log 2024-04-26T04:27:32.590 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-04-26T04:27:32.590 INFO:teuthology.orchestra.run.smithi186.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.29196.log.gz 2024-04-26T04:27:32.590 INFO:teuthology.orchestra.run.smithi186.stderr:gzip/var/log/ceph/ceph-client.admin.28621.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.30154.log 2024-04-26T04:27:32.590 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.30101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29371.log 2024-04-26T04:27:32.590 INFO:teuthology.orchestra.run.smithi186.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.30240.log.gz 2024-04-26T04:27:32.590 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.30154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30765.log 2024-04-26T04:27:32.590 INFO:teuthology.orchestra.run.smithi186.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.30101.log.gz 2024-04-26T04:27:32.590 INFO:teuthology.orchestra.run.smithi186.stderr: 89.3%/var/log/ceph/ceph-client.admin.29371.log: -- replaced with /var/log/ceph/ceph-client.admin.28621.log.gz 2024-04-26T04:27:32.590 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31220.log 2024-04-26T04:27:32.590 INFO:teuthology.orchestra.run.smithi186.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.30154.log.gz 2024-04-26T04:27:32.590 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.30765.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29667.log 2024-04-26T04:27:32.590 INFO:teuthology.orchestra.run.smithi186.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.29371.log.gz 2024-04-26T04:27:32.590 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.31220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31856.log 2024-04-26T04:27:32.591 INFO:teuthology.orchestra.run.smithi186.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.31220.log.gz 2024-04-26T04:27:32.591 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.29667.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30328.log 2024-04-26T04:27:32.591 INFO:teuthology.orchestra.run.smithi186.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.30765.log.gz 2024-04-26T04:27:32.591 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.31856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28012.log 2024-04-26T04:27:32.591 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.30328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27804.log 2024-04-26T04:27:32.591 INFO:teuthology.orchestra.run.smithi186.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.29667.log.gz 2024-04-26T04:27:32.591 INFO:teuthology.orchestra.run.smithi186.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.30328.log.gz 2024-04-26T04:27:32.591 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.28012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31911.log 2024-04-26T04:27:32.591 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.27804.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.31856.log.gz 2024-04-26T04:27:32.591 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29058.log 2024-04-26T04:27:32.591 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.31911.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23969.log 2024-04-26T04:27:32.592 INFO:teuthology.orchestra.run.smithi186.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.28012.log.gz 2024-04-26T04:27:32.592 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.29058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28675.log 2024-04-26T04:27:32.592 INFO:teuthology.orchestra.run.smithi186.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.27804.log.gz 2024-04-26T04:27:32.592 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.23969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28622.log 2024-04-26T04:27:32.592 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.28675.log: 89.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30670.log 2024-04-26T04:27:32.592 INFO:teuthology.orchestra.run.smithi186.stderr: -- replaced with /var/log/ceph/ceph-client.admin.31911.log.gz 90.4% 88.4% -- replaced with /var/log/ceph/ceph-client.admin.29058.log.gz -- replaced with /var/log/ceph/ceph-client.admin.23969.log.gz 2024-04-26T04:27:32.592 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-04-26T04:27:32.592 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-04-26T04:27:32.592 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.28622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28327.log 2024-04-26T04:27:32.592 INFO:teuthology.orchestra.run.smithi186.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.28675.log.gz 2024-04-26T04:27:32.592 INFO:teuthology.orchestra.run.smithi186.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.28622.log.gz 2024-04-26T04:27:32.592 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.30670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30854.log 2024-04-26T04:27:32.592 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.28327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29457.log 2024-04-26T04:27:32.592 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.30854.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.30670.log.gz 2024-04-26T04:27:32.593 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28153.log 2024-04-26T04:27:32.593 INFO:teuthology.orchestra.run.smithi186.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.28327.log.gz 2024-04-26T04:27:32.593 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.29457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28013.log 2024-04-26T04:27:32.593 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.28153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27927.log 2024-04-26T04:27:32.593 INFO:teuthology.orchestra.run.smithi186.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.30854.log.gz 2024-04-26T04:27:32.593 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.28013.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.28153.log.gz 2024-04-26T04:27:32.593 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31729.log 2024-04-26T04:27:32.593 INFO:teuthology.orchestra.run.smithi186.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.28013.log.gz 2024-04-26T04:27:32.593 INFO:teuthology.orchestra.run.smithi186.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.29457.log.gz 2024-04-26T04:27:32.593 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28883.log 2024-04-26T04:27:32.593 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.27927.log: /var/log/ceph/ceph-client.admin.31729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32002.log 2024-04-26T04:27:32.593 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.28883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29493.log 2024-04-26T04:27:32.593 INFO:teuthology.orchestra.run.smithi186.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.27927.log.gz 2024-04-26T04:27:32.593 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.32002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30636.log 2024-04-26T04:27:32.594 INFO:teuthology.orchestra.run.smithi186.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.31729.log.gz 88.0% 2024-04-26T04:27:32.594 INFO:teuthology.orchestra.run.smithi186.stderr: -- replaced with /var/log/ceph/ceph-client.admin.28883.log.gz 2024-04-26T04:27:32.594 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.29493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31219.log 2024-04-26T04:27:32.594 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.30636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28187.log 2024-04-26T04:27:32.594 INFO:teuthology.orchestra.run.smithi186.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.32002.log.gz 2024-04-26T04:27:32.594 INFO:teuthology.orchestra.run.smithi186.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.29493.log.gz 2024-04-26T04:27:32.594 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.31219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28710.log 2024-04-26T04:27:32.594 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.28187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31912.log 2024-04-26T04:27:32.594 INFO:teuthology.orchestra.run.smithi186.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.30636.log.gz 2024-04-26T04:27:32.594 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.28710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31036.log 2024-04-26T04:27:32.594 INFO:teuthology.orchestra.run.smithi186.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.28187.log.gz 2024-04-26T04:27:32.594 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.31912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30102.log 2024-04-26T04:27:32.594 INFO:teuthology.orchestra.run.smithi186.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.31219.log.gz 2024-04-26T04:27:32.595 INFO:teuthology.orchestra.run.smithi186.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.28710.log.gz 2024-04-26T04:27:32.595 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.31036.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-04-26T04:27:32.595 INFO:teuthology.orchestra.run.smithi186.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.31912.log.gz 2024-04-26T04:27:32.595 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31038.log 2024-04-26T04:27:32.595 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.30102.log: /var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28674.log 2024-04-26T04:27:32.595 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.31038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30819.log 2024-04-26T04:27:32.595 INFO:teuthology.orchestra.run.smithi186.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.31036.log.gz 2024-04-26T04:27:32.595 INFO:teuthology.orchestra.run.smithi186.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.30102.log.gz 2024-04-26T04:27:32.595 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.28674.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30909.log 2024-04-26T04:27:32.595 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.30819.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-26T04:27:32.595 INFO:teuthology.orchestra.run.smithi186.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.31038.log.gz 2024-04-26T04:27:32.595 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.30909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28240.log 2024-04-26T04:27:32.595 INFO:teuthology.orchestra.run.smithi186.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.30819.log.gz 2024-04-26T04:27:32.595 INFO:teuthology.orchestra.run.smithi186.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.28674.log.gz 2024-04-26T04:27:32.595 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29928.log 2024-04-26T04:27:32.596 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.28240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29580.log 2024-04-26T04:27:32.596 INFO:teuthology.orchestra.run.smithi186.stderr: 89.3%/var/log/ceph/ceph-client.admin.29928.log: -- replaced with /var/log/ceph/ceph-client.admin.30909.log.gz 2024-04-26T04:27:32.596 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29545.log 2024-04-26T04:27:32.596 INFO:teuthology.orchestra.run.smithi186.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.28240.log.gz 2024-04-26T04:27:32.596 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23993.log 2024-04-26T04:27:32.596 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.29580.log: /var/log/ceph/ceph-client.admin.29545.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28709.log 2024-04-26T04:27:32.596 INFO:teuthology.orchestra.run.smithi186.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.29928.log.gz 2024-04-26T04:27:32.596 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.23993.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.29545.log.gz 2024-04-26T04:27:32.596 INFO:teuthology.orchestra.run.smithi186.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.29580.log.gz 2024-04-26T04:27:32.596 INFO:teuthology.orchestra.run.smithi186.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.23993.log.gz 2024-04-26T04:27:32.596 INFO:teuthology.orchestra.run.smithi186.stderr: 87.1% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-26T04:27:32.596 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31130.log 2024-04-26T04:27:32.596 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27840.log 2024-04-26T04:27:32.596 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.28709.log: /var/log/ceph/ceph-client.admin.31130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31858.log 2024-04-26T04:27:32.597 INFO:teuthology.orchestra.run.smithi186.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.28709.log.gz 2024-04-26T04:27:32.597 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.27840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31585.log 2024-04-26T04:27:32.597 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.31858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28152.log 2024-04-26T04:27:32.597 INFO:teuthology.orchestra.run.smithi186.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.31130.log.gz 2024-04-26T04:27:32.597 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.31585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27891.log 2024-04-26T04:27:32.597 INFO:teuthology.orchestra.run.smithi186.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.27840.log.gz 2024-04-26T04:27:32.597 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.28152.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.31858.log.gz 2024-04-26T04:27:32.597 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31821.log 2024-04-26T04:27:32.597 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.27891.log: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.31585.log.gz 2024-04-26T04:27:32.597 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29892.log 2024-04-26T04:27:32.597 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30361.log 2024-04-26T04:27:32.597 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.31821.log: /var/log/ceph/ceph-client.admin.29892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31949.log 2024-04-26T04:27:32.597 INFO:teuthology.orchestra.run.smithi186.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.28152.log.gz 2024-04-26T04:27:32.598 INFO:teuthology.orchestra.run.smithi186.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.27891.log.gz 88.9% -- replaced with /var/log/ceph/ceph-client.admin.31821.log.gz 2024-04-26T04:27:32.598 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-04-26T04:27:32.598 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.30361.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.29892.log.gz 2024-04-26T04:27:32.598 INFO:teuthology.orchestra.run.smithi186.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.30361.log.gz 2024-04-26T04:27:32.598 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31128.log 2024-04-26T04:27:32.598 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.31949.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27926.log 2024-04-26T04:27:32.598 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31675.log 2024-04-26T04:27:32.598 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.31128.log: /var/log/ceph/ceph-client.admin.27926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30537.log 2024-04-26T04:27:32.598 INFO:teuthology.orchestra.run.smithi186.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.31949.log.gz 2024-04-26T04:27:32.598 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.31675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29143.log 2024-04-26T04:27:32.598 INFO:teuthology.orchestra.run.smithi186.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.27926.log.gz 2024-04-26T04:27:32.598 INFO:teuthology.orchestra.run.smithi186.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.31128.log.gz 2024-04-26T04:27:32.598 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.30537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30818.log 2024-04-26T04:27:32.599 INFO:teuthology.orchestra.run.smithi186.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.31675.log.gz 2024-04-26T04:27:32.599 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.29143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28275.log 2024-04-26T04:27:32.599 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.30818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30275.log 2024-04-26T04:27:32.599 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.28275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24041.log 2024-04-26T04:27:32.599 INFO:teuthology.orchestra.run.smithi186.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.29143.log.gz 2024-04-26T04:27:32.599 INFO:teuthology.orchestra.run.smithi186.stderr: 92.1% -- replaced with /var/log/ceph/ceph-client.admin.30537.log.gz 2024-04-26T04:27:32.599 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.30275.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.30818.log.gz 2024-04-26T04:27:32.599 INFO:teuthology.orchestra.run.smithi186.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.28275.log.gz 2024-04-26T04:27:32.599 INFO:teuthology.orchestra.run.smithi186.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.30275.log.gz 2024-04-26T04:27:32.599 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28970.log 2024-04-26T04:27:32.599 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.24041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31184.log 2024-04-26T04:27:32.600 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.28970.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.24041.log.gz 2024-04-26T04:27:32.600 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30276.log 2024-04-26T04:27:32.600 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32003.log 2024-04-26T04:27:32.600 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.31184.log: /var/log/ceph/ceph-client.admin.30276.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.28970.log.gz 2024-04-26T04:27:32.600 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28449.log 2024-04-26T04:27:32.600 INFO:teuthology.orchestra.run.smithi186.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.31184.log.gz 2024-04-26T04:27:32.600 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.32003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30448.log 2024-04-26T04:27:32.600 INFO:teuthology.orchestra.run.smithi186.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.30276.log.gz 2024-04-26T04:27:32.600 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.28449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23945.log 2024-04-26T04:27:32.600 INFO:teuthology.orchestra.run.smithi186.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.32003.log.gz 2024-04-26T04:27:32.600 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.30448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31638.log 2024-04-26T04:27:32.600 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.23945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28239.log 2024-04-26T04:27:32.600 INFO:teuthology.orchestra.run.smithi186.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.28449.log.gz 2024-04-26T04:27:32.601 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.31638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29980.log 2024-04-26T04:27:32.601 INFO:teuthology.orchestra.run.smithi186.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.23945.log.gz 2024-04-26T04:27:32.601 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.28239.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31730.log 2024-04-26T04:27:32.601 INFO:teuthology.orchestra.run.smithi186.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.30448.log.gz 2024-04-26T04:27:32.601 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.29980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31091.log 2024-04-26T04:27:32.601 INFO:teuthology.orchestra.run.smithi186.stderr: 89.4%/var/log/ceph/ceph-client.admin.31730.log: -- replaced with /var/log/ceph/ceph-client.admin.31638.log.gz 2024-04-26T04:27:32.601 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28186.log 2024-04-26T04:27:32.601 INFO:teuthology.orchestra.run.smithi186.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.29980.log.gz 2024-04-26T04:27:32.601 INFO:teuthology.orchestra.run.smithi186.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.28239.log.gz 2024-04-26T04:27:32.601 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.31091.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.31730.log.gz 2024-04-26T04:27:32.601 INFO:teuthology.orchestra.run.smithi186.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.31091.log.gz 2024-04-26T04:27:32.601 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31457.log 2024-04-26T04:27:32.601 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.28186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28535.log 2024-04-26T04:27:32.602 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.31457.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-26T04:27:32.602 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.28535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28360.log 2024-04-26T04:27:32.602 INFO:teuthology.orchestra.run.smithi186.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.31457.log.gz 2024-04-26T04:27:32.602 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph.audit.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.28186.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29840.log 2024-04-26T04:27:32.602 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-04-26T04:27:32.602 INFO:teuthology.orchestra.run.smithi186.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.28535.log.gz 2024-04-26T04:27:32.602 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.28360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30946.log 2024-04-26T04:27:32.602 INFO:teuthology.orchestra.run.smithi186.stderr: 89.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-26T04:27:32.602 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.29840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29405.log 2024-04-26T04:27:32.602 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.30946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29145.log 2024-04-26T04:27:32.602 INFO:teuthology.orchestra.run.smithi186.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.29840.log.gz 2024-04-26T04:27:32.602 INFO:teuthology.orchestra.run.smithi186.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.28360.log.gz 2024-04-26T04:27:32.602 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.29405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28362.log 2024-04-26T04:27:32.602 INFO:teuthology.orchestra.run.smithi186.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.30946.log.gz 2024-04-26T04:27:32.603 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.29145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29754.log 2024-04-26T04:27:32.603 INFO:teuthology.orchestra.run.smithi186.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.29405.log.gz 2024-04-26T04:27:32.603 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.28362.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28413.log 2024-04-26T04:27:32.603 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.29754.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.29145.log.gz 2024-04-26T04:27:32.603 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29458.log 2024-04-26T04:27:32.603 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.28413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29718.log 2024-04-26T04:27:32.603 INFO:teuthology.orchestra.run.smithi186.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.28362.log.gz 2024-04-26T04:27:32.603 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.29458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28099.log 2024-04-26T04:27:32.603 INFO:teuthology.orchestra.run.smithi186.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.29754.log.gz 2024-04-26T04:27:32.603 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30910.log 2024-04-26T04:27:32.603 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.29718.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.29458.log.gz 2024-04-26T04:27:32.603 INFO:teuthology.orchestra.run.smithi186.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.28413.log.gz 2024-04-26T04:27:32.603 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28101.log 2024-04-26T04:27:32.603 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.30910.log: /var/log/ceph/ceph-client.admin.28099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32040.log 2024-04-26T04:27:32.603 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.28101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30635.log 2024-04-26T04:27:32.604 INFO:teuthology.orchestra.run.smithi186.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.29718.log.gz 2024-04-26T04:27:32.604 INFO:teuthology.orchestra.run.smithi186.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.30910.log.gz 2024-04-26T04:27:32.604 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.32040.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.28099.log.gz 2024-04-26T04:27:32.604 INFO:teuthology.orchestra.run.smithi186.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.28101.log.gz 2024-04-26T04:27:32.604 INFO:teuthology.orchestra.run.smithi186.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.32040.log.gz 2024-04-26T04:27:32.604 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30362.log 2024-04-26T04:27:32.604 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.30635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31401.log 2024-04-26T04:27:32.604 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.30362.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28936.log 2024-04-26T04:27:32.604 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.31401.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29056.log 2024-04-26T04:27:32.604 INFO:teuthology.orchestra.run.smithi186.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.30635.log.gz 2024-04-26T04:27:32.604 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.28936.log: gzip 89.0% -- replaced with /var/log/ceph/ceph-client.admin.30362.log.gz -5 2024-04-26T04:27:32.604 INFO:teuthology.orchestra.run.smithi186.stderr: --verbose -- /var/log/ceph/ceph-client.admin.30855.log 2024-04-26T04:27:32.604 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28797.log 2024-04-26T04:27:32.604 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.29056.log: /var/log/ceph/ceph-client.admin.30855.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.28936.log.gz 2024-04-26T04:27:32.605 INFO:teuthology.orchestra.run.smithi186.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.31401.log.gz 2024-04-26T04:27:32.605 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27805.log 2024-04-26T04:27:32.605 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29806.log 2024-04-26T04:27:32.605 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.28797.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.30855.log.gz 2024-04-26T04:27:32.605 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30724.log 2024-04-26T04:27:32.605 INFO:teuthology.orchestra.run.smithi186.stderr: 89.5%/var/log/ceph/ceph-client.admin.27805.log: -- replaced with /var/log/ceph/ceph-client.admin.29056.log.gz 2024-04-26T04:27:32.605 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.29806.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.28797.log.gz 2024-04-26T04:27:32.605 INFO:teuthology.orchestra.run.smithi186.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.27805.log.gz 2024-04-26T04:27:32.605 INFO:teuthology.orchestra.run.smithi186.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.29806.log.gz 2024-04-26T04:27:32.605 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30536.log 2024-04-26T04:27:32.605 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.30724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28536.log 2024-04-26T04:27:32.605 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.30536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27925.log 2024-04-26T04:27:32.605 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.28536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30100.log 2024-04-26T04:27:32.605 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.27925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31948.log 2024-04-26T04:27:32.606 INFO:teuthology.orchestra.run.smithi186.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.30724.log.gz 2024-04-26T04:27:32.606 INFO:teuthology.orchestra.run.smithi186.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.30536.log.gz 2024-04-26T04:27:32.606 INFO:teuthology.orchestra.run.smithi186.stderr: 90.6%/var/log/ceph/ceph-client.admin.30100.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-04-26T04:27:32.606 INFO:teuthology.orchestra.run.smithi186.stderr: -- replaced with /var/log/ceph/ceph-client.admin.28536.log.gz 2024-04-26T04:27:32.606 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.31948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29318.log 2024-04-26T04:27:32.606 INFO:teuthology.orchestra.run.smithi186.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.27925.log.gz 2024-04-26T04:27:32.606 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30947.log 2024-04-26T04:27:32.606 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-mon.b.log: /var/log/ceph/ceph-client.admin.29318.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.30100.log.gz 2024-04-26T04:27:32.606 INFO:teuthology.orchestra.run.smithi186.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.31948.log.gz 2024-04-26T04:27:32.606 INFO:teuthology.orchestra.run.smithi186.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.29318.log.gz 2024-04-26T04:27:32.606 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-04-26T04:27:32.606 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.30947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31183.log 2024-04-26T04:27:32.606 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30671.log 2024-04-26T04:27:32.606 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.31183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29230.log 2024-04-26T04:27:32.607 INFO:teuthology.orchestra.run.smithi186.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.30947.log.gz 2024-04-26T04:27:32.607 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.30671.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.31183.log.gz 2024-04-26T04:27:32.607 INFO:teuthology.orchestra.run.smithi186.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.30671.log.gz 2024-04-26T04:27:32.607 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30725.log 2024-04-26T04:27:32.607 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.29230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31092.log 2024-04-26T04:27:32.607 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.30725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31402.log 2024-04-26T04:27:32.607 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.31092.log: 89.5% 88.0% -- replaced with /var/log/ceph/ceph-client.admin.30725.log.gz 2024-04-26T04:27:32.607 INFO:teuthology.orchestra.run.smithi186.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29230.log.gz 2024-04-26T04:27:32.607 INFO:teuthology.orchestra.run.smithi186.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.31092.log.gz 2024-04-26T04:27:32.607 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30535.log 2024-04-26T04:27:32.607 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.31402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28065.log 2024-04-26T04:27:32.607 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.30535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28849.log 2024-04-26T04:27:32.607 INFO:teuthology.orchestra.run.smithi186.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.31402.log.gz 2024-04-26T04:27:32.607 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.28065.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.30535.log.gz 2024-04-26T04:27:32.608 INFO:teuthology.orchestra.run.smithi186.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.28065.log.gz 2024-04-26T04:27:32.608 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31493.log 2024-04-26T04:27:32.608 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.28849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29839.log 2024-04-26T04:27:32.608 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.31493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28188.log 2024-04-26T04:27:32.608 INFO:teuthology.orchestra.run.smithi186.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.28849.log.gz 2024-04-26T04:27:32.608 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.29839.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.31493.log.gz 2024-04-26T04:27:32.608 INFO:teuthology.orchestra.run.smithi186.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.29839.log.gz 2024-04-26T04:27:32.608 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31639.log 2024-04-26T04:27:32.608 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.28188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28935.log 2024-04-26T04:27:32.608 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.31639.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31548.log 2024-04-26T04:27:32.608 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.28935.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.28188.log.gz 2024-04-26T04:27:32.608 INFO:teuthology.orchestra.run.smithi186.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.31639.log.gz 2024-04-26T04:27:32.608 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30013.log 2024-04-26T04:27:32.608 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.31548.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.28935.log.gz 2024-04-26T04:27:32.609 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29579.log 2024-04-26T04:27:32.609 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30502.log 2024-04-26T04:27:32.609 INFO:teuthology.orchestra.run.smithi186.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.31548.log.gz/var/log/ceph/ceph-client.admin.30013.log: /var/log/ceph/ceph-client.admin.29579.log: 2024-04-26T04:27:32.609 INFO:teuthology.orchestra.run.smithi186.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.29579.log.gz 2024-04-26T04:27:32.609 INFO:teuthology.orchestra.run.smithi186.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.30013.log.gz 2024-04-26T04:27:32.609 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28326.log 2024-04-26T04:27:32.609 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.30502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31766.log 2024-04-26T04:27:32.609 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.28326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31365.log 2024-04-26T04:27:32.609 INFO:teuthology.orchestra.run.smithi186.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.30502.log.gz 2024-04-26T04:27:32.609 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.31766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31584.log 2024-04-26T04:27:32.609 INFO:teuthology.orchestra.run.smithi186.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.31766.log.gz 2024-04-26T04:27:32.609 INFO:teuthology.orchestra.run.smithi186.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.28326.log.gz 2024-04-26T04:27:32.609 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.31365.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.31365.log.gz 2024-04-26T04:27:32.609 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30945.log 2024-04-26T04:27:32.610 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.31584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29632.log 2024-04-26T04:27:32.610 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.30945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28588.log 2024-04-26T04:27:32.610 INFO:teuthology.orchestra.run.smithi186.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.31584.log.gz 2024-04-26T04:27:32.610 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.29632.log: 88.9% 89.3% -- replaced with /var/log/ceph/ceph-client.admin.30945.log.gz 2024-04-26T04:27:32.610 INFO:teuthology.orchestra.run.smithi186.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29632.log.gz 2024-04-26T04:27:32.610 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29492.log 2024-04-26T04:27:32.610 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.28588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30014.log 2024-04-26T04:27:32.610 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.29492.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.28588.log.gz 2024-04-26T04:27:32.610 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.30014.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.29492.log.gz 2024-04-26T04:27:32.610 INFO:teuthology.orchestra.run.smithi186.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.30014.log.gz 2024-04-26T04:27:32.611 INFO:teuthology.orchestra.run.smithi186.stderr: 90.8% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-04-26T04:27:32.731 INFO:teuthology.orchestra.run.smithi063.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-04-26T04:27:32.864 INFO:teuthology.orchestra.run.smithi186.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-04-26T04:27:32.885 INFO:teuthology.orchestra.run.smithi063.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-04-26T04:27:33.733 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2024-04-26T04:27:34.679 INFO:teuthology.orchestra.run.smithi186.stderr: 91.9% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-04-26T04:27:34.682 INFO:teuthology.orchestra.run.smithi186.stderr: 91.9% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-04-26T04:27:34.951 INFO:teuthology.orchestra.run.smithi186.stderr: 92.4% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-04-26T04:27:34.953 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-04-26T04:27:34.953 INFO:teuthology.orchestra.run.smithi186.stderr:real 0m2.558s 2024-04-26T04:27:34.953 INFO:teuthology.orchestra.run.smithi186.stderr:user 0m7.455s 2024-04-26T04:27:34.953 INFO:teuthology.orchestra.run.smithi186.stderr:sys 0m0.378s 2024-04-26T04:27:35.086 INFO:teuthology.orchestra.run.smithi063.stderr: 92.4% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-04-26T04:27:35.139 INFO:teuthology.orchestra.run.smithi063.stderr: 92.4% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-04-26T04:27:35.335 INFO:teuthology.orchestra.run.smithi063.stderr: 92.4% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-04-26T04:27:35.336 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-26T04:27:35.337 INFO:teuthology.orchestra.run.smithi063.stderr:real 0m2.980s 2024-04-26T04:27:35.337 INFO:teuthology.orchestra.run.smithi063.stderr:user 0m8.717s 2024-04-26T04:27:35.337 INFO:teuthology.orchestra.run.smithi063.stderr:sys 0m0.339s 2024-04-26T04:27:35.338 INFO:tasks.ceph:Archiving logs... 2024-04-26T04:27:35.338 DEBUG:teuthology.misc:Transferring archived files from smithi063:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-04-26_01:23:56-rados-wip-yuri2-testing-2024-04-24-0914-squid-distro-default-smithi/7674515/remote/smithi063/log 2024-04-26T04:27:35.339 DEBUG:teuthology.orchestra.run.smithi063:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-26T04:27:36.676 DEBUG:teuthology.misc:Transferring archived files from smithi186:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-04-26_01:23:56-rados-wip-yuri2-testing-2024-04-24-0914-squid-distro-default-smithi/7674515/remote/smithi186/log 2024-04-26T04:27:36.676 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-26T04:27:38.016 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-26T04:27:38.025 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-26T04:27:38.026 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-26T04:27:38.028 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-26T04:27:38.094 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-26T04:27:38.106 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-26T04:27:38.107 DEBUG:teuthology.orchestra.run.smithi063:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T04:27:38.110 DEBUG:teuthology.orchestra.run.smithi186:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-26T04:27:38.121 INFO:teuthology.orchestra.run.smithi063.stdout: remote refid st t when poll reach delay offset jitter 2024-04-26T04:27:38.121 INFO:teuthology.orchestra.run.smithi063.stdout:============================================================================== 2024-04-26T04:27:38.121 INFO:teuthology.orchestra.run.smithi063.stdout:*hv01.front.sepi 67.205.162.81 3 u 37 64 377 0.088 -1.722 0.135 2024-04-26T04:27:38.121 INFO:teuthology.orchestra.run.smithi063.stdout:+hv02.front.sepi 63.231.80.2 3 u 34 64 377 0.092 +1.083 0.204 2024-04-26T04:27:38.121 INFO:teuthology.orchestra.run.smithi063.stdout:+hv03.front.sepi 74.6.168.72 3 u 33 64 377 0.109 -0.055 0.186 2024-04-26T04:27:38.121 INFO:teuthology.orchestra.run.smithi063.stdout: hv04.front.sepi .INIT. 16 u - 128 0 0.000 +0.000 0.000 2024-04-26T04:27:38.122 INFO:teuthology.orchestra.run.smithi186.stdout: remote refid st t when poll reach delay offset jitter 2024-04-26T04:27:38.123 INFO:teuthology.orchestra.run.smithi186.stdout:============================================================================== 2024-04-26T04:27:38.123 INFO:teuthology.orchestra.run.smithi186.stdout:*hv01.front.sepi 67.205.162.81 3 u 35 64 377 0.074 -2.750 0.818 2024-04-26T04:27:38.123 INFO:teuthology.orchestra.run.smithi186.stdout:+hv02.front.sepi 63.231.80.2 3 u 39 64 377 0.100 +0.835 0.554 2024-04-26T04:27:38.123 INFO:teuthology.orchestra.run.smithi186.stdout:+hv03.front.sepi 74.6.168.72 3 u 35 64 377 0.082 -0.629 0.548 2024-04-26T04:27:38.123 INFO:teuthology.orchestra.run.smithi186.stdout: hv04.front.sepi .INIT. 16 u - 128 0 0.000 +0.000 0.000 2024-04-26T04:27:38.124 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-26T04:27:38.133 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-26T04:27:38.134 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-26T04:27:38.143 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-26T04:27:38.154 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-26T04:27:38.171 INFO:teuthology.task.internal:Duration was 1127.469928 seconds 2024-04-26T04:27:38.171 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-26T04:27:38.180 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-26T04:27:38.180 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-26T04:27:38.183 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-26T04:27:38.218 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-26T04:27:38.219 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi063.front.sepia.ceph.com 2024-04-26T04:27:38.219 DEBUG:teuthology.orchestra.run.smithi063:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-26T04:27:38.270 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi186.front.sepia.ceph.com 2024-04-26T04:27:38.271 DEBUG:teuthology.orchestra.run.smithi186:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-26T04:27:38.280 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-26T04:27:38.281 DEBUG:teuthology.orchestra.run.smithi063:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-26T04:27:38.315 DEBUG:teuthology.orchestra.run.smithi186:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-26T04:27:38.354 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-26T04:27:38.354 DEBUG:teuthology.orchestra.run.smithi063:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-26T04:27:38.383 DEBUG:teuthology.orchestra.run.smithi186:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-26T04:27:38.452 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-26T04:27:38.463 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-26T04:27:38.463 DEBUG:teuthology.orchestra.run.smithi063:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-26T04:27:38.477 DEBUG:teuthology.orchestra.run.smithi186:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-26T04:27:38.491 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-26T04:27:38.503 DEBUG:teuthology.orchestra.run.smithi063:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-26T04:27:38.523 DEBUG:teuthology.orchestra.run.smithi186:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-26T04:27:38.532 INFO:teuthology.orchestra.run.smithi063.stdout:kernel.core_pattern = core 2024-04-26T04:27:38.546 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern = core 2024-04-26T04:27:38.638 DEBUG:teuthology.orchestra.run.smithi063:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-26T04:27:38.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T04:27:38.643 DEBUG:teuthology.orchestra.run.smithi186:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-26T04:27:38.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-26T04:27:38.684 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-26T04:27:38.695 INFO:teuthology.task.internal:Transferring archived files... 2024-04-26T04:27:38.695 DEBUG:teuthology.misc:Transferring archived files from smithi063:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-26_01:23:56-rados-wip-yuri2-testing-2024-04-24-0914-squid-distro-default-smithi/7674515/remote/smithi063 2024-04-26T04:27:38.696 DEBUG:teuthology.orchestra.run.smithi063:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-26T04:27:38.718 DEBUG:teuthology.misc:Transferring archived files from smithi186:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-26_01:23:56-rados-wip-yuri2-testing-2024-04-24-0914-squid-distro-default-smithi/7674515/remote/smithi186 2024-04-26T04:27:38.718 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-26T04:27:38.748 INFO:teuthology.task.internal:Removing archive directory... 2024-04-26T04:27:38.748 DEBUG:teuthology.orchestra.run.smithi063:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-26T04:27:38.759 DEBUG:teuthology.orchestra.run.smithi186:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-26T04:27:38.792 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-26T04:27:38.804 INFO:teuthology.task.internal:Not uploading archives. 2024-04-26T04:27:38.804 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-26T04:27:38.818 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-26T04:27:38.819 DEBUG:teuthology.orchestra.run.smithi063:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-26T04:27:38.821 DEBUG:teuthology.orchestra.run.smithi186:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-26T04:27:38.825 INFO:teuthology.orchestra.run.smithi063.stdout: 658630 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 26 04:27 /home/ubuntu/cephtest 2024-04-26T04:27:38.840 INFO:teuthology.orchestra.run.smithi186.stdout: 658638 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 26 04:27 /home/ubuntu/cephtest 2024-04-26T04:27:38.841 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-26T04:27:38.855 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-26T04:27:38.879 INFO:teuthology.run:Summary data: description: rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/sync-many workloads/pool-create-delete} duration: 1127.4699277877808 flavor: default owner: scheduled_yuriw@teuthology success: true 2024-04-26T04:27:38.879 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-26T04:27:38.961 INFO:teuthology.run:pass